Skip to main content
SpringerLink
Log in

A real-time multi-camera vision system for UAV collision warning and navigation

  • Special Issue Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

A real-time vision system with multiple cameras was developed for UAV collision warning and visual navigation for fixed-wing small- or medium-sized aircrafts. The embedded vision system simultaneously acquires images using five cameras, stores, and evaluates the visual data with an FPGA-based multi-core processor system. The system was designed to fulfill the strict size, power, and weight requirements arising from UAV on-board restrictions. The hardware parameters of the system and the performance of the algorithm are compared to the state of the art.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Felder, W.: Unmanned System Integration into the National Airspace System. Philadelphia, PA, USA: Keynote Presented at ICUAS 2012, June (2012)

  2. Dey, D., Geyer, C., Singh, S., Digioia, M.: Passive, long-range detection of aircraft: towards a field deployable sense and avoid system. Field and Service Robotics, Springer Tracts in Advanced Robotics, 2010, Vol. 62, pp. 113–123 (2010). doi: 10.1007/978-3-642-13408-1_11

  3. Federal Aviation Administration: Fact sheet—unmanned aircraft systems (UAS) (2010)

  4. Department of Defense: Unmanned aircraft system airspace integration plan. Tech. Rep. March, Department of Defense (2011)

  5. Federal Aviation Administration: Integration of unmanned aircraft systems into the National Airspace System Concept of Operations (2012)

  6. Mejias, L., McNamara, S., Lai, J., Ford, J.: Vision-based detection and tracking of aerial targets for UAV collision avoidance. International Conference on Intelligent Robots and Systems (IROS), pp. 87–92 (2010)

  7. Dey, D., Geyer, C., Singh, S., Digioia, M.: Passive, long-range detection of aircraft: towards a field deployable sense and avoid system. Robotics Institute. Paper 962 (2009)

  8. Mejias, L., Ford, J. J., Lai, J. S.: Towards the implementation of vision-based UAS sense-and-avoid. In: Proceedings of the 27th International Congress of the Aeronautical Sciences (ICAS 2010), 19–24. September 2010, Acropolis Conference, Centre, Nice (2010)

  9. Livadas, C., Lygeros, J., Lynch, A., Nancy, : High-level modeling and analysis of the traffic alert and collision avoidance system (TCAS). Proc. IEEE 88(7), 926–948 (2000)

    Article  Google Scholar 

  10. Federal Aviation Administration: Fact sheet—automatic dependent surveillance-broadcast (ADB-S) (2010)

  11. Fasano, G., Accardo, D., Moccia, A., Carbone, C., Ciniglio, U., Corraro, F., Luongo, S.: Multi-sensor-based fully autonomous non-cooperative collision avoidance system for unmanned air vehicles. J. Aerosp. Comput. Inf. Commun. 5(10), 338–360 (2008)

    Article  Google Scholar 

  12. Achtelik, M., Stephan W., Siegwart, R.: Onboard IMU and monocular vision based control for MAVs in unknown in-and outdoor environments. Robotics and automation (ICRA), 2011 IEEE international conference on. IEEE, (2011)

  13. Meier, L., Tanskanen, P., Heng, L., Lee, G.H., Fraundorfer, F., Pollefeys, M.: PIXHAWK: a micro aerial vehicle design for autonomous flight using onboard computer vision. Auton. Robots 33(1–2), 21–39 (2012)

    Article  Google Scholar 

  14. Nikolic, J., Burri, M., Rehder, J., Leutenegger, S., Huerzeler, C., Siegwart, R.: A UAV system for inspection of industrial facilities. In: Aerospace Conference, 2013 IEEE (pp. 1–8) IEEE (2013, March)

  15. Ahlberg, C., Lidholm, J., Ekstrand, F., Spampinato, G., Ekstrom, M., Asplund, L.: Gimme-a general image multiview manipulation engine. In: Reconfigurable Computing and FPGAs (ReConFig), 2011 International Conference on (pp. 129–134) IEEE (2011, November)

  16. Mejias, L., McNamara, S., Lai, J., Ford, J.: Vision-based detection and tracking of aerial targets for UAV collision avoidance. In: Intelligent Robots and Systems (IROS), 2010 IEEE/RSJ International Conference on (pp. 87–92) IEEE (2010, October)

  17. International Civil Aviation Organization: Air traffic management. ICAO Doc 4444, fifteenth edition (2007)

  18. Zarándy, Á., Zsedrovits, T., Nagy, Z., Kiss, A., Roska, T.: On-board see-and-avoid system. Conference of the Hungarian Association for Image Processing and Pattern Recognition (Kepaf 2013) pp. 604–617 Bakonybel, (2013)

  19. http://www.framos.com/products/en/cameras/camera-modules/fcb-ma130-15447.html (2014). Accessed 2 Sept 2014

  20. http://www.sedeco.nl/sedeco/index.php/lenses/smount  (2014). Accessed 2 Sept 2014

  21. http://www.mobisensesystems.com/pages_en/aptina_modules.html (2014). Accessed 2 Sept 2014

  22. https://www.leopardimaging.com/LI-USB30-M021.html (2014). Accessed 2 Sept 2014

  23. http://www.ptgrey.com/products/index.asp (2014). Accessed 2 Sept 2014

  24. http://www.edmundoptics.com/imaging/imaging-lenses/ (2014). Accessed 2 Sept 2014

  25. http://defense.ge-ip.com/products/gpgpu/c497 (2014). Accessed 2 Sept 2014

  26. http://www.tokudenkairo.co.jp/exp6t/ (2014). Accessed 2 Sept 2014

  27. http://www.legitreviews.com/article/1980/1/ (2014). Accessed 2 Sept 2014

  28. Zsedrovits, T.: Visual sense and avoid for fixed-wing small size unmanned aerial vehicles. Dissertation. Pazmany Peter Catholic University, Budapest, Hungary, 2014. https://itk.ppke.hu/uploads/articles/162/file/Zsedrovits%20Tamas%20dissertation.pdf. Accessed 2 Sept 2014

  29. Zsedrovits, T., Zarandy, A., Vanek, B., Peni, T., Bokor, J., Roska, T.: Estimation of relative direction angle of distant, approaching airplane in sense-and-avoid. J. Intell. Rob. Syst. 69(1–4), 407–415 (2013)

    Article  Google Scholar 

  30. Chu, T., Guo, N., Backén, S., Akos, D.: Monocular camera/IMU/GNSS integration for ground vehicle navigation in challenging GNSS environments. Sensors (Basel, Switzerland), vol. 12, no. 3, pp. 3162–85, Jan. 2012

  31. Zsedrovits, T., Bauer, P., Zarandy, A., Vanek, B., Bokor, J., Roska, T.: Error analysis of algorithms for camera rotation calculation in GPS/IMU/camera fusion for UAV sense and avoid systems. ICUAS (2014)

  32. Zsedrovits, T., Bauer, P., Zarandy, A., Vanek, B., Bokor, J., Roska, T.: Towards real-time visual and IMU data fusion. Presented at the AIAA Guidance, Navigation, and Control Conference and Exhibit, (2014)

  33. Ristic, B., Vo, B.N., Clark, D., Vo, B.T.: A metric for performance evaluation of multi-target tracking algorithms. Signal Process. IEEE Trans. 59(7), 3452–3457 (2011)

    Article  MathSciNet  Google Scholar 

  34. Nistér, D.: An efficient solution to the five-point relative pose problem. IEEE Trans. Pattern Anal. Mach. Intell. 26(6), 756–777 (2004)

    Article  Google Scholar 

  35. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge University Press, 2004

  36. Torr, P.H.S., Zisserman, A.: MLESAC: a new robust estimator with application to estimating image geometry. Comput. Vis. Image Underst. 78(1), 138–156 (2000)

    Article  Google Scholar 

  37. Stewénius, H., et al.: Recent developments on direct relative orientation. ISPRS J. Photogramm. Remote Sens. 60(4), 284–294 (2006)

    Article  Google Scholar 

  38. Bailey, D. G., Johnston, C. T.: Single pass connected components analysis. In: Proceedings of image and vision computing New Zealand 2007, pp. 282–287, Hamilton, New Zealand, December 2007

  39. Nagy, Z., Kiss, A., Zarándy, Á., Vanek, B., Péni, T., Bokor, J., Roska, T.: Volume and power optimized high-performance system for UAV collision avoidance ISCAS-2012, Seoul, Korea (2012)

Download references

Acknowledgments

The ONR Grants (N62909-11-1-7039, N62909-10-1-7081) is greatly acknowledged., The authors express their thanks to grants TÁMOP- 4.2.1.B-11/2/KRM-2011-0002 and TÁMOP-4.2.2/B-10/1-2010-0014.

Author information

Authors and Affiliations

  1. Institute for Computer Science and Control, 13-17 Kende Street, Budapest, H-1111, Hungary

    Ákos Zarándy, Mate Nemeth, Zoltan Nagy, Andras Kiss & Levente Santha

  2. Pazmany Peter Catholic University, Budapest, Hungary

    Ákos Zarándy, Mate Nemeth, Zoltan Nagy, Andras Kiss & Tamás Zsedrovits

Authors
  1. Ákos Zarándy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mate Nemeth
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zoltan Nagy
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Andras Kiss
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Levente Santha
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tamás Zsedrovits
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ákos Zarándy.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zarándy, Á., Nemeth, M., Nagy, Z. et al. A real-time multi-camera vision system for UAV collision warning and navigation. J Real-Time Image Proc 12, 709–724 (2016). https://doi.org/10.1007/s11554-014-0449-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-014-0449-3

Keywords

Search

Navigation