Skip to main content
SpringerLink
Log in

UAV-based autonomous detection and tracking of beyond visual range (BVR) non-stationary targets using deep learning

  • Original Research Paper
  • Published:
Journal of Real-Time Image Processing Aims and scope Submit manuscript

Abstract

Aerial surveillance and tracking have gained significant traction in recent years for both civilian applications and military reconnaissance. Disaster analysis, emergency medical response, pandemic spread analysis, etc. have significantly improved with the availability of aerial data. The next big step is to push the system for autonomous detection and tracking of targets beyond visual range (BVR). Presently, this is done using GPS-based techniques in which the target information is assumed to be precisely known. In situations where such information is unavailable or if the target of interest is non-stationary, this method is not applicable and currently, no alternative exists. In this work, we aim to address this limitation and propose a deep learning-based algorithm for terminal guidance of aerial vehicle BVR with only bearing information about the target of interest. The algorithm operates in search and track modes. We describe both the modes and also discuss the challenges associated with this kind of deployment in real time. Since the weight and power requirements of the payload directly translate to the cost of deployment and endurance of aerial vehicles, we have configured a custom lightweight convolutional neural network (CNN) with minimal layers and successfully deployed the system on Jetson Nano, the smallest GPU available from NVIDIA as of this writing. We evaluated the performance of the proposed algorithm on proprietary and open-source datasets and achieved detection accuracy greater than 98.6% on custom datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Genovese, A.F.: The interacting multiple model algorithm for accurate state estimation of maneuvering targets. Johns Hopkins APL Tech. Dig. 22(4), 614–623 (2001)

    Google Scholar 

  2. Bar, S.Y., Fortmann, T., et al.: Tracking and data association. PhD thesis, Academic Press, Cambridge (1988)

  3. Israelsen, J., Beall, M., Bareiss, D., Stuart, D., Keeney, E., van den Berg, J.: Automatic collision avoidance for manually tele-operated unmanned aerial vehicles. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 6638–6643 (2014)

  4. Ding, W., Wang, J., Han, S., Almagbile, A., Garratt, M.A. , Lambert, A., Wang, J.J.: Adding optical flow into the gps/ins integration for uav navigation. In: International Global Navigation Satellite Systems Society Symposium, pp. 1–13 (2009)

  5. Mercado, D., Flores, G., Castillo, P., Escareno, J., Lozano, R.: Gps/ins/optic flow data fusion for position and velocity estimation. In: International Conference on Unmanned Aircraft Systems (ICUAS), pp. 486–491 (2013)

  6. Peng, X.-Z., Lin, H.-Y., Dai, J.-M.: Path planning and obstacle avoidance for vision guided quadrotor UAV navigation. In: 12th IEEE International Conference on Control and Automation (ICCA), pp. 984–989 (2016)

  7. Chee, K., Zhong, Z.: Control, navigation and collision avoidance for an unmanned aerial vehicle. Sens. Actuators A Phys. 190, 66–76 (2013)

    Article  Google Scholar 

  8. Cui, J.Q., Lai, S., Dong, X., Chen, B.M.: Autonomous navigation of UAV in foliage environment. J. Intell. Robot. Syst. 84(1–4), 259–276 (2016)

    Article  Google Scholar 

  9. Roberge, V., Tarbouchi, M.: Fast path planning for unmanned aerial vehicle using embedded GPU system. In: 14th International Multi-Conference on Systems, Signals & Devices (SSD), pp. 145–150, IEEE (2017)

  10. Roelofsen, S., Gillet, D., Martinoli, A.: Reciprocal collision avoidance for quadrotors using on-board visual detection. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4810–4817, IEEE (2015)

  11. Gageik, N., Benz, P., Montenegro, S.: Obstacle detection and collision avoidance for a UAV with complementary low-cost sensors. IEEE Access 3, 599–609 (2015)

    Article  Google Scholar 

  12. Fu, C., Olivares-Mendez, M.A., Suarez-Fernandez, R., Campoy, P.: Monocular visual-inertial slam-based collision avoidance strategy for fail-safe UAV using fuzzy logic controllers. J. Intell. Robot. Syst. 73(1–4), 513–533 (2014)

    Article  Google Scholar 

  13. Venkateswarlu, R., Sujata, K., Rao, B.V.: Centroid tracker and aimpoint selection. In: Acquisition, Tracking, and Pointing VI, vol. 1697, pp. 520–529. International Society for Optics and Photonics (1992)

  14. Ellis, J.G., Kramer, K.A., Stubberud, S.C.: Image correlation based video tracking. In: 21st International Conference on Systems Engineering, pp. 132–136, IEEE (2011)

  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

  16. Ye, D.H., Li, J., Chen, Q., Wachs, J., Bouman, C.: Deep learning for moving object detection and tracking from a single camera in unmanned aerial vehicles (UAVS). Electron. Imaging 2018(10), 466–1 (2018)

    Article  Google Scholar 

  17. Geng, L., Zhang, Y., Wang, J., Fuh, J.Y., Teo, S.: Mission planning of autonomous UAVS for urban surveillance with evolutionary algorithms. In: 10th IEEE International Conference on Control and Automation (ICCA), pp. 828–833 (2013)

  18. Bah, M.D., Hafiane, A., Canals, R.: Deep learning with unsupervised data labeling for weed detection in line crops in UAV images. Remote Sens. 10(11), 1690 (2018)

    Article  Google Scholar 

  19. Tri, N.C., Duong, H.N., Van Hoai, T., Van Hoa, T., Nguyen, V.H., Toan, N.T., Snasel, V.: A novel approach based on deep learning techniques and uavs to yield assessment of paddy fields. In: 9th International Conference on Knowledge and Systems Engineering (KSE), pp. 257–262 (2017)

  20. Paredes, J.A., González, J., Saito, C., Flores, A.: Multispectral imaging system with uav integration capabilities for crop analysis. In: First IEEE International Symposium of Geoscience and Remote Sensing (GRSS-CHILE), pp. 1–4 (2017)

  21. Nagai, M., Chen, T., Shibasaki, R., Kumagai, H., Ahmed, A.: Uav-borne 3-d mapping system by multisensor integration. IEEE Trans. Geosci. Remote Sens. 47(3), 701–708 (2009)

    Article  Google Scholar 

  22. Metni, N., Hamel, T.: A UAV for bridge inspection: Visual servoing control law with orientation limits. Autom. Constr. 17(1), 3–10 (2007)

    Article  Google Scholar 

  23. Waharte, S., Trigoni, N.: Supporting search and rescue operations with UAVS. In: International Conference on Emerging Security Technologies, pp. 142–147, IEEE (2010)

  24. Rozantsev, A., Lepetit, V., Fua, P.: Flying objects detection from a single moving camera. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4128–4136 (2015)

  25. Oliver, N.M., Rosario, B., Pentland, A.P.: A Bayesian computer vision system for modeling human interactions. IEEE Trans. Pattern Anal. Mach. Intell. 22(8), 831–843 (2000)

    Article  Google Scholar 

  26. Noh, S., Jeon, M.: A new framework for background subtraction using multiple cues. In: Asian Conference on Computer Vision, pp. 493–506. Springer, New York (2012)

  27. Lucas, B.D., Kanade, T. et al.: An iterative image registration technique with an application to stereo vision. In: International Joint Conference on Artificial Intelligence (IJCAI), Vancouver, British Columbia (1981)

  28. Brox, T., Malik, J.: Large displacement optical flow: descriptor matching in variational motion estimation. IEEE Trans. Pattern Anal. Mach. Intell. 33(3), 500–513 (2010)

    Article  Google Scholar 

  29. Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2544–2550 (2010)

  30. Danelljan, M., Hager, G., Shahbaz Khan, F., Felsberg, M.: Learning spatially regularized correlation filters for visual tracking. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 4310–4318 (2015)

  31. Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8971–8980 (2018)

  32. Felsberg, M., Berg, A., Hager, G., Ahlberg, J., Kristan, M., Matas, J., Leonardis, A., Cehovin, L., Fernandez, G., Vojir, T., et al.: The thermal infrared visual object tracking vot-tir2015 challenge results. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 76–88 (2015)

  33. Wu, Y., Lim, J., Yang, M.-H.: Online object tracking: a benchmark. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2411–2418 (2013)

  34. Danelljan, M., Shahbaz Khan, F., Felsberg, M., Van de Weijer, J.: Adaptive color attributes for real-time visual tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1090–1097 (2014)

  35. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2014)

    Article  Google Scholar 

  36. Lowe, G.: Sift-the scale invariant feature transform. Int. J. 2, 91–110 (2004)

    Google Scholar 

  37. Dalal, N., triggs, B.: Histograms of oriented gradients. In: IEEE Conf. Comp. Vision and Pattern Recog, vol. 1, pp. 886–893 (2005)

  38. Felzenszwalb, P., McAllester, D., Ramanan, D.: A discriminatively trained, multiscale, deformable part model. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2008)

  39. Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009)

  40. Danelljan, M., Bhat, G., Khan, F.S., Felsberg, M.: Atom: Accurate tracking by overlap maximization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4660–4669 (2019)

  41. Milan, A., Rezatofighi, S. H., Dick, A., Reid, I., Schindler, K.: Online multi-target tracking using recurrent neural networks. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

  42. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

  43. He, K.: Author’s webpage(kaiminghe). http://kaiminghe.com/

  44. Iandola, F.N., Han, S., Moskewicz, M.W., Ashraf, K., Dally, W.J., Keutzer, K.: Squeezenet: Alexnet-level accuracy with 50x fewer parameters and \(<\) 0.5 mb model size. arXiv preprintarXiv: 1602.07360 (2016)

  45. Redmon, J., Farhadi, A.: Tiny yolo (2017)

  46. Polvara, R., Patacchiola, M., Sharma, S., Wan, J., Manning, A., Sutton, R., Cangelosi, A.: Toward end-to-end control for uav autonomous landing via deep reinforcement learning. In: International Conference on Unmanned Aircraft Systems (ICUAS), pp. 115–123. IEEE (2018)

  47. Lee, H., Jung, S., Shim, D.H.: Vision-based UAV landing on the moving vehicle. In: International conference on unmanned aircraft systems (ICUAS), pp. 1–7, IEEE (2016)

  48. Wu, C., Ju, B., Wu, Y., Lin, X., Xiong, N., Xu, G., Li, H., Liang, X.: UAV autonomous target search based on deep reinforcement learning in complex disaster scene. IEEE Access 7, 117227–117245 (2019)

    Article  Google Scholar 

  49. Geraldes, R., Goncalves, A., Lai, T., Villerabel, M., Deng, W., Salta, A., Nakayama, K., Matsuo, Y., Prendinger, H.: UAV-based situational awareness system using deep learning. IEEE Access 7, 122583–122594 (2019)

    Article  Google Scholar 

  50. Zhu, Y., Mottaghi, R., Kolve, E., Lim, J.J., Gupta, A., Fei-Fei, L., Farhadi, A.: Target-driven visual navigation in indoor scenes using deep reinforcement learning. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 3357–3364 (2017)

  51. Wang, C., Wang, J., Shen, Y., Zhang, X.: Autonomous navigation of UAVS in large-scale complex environments: a deep reinforcement learning approach. IEEE Trans. Veh. Technol. 68(3), 2124–2136 (2019)

    Article  Google Scholar 

  52. Tijtgat, N., Van Ranst, W., Goedeme, T., Volckaert, B., De Turck, F.: Embedded real-time object detection for a UAV warning system. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 2110–2118 (2017)

  53. Corp, N.: Gpus from nvidia. https://www.nvidia.com/

  54. Kwan, C., Chou, B., Echavarren, A., Budavari, B., Li, J., Tran, T.: Compressive vehicle tracking using deep learnin. In: 9th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON), pp. 51–56 (2018)

  55. Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), vol. 2, pp. 246–252 (1999)

  56. Bertinetto, L., Valmadre, J., Golodetz, S., Miksik, O., Torr, P.H.: Staple: Complementary learners for real-time tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1401–1409 (2016)

  57. Qi, Y., Zhang, S., Qin, L., Yao, H., Huang, Q., Lim, J., Yang, M.-H.: Hedged deep tracking. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4303–4311 (2016)

  58. El-Shafie, A.-H.A., Zaki, M., Habib, S.E.-D.: Fast cnn-based object tracking using localization layers and deep features interpolation. In: 15th International Wireless Communications & Mobile Computing Conference (IWCMC), pp. 1476–1481 (2019)

  59. Inspire, D.: Dji inspire. https://www.dji.com/inspire-2/info#specs

  60. Dutta, A., Zisserman, A.: The via annotation software for images, audio and video. In: Proceedings of the 27th ACM International Conference on Multimedia, pp. 2276–2279 (2019)

  61. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)

  62. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C.: Ssd: Single shot multibox detector. In: European Conference on Computer Vision, pp. 21–37. Springer, New York (2016)

  63. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

  64. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

  65. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprintarXiv: 1409.1556 (2014)

  66. Szegedy, C., Reed, S., Erhan, D., Anguelov, D., Ioffe, S.: Scalable, high-quality object detection. arXiv preprintarXiv: 1412.1441 (2014)

  67. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.04861 (2017)

  68. Rohling, H.: Radar cfar thresholding in clutter and multiple target situations. IEEE Trans. Aerosp. Electron. Syst. 4, 608–621 (1983)

    Article  Google Scholar 

  69. Li, F., Zhang, B., Liu, B.: Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016)

  70. Mueller, M., Smith, N., Ghanem, B.: A benchmark and simulator for UAV tracking. In: European Conference on Computer Vision, pp. 445–461. Springer, New York (2016)

  71. U. C. for Research in Computer Vision: Ucf aerial action data. https://www.crcv.ucf.edu/data/UCF_Aerial Action.php (2009)

  72. Maggiori, E., Tarabalka, Y., Charpiat, G., Alliez, P.: Can semantic labeling methods generalize to any city? the inria aerial image labeling benchmark. In: IEEE International Geoscience and Remote Sensing Symposium (IGARSS), pp. 3226–3229 (2017)

  73. Robicquet, A., Sadeghian, A., Alahi, A., Savarese, S.: Learning social etiquette: Human trajectory understanding in crowded scenes. In: European Conference on Computer Vision, pp. 549–565. Springer, New York(2016)

  74. Choi, Y., Kim, N., Hwang, S., Park, K., Yoon, J.S., An, K., Kweon, I.S.: Kaist multi-spectral day/night data set for autonomous and assisted driving. IEEE Trans. Intell. Transport. Syst. 19(3), 934–948 (2018)

    Article  Google Scholar 

  75. Kristan, M., Leonardis, A., Matas, J., Felsberg, M., Pflugfelder, R., Kämäräinen, J.-K., Danelljan, M., Zajc, L. Č., Lukežič, A., Drbohlav, O., et al.: The eighth visual object tracking vot2020 challenge results. In: European Conference on Computer Vision, pp. 547–601. Springer, New York (2020)

  76. Davis, J.W., Keck, M.A.: A two-stage template approach to person detection in thermal imagery. In: 2005 Seventh IEEE Workshops on Applications of Computer Vision (WACV/MOTION’05)-Volume 1, vol. 1, pp. 364–369. IEEE (2005)

Download references

Author information

Authors and Affiliations

  1. Indian Institute of Technology Hyderabad, Hyderabad, India

    V. Chandrakanth, V. S. N. Murthy & Sumohana S. Channappayya

Authors
  1. V. Chandrakanth
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. V. S. N. Murthy
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sumohana S. Channappayya
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to V. Chandrakanth.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (MP4 18476 KB)

Supplementary file2 (MP4 39334 KB)

Supplementary file3 (MP4 275680 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chandrakanth, V., Murthy, V.S.N. & Channappayya, S.S. UAV-based autonomous detection and tracking of beyond visual range (BVR) non-stationary targets using deep learning. J Real-Time Image Proc 19, 345–361 (2022). https://doi.org/10.1007/s11554-021-01185-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11554-021-01185-w

Keywords

Search

Navigation