Skip to main content
SpringerLink
Log in

Fully Onboard Single Pedestrian Tracking on Nano-UAV Platform

  • Short Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Nano-sized unmanned aerial vehicles (UAVs) have been attracting growing attention from both academia and industry with their compact and versatile nature. With the size of merely several centimeters, they are particularly well-suited for executing targeted missions in confined spaces, such as tracking a single pedestrian indoors. However, due to strict constraints on payload and power consumption, the onboard computing unit of nano-UAV is limited to the microcontroller Unit (MCU), which makes the novel single object tracking algorithms not applicable on nano-UAV platform. In this paper, we present a lightweight single pedestrian tracking algorithm to meet the strict resource constraints of nano-UAV system. The algorithm consists of an object detection frontend and a motion-based tracking backend. Based on our deployment methodology, covering from dataset production, model training to layer fusion and deployment, the proposed algorithm achieves extreme complexity reduction (13\(\times \) fewer operations and 106\(\times \) less memory), and keeps good tracking performance compared with state-of-the-art (SOTA) research work. The full range of onboard experiments illustrate the efficiency our algorithm for real-time tracking (up to 43 fps). Meanwhile, both high computing efficiency (5.2 MACs/cycle) and energy efficiency (5.3 mJ/frame) are achieved and exceed similar works on nano-UAV.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Code Availability

Not applicable

References

  1. Liu, Y., Meng, Z., Zou, Y., Cao, M.: Visual object tracking and servoing control of a nano-scale quadrotor: system, algorithms, and experiments. IEEE CAA J. Autom. Sinica 8(2), 344–360 (2021)

    Article  MathSciNet  Google Scholar 

  2. Niculescu, V., Lamberti, L., Conti, F., Benini, L., Palossi, D.: Improving autonomous nano-drones performance via automated end-to-end optimization and deployment of dnns. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 11(4), 548–562 (2021)

    Article  Google Scholar 

  3. Palossi, D., Zimmerman, N., Burrello, A., Conti, F., Müller, H., Gambardella, L.M., Benini, L., Giusti, A., Guzzi, J.: Fully onboard ai-powered human-drone pose estimation on ultra-low power autonomous flying nano-uavs. IEEE Internet of Things Journal (2021)

  4. Palossi, D., Conti, F., Benini, L.: An open source and open hardware deep learning-powered visual navigation engine for autonomous nano-uavs. In: 2019 15th International conference on Distributed Computing in Sensor Systems (DCOSS), pp. 604–611 (2019)

  5. Palossi, D., Loquercio, A., Conti, F., Flamand, E., Scaramuzza, D., Benini, L.: A 64-mw dnn-based visual navigation engine for autonomous nano-drones. IEEE Internet Things J. 6(5), 8357–8371 (2019)

    Article  Google Scholar 

  6. Palossi, D., Gomez, A., Draskovic, S., Marongiu, A., Thiele, L., Benini, L.: Extending the lifetime of nano-blimps via dynamic motor control. Journal of Signal Processing Systems 91(3), 339–361 (2019)

    Article  Google Scholar 

  7. Kang, K., Belkhale, S., Kahn, G., Abbeel, P., Levine, S.: Generalization through simulation: integrating simulated and real data into deep reinforcement learning for vision-based autonomous flight. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 6008–6014 (2019)

  8. Srisamosorn, V., Kuwahara, N., Yamashita, A., Ogata, T., Ota, J.: Human-tracking system using quadrotors and multiple environmental cameras for face-tracking application. Int. J. Adv. Rob. Syst. 14(5), 1729881417727357 (2017)

    Google Scholar 

  9. Palossi, D., Singh, J., Magno, M., Benini, L.: Target following on nano-scale unmanned aerial vehicles. In: 2017 7th IEEE International Workshop on Advances in Sensors and Interfaces (IWASI), pp. 170–175 (2017)

  10. Zhang, X., Xian, B., Zhao, B., Zhang, Y.: Autonomous flight control of a nano quadrotor helicopter in a gps-denied environment using on-board vision. IEEE Trans. Industr. Electron. 62(10), 6392–6403 (2015)

    Article  Google Scholar 

  11. Cai, G., Dias, J., Seneviratne, L.: A survey of small-scale unmanned aerial vehicles: recent advances and future development trends. Unmanned Systems 2(02), 175–199 (2014)

    Article  Google Scholar 

  12. Briod, A., Zufferey, J.-C., Floreano, D.: Optic-flow based control of a 46g quadrotor. In: Workshop on vision-based closed-loop control and navigation of micro helicopters in GPS-denied environments, IROS 2013. Workshop on vision-based closed-loop control and navigation of micro helicopters in GPS-denied environments, IROS 2013 (2013). CONF

  13. Shakhatreh, H., Sawalmeh, A.H., Al-Fuqaha, A., Dou, Z., Almaita, E., Khalil, I., Othman, N.S., Khreishah, A., Guizani, M.: Unmanned aerial vehicles (uavs): a survey on civil applications and key research challenges. Ieee Access 7, 48572–48634 (2019)

    Article  Google Scholar 

  14. Motlagh, N.H., Taleb, T., Arouk, O.: Low-altitude unmanned aerial vehicles-based internet of things services: comprehensive survey and future perspectives. IEEE Internet Things J. 3(6), 899–922 (2016)

    Article  Google Scholar 

  15. Lo, L.-Y., Yiu, C.H., Tang, Y., Yang, A.-S., Li, B., Wen, C.-Y.: Dynamic object tracking on autonomous uav system for surveillance applications. Sensors 21(23), 7888 (2021)

    Article  Google Scholar 

  16. Rodriguez-Ramos, A., Alvarez-Fernandez, A., Bavle, H., Campoy, P., How, J.P.: Vision-based multirotor following using synthetic learning techniques. Sensors 19(21), 4794 (2019)

    Article  Google Scholar 

  17. Cheng, H., Lin, L., Zheng, Z., Guan, Y., Liu, Z.: An autonomous vision-based target tracking system for rotorcraft unmanned aerial vehicles. In: 2017 IEEE/RSJ International conference on Intelligent Robots and Systems (IROS), pp. 1732–1738 (2017)

  18. Floreano, D., Wood, R.J.: Science, technology and the future of small autonomous drones. Nature 521(7553), 460–466 (2015)

    Article  Google Scholar 

  19. Zhang, Y., Sun, P., Jiang, Y., Yu, D., Yuan, Z., Luo, P., Liu, W., Wang, X.: Bytetrack: multi-object tracking by associating every detection box. arXiv preprint arXiv:2110.06864 (2021)

  20. Li, R., Pang, M., Zhao, C., Zhou, G., Fang, L.: Monocular long-term target following on uavs. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 29–37 (2016)

  21. Bochkovskiy, A., Wang, C.-Y., Liao, H.-Y.M.: Yolov4: optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 (2020)

  22. Conti, F., Palossi, D., Marongiu, A., Rossi, D., Benini, L.: Enabling the heterogeneous accelerator model on ultra-low power microcontroller platforms. In: 2016 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1201–1206 (2016)

  23. Gautschi, M., Schiavone, P.D., Traber, A., Loi, I., Pullini, A., Rossi, D., Flamand, E., Gürkaynak, F.K., Benini, L.: Near-threshold risc-v core with dsp extensions for scalable iot endpoint devices. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 25(10), 2700–2713 (2017)

    Article  Google Scholar 

  24. Pullini, A., Rossi, D., Haugou, G., Benini, L.: \(\mu \)dma: an autonomous i/o subsystem for iot end-nodes. In: 2017 27th International symposium on Power and Timing Modeling, Optimization and Simulation (PATMOS), pp. 1–8 (2017)

  25. Li, B., Wu, W., Wang, Q., Zhang, F., Xing, J., Yan, J.: Siamrpn++: Evolution of siamese visual tracking with very deep networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4282–4291 (2019)

  26. Wang, Q., Zhang, L., Bertinetto, L., Hu, W., Torr, P.H.: Fast online object tracking and segmentation: a unifying approach. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1328–1338 (2019)

  27. Bolme, D.S., Beveridge, J.R., Draper, B.A., Lui, Y.M.: Visual object tracking using adaptive correlation filters. In: 2010 IEEE Computer society conference on computer vision and pattern recognition, pp. 2544–2550 (2010)

  28. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2014)

    Article  Google Scholar 

  29. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: Exploiting the circulant structure of tracking-by-detection with kernels. In: European conference on computer vision, pp. 702–715 (2012)

  30. Danelljan, M., Häger, G., Khan, F.S., Felsberg, M.: Discriminative scale space tracking. IEEE Trans. Pattern Anal. Mach. Intell. 39(8), 1561–1575 (2016)

    Article  Google Scholar 

  31. Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High performance visual tracking with siamese region proposal network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8971–8980 (2018)

  32. Zhou, X., Wang, D., Krähenbühl P.: Objects as points. arXiv preprint arXiv:1904.07850 (2019)

  33. Zhang, X., Zhou, X., Lin, M., Sun, J.: Shufflenet: an extremely efficient convolutional neural network for mobile devices. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848–6856 (2018)

  34. Redmon, J., Farhadi, A.: Yolov3: an incremental improvement. arXiv preprint arXiv:1804.02767 (2018)

  35. Kuhn, H.W.: The hungarian method for the assignment problem. Naval research logistics quarterly 2(1–2), 83–97 (1955)

    Article  MathSciNet  MATH  Google Scholar 

  36. Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. Int. J. Comput. Vision 88(2), 303–338 (2010)

    Article  Google Scholar 

  37. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European conference on computer vision, pp. 740–755 (2014)

  38. Wu, Y., Lim, J., Yang, M.-H.: Online object tracking: a benchmark. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2411–2418 (2013)

Download references

Acknowledgements

This work is supported by the Qiyuan Laboratory and the National Natural Science Foundation of China (62171156).

Funding

This work is supported by the National Natural Science Foundation of China (62171156).

Author information

Authors and Affiliations

  1. School of Electronics and Information Engineering, Harbin Institute of Technology, Yikuang Street, Harbin, 150001, Heilongjiang Province, China

    Haolin Chen, Ruidong Wu & Bing Liu

  2. Qiyuan Laboratory, 60, Yinhua Rd, Beijing, 100095, China

    Wenshuai Lu, Xinglong Ji, Tao Wang, Haolun Ding & Yuxiang Dai

Authors
  1. Haolin Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ruidong Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenshuai Lu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xinglong Ji
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Tao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Haolun Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yuxiang Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Bing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Haolin Chen, Ruidong Wu, Wenshuai Lu, Xinglong Ji, Tao Wang, Haolun Ding, Yuxiang Dai and Bing Liu designed the study. Haolin Chen, Ruidong Wu, Tao Wang and Haolun Ding designed the tracking algorithm. Haolin Chen, Wenshuai Lu and Xinglong Ji designed the development methodology. Haolin Chen, Xionglong Ji, Tao Wang and Bing Liu designed the dataset and the experiment. Haolin Chen and Ruidong Wu coordinated the experiment and analyzed the data. All authors wrote the final manuscript.

Corresponding author

Correspondence to Bing Liu.

Ethics declarations

Ethics approval

Not applicable as this study does not contain biological applications.

Consent for participate

All authors of this research paper have consented to participate in the research study.

Consent for publication

All authors and involved researchers give their consent for the publication of the present paper, including photographs and video, to be published in the Journal of Intelligent and Robotic Systems.

Conflicts of interest

Not applicable

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Wu, R., Lu, W. et al. Fully Onboard Single Pedestrian Tracking on Nano-UAV Platform. J Intell Robot Syst 109, 50 (2023). https://doi.org/10.1007/s10846-023-01979-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01979-z

Keywords

Search

Navigation