Skip to main content

Fusing Multi-sensor Input with State Information on TinyML Brains for Autonomous Nano-drones

  • Conference paper
  • First Online:
European Robotics Forum 2024 (ERF 2024)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 32))

Included in the following conference series:

  • 65 Accesses

Abstract

Autonomous nano-drones (\(\sim \)10 cm in diameter), thanks to their ultra-low power TinyML-based brains, are capable of coping with real-world environments. However, due to their simplified sensors and compute units, they are still far from the sense-and-act capabilities shown in their bigger counterparts. This system paper presents a novel deep learning-based pipeline that fuses multi-sensorial input (i.e., low-resolution images and \(8\times 8\) depth map) with the robot’s state information to tackle a human pose estimation task. Thanks to our design, the proposed system – trained in simulation and tested on a real-world dataset – improves a state-unaware State-of-the-Art baseline by increasing the \(R^2\) regression metric up to 0.10 on the distance’s prediction.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Clark, R., et al.: VINet: visual-inertial odometry as a sequence-to-sequence learning problem. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)

    Google Scholar 

  2. Crupi, L., et al.: Sim-to-real vision-depth fusion CNNs for robust pose estimation aboard autonomous nano-quadcopters. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7711–7717 (2023)

    Google Scholar 

  3. Han, L., et al.: DeepVIO: self-supervised deep learning of monocular visual inertial odometry using 3D geometric constraints. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6906–6913 (2019)

    Google Scholar 

  4. Kaufmann, E., et al.: Beauty and the beast: optimal methods meet learning for drone racing. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 690–696. IEEE (2019)

    Google Scholar 

  5. Liu, C., et al.: Nano quadcopter obstacle avoidance with a lightweight monocular depth network. IFAC-PapersOnLine 56(2), 9312–9317 (2023). 22nd IFAC World Congress

    Google Scholar 

  6. Palossi, D., et al.: Self-sustainability in nano unmanned aerial vehicles: a blimp case study. In: Proceedings of the Computing Frontiers Conference, pp. 79–88 (2017)

    Google Scholar 

  7. Palossi, D., et al.: Ultra low-power visual odometry for nano-scale unmanned aerial vehicles. In: Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 1647–1650 (2017)

    Google Scholar 

  8. Shakhatreh, H., et al.: Unmanned Aerial Vehicles (UAVs): a survey on civil applications and key research challenges. IEEE Access: Pract. Innov. Open Solutions 7, 48572–48634 (2019)

    Article  MATH  Google Scholar 

  9. Zhang, N., et al.: End-to-end nano-drone obstacle avoidance for indoor exploration. Drones 8(2), 33 (2024)

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Luca Crupi .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Crupi, L., Cereda, E., Palossi, D. (2024). Fusing Multi-sensor Input with State Information on TinyML Brains for Autonomous Nano-drones. In: Secchi, C., Marconi, L. (eds) European Robotics Forum 2024. ERF 2024. Springer Proceedings in Advanced Robotics, vol 32. Springer, Cham. https://doi.org/10.1007/978-3-031-76424-0_53

Download citation

Publish with us

Policies and ethics