Skip to main content
Springer Nature Link
Log in

Learning Agile, Vision-Based Drone Flight: From Simulation to Reality

  • Conference paper
  • First Online:
Robotics Research (ISRR 2022)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 27))

Included in the following conference series:

Abstract

We present our latest research in learning deep sensorimotor policies for agile, vision-based quadrotor flight. We show methodologies for the successful transfer of such policies from simulation to the real world. In addition, we discuss the open research questions that still need to be answered to improve the agility and robustness of autonomous drones toward human-pilot performance.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    A video of the results can be found here: https://youtu.be/m89bNn6RFoQ.

  2. 2.

    A video of the results can be found here: https://youtu.be/2N_wKXQ6MXA.

  3. 3.

    A video of the results can be found here: https://youtu.be/DGjwm5PZQT8.

References

  1. Cieslewski, T., Kaufmann, E., Scaramuzza, D.: Rapid exploration with multi-rotors: a frontier selection method for high speed flight. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2135–2142 (2017)

    Google Scholar 

  2. Ryll, M., Ware, J., Carter, J., Roy, N.: Efficient trajectory planning for high speed flight in unknown environments. In: International Conference on Robotics and Automation (ICRA), pp. 732–738. IEEE (2019)

    Google Scholar 

  3. Mahony, R., Kumar, V., Corke, P.: Multirotor aerial vehicles: modeling, estimation, and control of quadrotor. IEEE Robot. Autom. Mag. 19(3), 20–3 (2012)

    Article  Google Scholar 

  4. Shen, S., Mulgaonkar, Y., Michael, N., Kumar, V.: Vision-based state estimation and trajectory control towards high-speed flight with a quadrotor. In: Robotics: Science and Systems (RSS) (2013)

    Google Scholar 

  5. Falanga, D., Mueggler, E., Faessler, M., Scaramuzza, D.: Aggressive quadrotor flight through narrow gaps with onboard sensing and computing using active vision. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 5774–5781 (2017)

    Google Scholar 

  6. Faessler, M., Franchi, A., Scaramuzza, D.: Differential flatness of quadrotor dynamics subject to rotor drag for accurate tracking of high-speed trajectories. IEEE Robot. Autom. Lett. 3(2), 620–626 (2018)

    Article  Google Scholar 

  7. Sun, S., Sijbers, L., Wang, X., de Visser, C.: High-speed flight of quadrotor despite loss of single rotor. IEEE Robot. Autom. Lett. 3(4), 3201–3207 (2018)

    Article  Google Scholar 

  8. Falanga, D., Foehn, P., Lu, P., Scaramuzza, D.: PAMPC: perception-aware model predictive control for quadrotors. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2018)

    Google Scholar 

  9. Nan, F., Sun, S., Foehn, P., Scaramuzza, D.: Nonlinear MPC for quadrotor fault-tolerant control. IEEE Robot. Autom. Lett. 7(2), 5047–5054 (2022)

    Article  Google Scholar 

  10. Foehn, P., Romero, A., Scaramuzza, D.: Time-optimal planning for quadrotor waypoint flight. Sci. Robot. 6(56), eabh1221 (2021). https://robotics.sciencemag.org/content/6/56/eabh1221

  11. Loquercio, A., Kaufmann, E., Ranftl, R., Müller, M., Koltun, V., Scaramuzza, D.: Learning high-speed flight in the wild. Sci. Robot. 6(59), eabg5810 (2021)

    Article  Google Scholar 

  12. Müller, M., Dosovitskiy, A., Ghanem, B., Koltun, V.: Driving policy transfer via modularity and abstraction. In: Conference on Robot Learning, pp. 1–15 (2018)

    Google Scholar 

  13. Kaufmann, E. Loquercio, A., Ranftl, R., Müller, M., Koltun, V., Scaramuzza, D.: Deep drone acrobatics. In: Robotics: Science and Systems (RSS) (2020)

    Google Scholar 

  14. Kaufmann, E., Loquercio, A., Ranftl, R., Dosovitskiy, A., Koltun, V., Scaramuzza, D.: Deep drone racing: learning agile flight in dynamic environments. In: Conference on Robotics Learning (CoRL) (2018)

    Google Scholar 

  15. Kaufmann, E., et al.: Beauty and the beast: optimal methods meet learning for drone racing. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 690–696 (2019). https://doi.org/10.1109/ICRA.2019.8793631

  16. Loquercio, A., Kaufmann, E., Ranftl, R., Dosovitskiy, A., Koltun, V., Scaramuzza, D.: Deep drone racing: from simulation to reality with domain randomization. IEEE Trans. Robot. 36(1), 1–14 (2019)

    Article  Google Scholar 

  17. Foehn, P., et al.: Alphapilot: autonomous drone racing. In: Robotics: Science and System (RSS) (2022). https://link.springer.com/article/10.1007/s11370-018-00271-6

  18. Haarnoja, T., Hartikainen, K., Abbeel, P., Levine, S.: Latent space policies for hierarchical reinforcement learning. In: International Conference on Machine Learning, pp. 1851–1860. PMLR (2018)

    Google Scholar 

  19. Peng, X. B., Coumans, E., Zhang, T., Lee, T.W., Tan, J., Levine, S.: Learning agile robotic locomotion skills by imitating animals. In: Robotics: Science and Systems (RSS) (2020)

    Google Scholar 

  20. Smith, L., Kew, J.C., Peng, X.B., Ha, S., Tan, J., Levine, S.: Legged robots that keep on learning: fine-tuning locomotion policies in the real world. arXiv preprint arXiv:2110.05457 (2021)

  21. Kumar, A., Fu, Z., Pathak, D., Malik, J.: RMA: rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034 (2021)

  22. Andrychowicz, M., et al.: Learning to learn by gradient descent by gradient descent. In: Advances in Neural Information Processing Systems, vol. 29 (2016)

    Google Scholar 

  23. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International Conference on Machine Learning, pp. 1126–1135. PMLR (2017)

    Google Scholar 

  24. Spica, R., Falanga, D., Cristofalo, E., Montijano, E., Scaramuzza, D., Schwager, M.: A real-time game theoretic planner for autonomous two-player drone racing. In: Robotics: Science and Systems (RSS) (2018)

    Google Scholar 

  25. Baker, B., et al.: Emergent tool use from multi-agent autocurricula. arXiv preprint arXiv:1909.07528 (2019)

  26. Silver, D., et al.: Mastering the game of go without human knowledge. Nature 550(7676), 354–359 (2017)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the National Centre of Competence in Research (NCCR) Robotics through the Swiss National Science Foundation (SNSF) and the European Union’s Horizon 2020 Research and Innovatifon Program under grant agreement No. 871479 (AERIAL-CORE) and the European Research Council (ERC) under grant agreement No. 864042 (AGILEFLIGHT).

Author information

Authors and Affiliations

  1. University of Zurich, Zürich, Switzerland

    Davide Scaramuzza & Elia Kaufmann

Authors
  1. Davide Scaramuzza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Elia Kaufmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Davide Scaramuzza .

Editor information

Editors and Affiliations

  1. EPFL STI SMT-GE, Ecole Polytechnique Federale de Lausanne, Lausanne, Vaud, Switzerland

    Aude Billard

  2. Institute for Anthropomatics and Robotic, KIT, Karlsruhe, Baden-Württemberg, Germany

    Tamim Asfour

  3. Artificial Intelligence Laboratory, Department of Computer Science, Stanford University, Stanford, CA, USA

    Oussama Khatib

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Scaramuzza, D., Kaufmann, E. (2023). Learning Agile, Vision-Based Drone Flight: From Simulation to Reality. In: Billard, A., Asfour, T., Khatib, O. (eds) Robotics Research. ISRR 2022. Springer Proceedings in Advanced Robotics, vol 27. Springer, Cham. https://doi.org/10.1007/978-3-031-25555-7_2

Download citation

Publish with us

Policies and ethics