Skip to main content
Springer Nature Link
Log in

Autonomous Drone Cinematographer: Using Artistic Principles to Create Smooth, Safe, Occlusion-Free Trajectories for Aerial Filming

  • Conference paper
  • First Online:
Proceedings of the 2018 International Symposium on Experimental Robotics (ISER 2018)

Part of the book series: Springer Proceedings in Advanced Robotics ((SPAR,volume 11))

Included in the following conference series:

  • 2369 Accesses

Abstract

Autonomous aerial cinematography has the potential to enable automatic capture of aesthetically pleasing videos without requiring human intervention, empowering individuals with the capability of high-end film studios. Current approaches either only handle off-line trajectory generation, or offer strategies that reason over short time horizons and simplistic representations for obstacles, which result in jerky movement and low real-life applicability. In this work we develop a method for aerial filming that is able to trade off shot smoothness, occlusion, and cinematography guidelines in a principled manner, even under noisy actor predictions. We present a novel algorithm for real-time covariant gradient descent that we use to efficiently find the desired trajectories by optimizing a set of cost functions. Experimental results show that our approach creates attractive shots, avoiding obstacles and occlusion 65 times over 1.25 h of flight time, re-planning at 5 Hz with a 10 s time horizon. We robustly film human actors, cars and bicycles performing different motion among obstacles, using various shot types.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Arijon, D.: Grammar of the Film Language (1976)

    Google Scholar 

  2. Barry, A.J., Florence, P.R., Tedrake, R.: High-speed autonomous obstacle avoidance with pushbroom stereo. J. Field Robot. 35(1), 52–68 (2018)

    Article  Google Scholar 

  3. Bowen, C.J., Thompson, R.: Grammar of the Shot. Taylor & Francis (2013)

    Google Scholar 

  4. Christie, M., Olivier, P., Normand, J.-M.: Camera control in computer graphics. In: Computer Graphics Forum, vol. 27, pp. 2197–2218. Wiley (2008)

    Google Scholar 

  5. Drucker, S.M., Zeltzer, D.: Intelligent camera control in a virtual environment. In: Graphics Interface, pp. 190–190. Citeseer (1994)

    Google Scholar 

  6. Galvane, Q., Fleureau, J., Tariolle, F.-L., Guillotel, P.: Automated cinematography with unmanned aerial vehicles. arXiv preprint arXiv:1712.04353 (2017)

  7. Galvane, Q., Lino, C., Christie, M., Fleureau, J., Servant, F., Guillotel, P.: Directing cinematographic drones. arXiv preprint arXiv:1712.04216 (2017)

  8. Gebhardt, C., Hepp, B., Nägeli, T., Stevšić, S., Hilliges, O.: Airways: optimization-based planning of quadrotor trajectories according to high-level user goals. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pp. 2508–2519. ACM (2016)

    Google Scholar 

  9. Gebhardt, C., Stevsic, S., Hilliges, O.: Optimizing for aesthetically pleasing quadrotor camera motion (2018)

    Google Scholar 

  10. Gleicher, M., Witkin, A.: Through-the-lens camera control. In: ACM SIGGRAPH Computer Graphics, vol. 26, pp. 331–340. ACM (1992)

    Google Scholar 

  11. Henriques, J.F., Caseiro, R., Martins, P., Batista, J.: High-speed tracking with kernelized correlation filters. IEEE Trans. Pattern Anal. Mach. Intell. 37(3), 583–596 (2015)

    Article  Google Scholar 

  12. Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 (2017)

  13. Joubert, N., Roberts, M., Truong, A., Berthouzoz, F., Hanrahan, P.: An interactive tool for designing quadrotor camera shots. ACM Trans. Graph. (TOG) 34(6), 238 (2015)

    Article  Google Scholar 

  14. Joubert, N., Goldman, D.B., Berthouzoz, F., Roberts, M., Landay, J.A., Hanrahan, P., et al.: Towards a drone cinematographer: guiding quadrotor cameras using visual composition principles. arXiv preprint arXiv:1610.01691 (2016)

  15. Lan, Z., Shridhar, M., Hsu, D., Zhao, S.: Xpose: reinventing user interaction with flying cameras. In: Robotics Science and Systems (2017)

    Google Scholar 

  16. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: common objects in context. In: European Conference on Computer Vision, pp. 740–755. Springer, Heidelberg (2014)

    Chapter  Google Scholar 

  17. Lino, C., Christie, M.: Intuitive and efficient camera control with the toric space. ACM Trans. Graph. (TOG) 34(4), 82 (2015)

    Article  Google Scholar 

  18. Lino, C., Christie, M., Ranon, R., Bares, W.: The director’s lens: an intelligent assistant for virtual cinematography. In: Proceedings of the 19th ACM International Conference on Multimedia, pp. 323–332. ACM (2011)

    Google Scholar 

  19. Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.Y. and Berg, A.C.: SSD: single shot multibox detector. In: European Conference on Computer Vision, pp. 21–37. Springer, Heidelberg (2016)

    Chapter  Google Scholar 

  20. Mellinger, D., Kumar, V.: Minimum snap trajectory generation and control for quadrotors. In: IEEE International Conference on Robotics and Automation (ICRA), pp. 2520–2525. IEEE (2011)

    Google Scholar 

  21. Mohta, K., Watterson, M., Mulgaonkar, Y., Liu, S., Chao, Q., Makineni, A., Saulnier, K., Sun, K., Zhu, A., Delmerico, J., et al.: Fast, autonomous flight in GPS-denied and cluttered environments. J. Field Robot. 35(1), 101–120 (2018)

    Article  Google Scholar 

  22. Nägeli, T., Meier, L., Domahidi, A., Alonso-Mora, J., Hilliges, O.: Real-time planning for automated multi-view drone cinematography. ACM Trans. Graph. (TOG) 36(4), 132 (2017)

    Article  Google Scholar 

  23. Newcombe, R.A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A.J., Kohli, P., Shotton, J., Hodges, S., Fitzgibbon, A.W.: Kinectfusion: real-time dense surface mapping and tracking. In: 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136. IEEE (2011)

    Google Scholar 

  24. Ratliff, N., Zucker, M., Bagnell, J.A., Srinivasa, S.: CHOMP: gradient optimization techniques for efficient motion planning. In: IEEE International Conference on Robotics and Automation, ICRA 2009, pp. 489–494. IEEE (2009)

    Google Scholar 

  25. Ratliff, N.D., Silver, D., Bagnell, J.A.: Learning to search: functional gradient techniques for imitation learning. Auton. Robot. 27(1), 25–53 (2009)

    Article  Google Scholar 

  26. Redmon, J., Farhadi, A.: YOLO9000: better, faster, stronger. arXiv preprint (2017)

    Google Scholar 

  27. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: Advances in Neural Information Processing Systems, pp. 91–99 (2015)

    Google Scholar 

  28. Roberts, M., Hanrahan, P.: Generating dynamically feasible trajectories for quadrotor cameras. ACM Trans. Graph. (TOG) 35(4), 61 (2016)

    Article  Google Scholar 

  29. Schulman, J., Ho, J., Lee, A.X., Awwal, I., Bradlow, H., Abbeel, P.: Finding locally optimal, collision-free trajectories with sequential convex optimization. In: Robotics: Science and Systems, vol. 9, pp. 1–10. Citeseer (2013)

    Google Scholar 

  30. Shah, S., Dey, D., Lovett, C., Kapoor, A.: AirSim: high-fidelity visual and physical simulation for autonomous vehicles (2017). https://arxiv.org/abs/1705.05065

    Google Scholar 

  31. Shim, D.H., Kim, H.J., Sastry, S.: Decentralized nonlinear model predictive control of multiple flying robots. In: Proceedings of 42nd IEEE Conference on Decision and control, vol. 4, pp. 3621–3626. IEEE (2003)

    Google Scholar 

  32. Turpin, M., Michael, N., Kumar, V.: Trajectory design and control for aggressive formation flight with quadrotors. Auton. Robot. 33(1–2), 143–156 (2012)

    Article  Google Scholar 

  33. Xie, K., Yang, H., Huang, S., Lischinski, D., Christie, M., Kai, X., Gong, M., Cohen-Or, D., Huang, H.: Creating and chaining camera moves for quadrotor videography. ACM Trans. Graph. 37, 14 (2018)

    Google Scholar 

  34. Zucker, M., Ratliff, N., Dragan, A.D., Pivtoraiko, M., Klingensmith, M., Dellin, C.M., Bagnell, J.A., Srinivasa, S.S.: CHOMP: covariant Hamiltonian optimization for motion planning. Int. J. Robot. Res. 32(9–10), 1164–1193 (2013)

    Article  Google Scholar 

Download references

Acknowledgements

We thank Lentin Joseph, Aayush Ahuja, Delong Zhu, and Greg Armstrong for the assistance in field experiments and robot construction. Research presented in this paper was funded by Yamaha Motor Co., Ltd.

Author information

Authors and Affiliations

  1. Robotics Institute, Carnegie Mellon University, Pittsburgh, PA, USA

    Rogerio Bonatti, Sanjiban Choudhury, Wenshan Wang & Sebastian Scherer

  2. Advanced Technology Research Division, Yamaha Motor Co., Ltd., Shizuoka, Japan

    Yanfu Zhang

Authors
  1. Rogerio Bonatti
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yanfu Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanjiban Choudhury
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wenshan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Sebastian Scherer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rogerio Bonatti .

Editor information

Editors and Affiliations

  1. Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, USA

    Jing Xiao

  2. Karlsruhe Institute of Technology, Karlsruhe, Baden-Württemberg, Germany

    Torsten Kröger

  3. Department of Computer Science, Stanford University, Stanford, CA, USA

    Oussama Khatib

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 33922 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bonatti, R., Zhang, Y., Choudhury, S., Wang, W., Scherer, S. (2020). Autonomous Drone Cinematographer: Using Artistic Principles to Create Smooth, Safe, Occlusion-Free Trajectories for Aerial Filming. In: Xiao, J., Kröger, T., Khatib, O. (eds) Proceedings of the 2018 International Symposium on Experimental Robotics. ISER 2018. Springer Proceedings in Advanced Robotics, vol 11. Springer, Cham. https://doi.org/10.1007/978-3-030-33950-0_11

Download citation

Publish with us

Policies and ethics