skip to main content
research-article

Real-time planning for automated multi-view drone cinematography

Published: 20 July 2017 Publication History

Abstract

We propose a method for automated aerial videography in dynamic and cluttered environments. An online receding horizon optimization formulation facilitates the planning process for novices and experts alike. The algorithm takes high-level plans as input, which we dub virtual rails, alongside interactively defined aesthetic framing objectives and jointly solves for 3D quadcopter motion plans and associated velocities. The method generates control inputs subject to constraints of a non-linear quadrotor model and dynamic constraints imposed by actors moving in an a priori unknown way. The output plans are physically feasible, for the horizon length, and we apply the resulting control inputs directly at each time-step, without requiring a separate trajectory tracking algorithm. The online nature of the method enables incorporation of feedback into the planning and control loop, makes the algorithm robust to disturbances. Furthermore, we extend the method to include coordination between multiple drones to enable dynamic multi-view shots, typical for action sequences and live TV coverage. The algorithm runs in real-time on standard hardware and computes motion plans for several drones in the order of milliseconds. Finally, we evaluate the approach qualitatively with a number of challenging shots, involving multiple drones and actors and qualitatively characterize the computational performance experimentally.

Supplementary Material

ZIP File (a132-nageli.zip)
Supplemental files.
MP4 File (papers-0587.mp4)

References

[1]
2015. 3DR Solo. (2015). http://3drobotics.com/solo.
[2]
2015. APM Autopilot Suite. (2015). http://ardupilot.com.
[3]
2015. DJI Ground Station. (2015). http://www.dji.com/product/pc-ground-station.
[4]
2015. Parrot SDK. (2015). http://developer.parrot.com/.
[5]
2015. VC Technology Litchi Tool. (2015). https://flylitchi.com/.
[6]
A Pedro Aguiar, João P Hespanha, and Petar V Kokotović. 2008. Performance limitations in reference tracking and path following for nonlinear systems. Automatica 44, 3 (2008), 598--610.
[7]
Javier Alonso-Mora, Tobias Naegeli, Roland Siegwart, and Paul Beardsley. 2015. Collision Avoidance for Aerial Vehicles in Multi-agent Scenarios. Auton. Robot. (Jan. 2015).
[8]
John T Betts. 2010. Practical methods for optimal control and estimation using nonlinear programming. Vol. 19. Siam.
[9]
Adam Bry, Charles Richter, Abraham Bachrach, and Nicholas Roy. 2015. Aggressive flight of fixed-wing and quadrotor aircraft in dense indoor environments. The International Journal of Robotics Research 34, 7 (2015), 969--1002.
[10]
Yaobin Chen, Stanley Y-P Chien, and ALAN A DESROCHERS. 1992. General structure of time-optimal control of robotic manipulators moving along prescribed paths. Internat. J. Control 56, 4 (1992), 767--782.
[11]
Marc Christie, Éric Languénou, and Laurent Granvilliers. 2002. Modeling Camera Control with Constrained Hypertubes. Springer Berlin Heidelberg, Berlin, Heidelberg, 618--632.
[12]
Marc Christie, Patrick Olivier, and Jean-Marie Normand. 2008. Camera Control in Computer Graphics. Computer Graphics Forum 27, 8 (Dec. 2008), 2197--2218.
[13]
Daniel Arijon. 1976. Grammar of the film language. https://scholar.google.ch/citations?view
[14]
Alexander Domahidi and Juan Jerez. 2016. FORCES Pro: code generation for embedded optimization. (September 2016). https://www.embotech.com/FORCES-Pro.
[15]
Alexander Domahidi, Aldo U Zgraggen, Melanie N Zeilinger, Manfred Morari, and Colin N Jones. 2012. Efficient interior point methods for multistage problems arising in receding horizon control. In Decision and Control, 2008. CDC 2008. 47th IEEE Conference on. IEEE, 668--674.
[16]
Steven M. Drucker and David Zeltzer. 1994. Intelligent Camera Control in a Virtual Environment. In In Proceedings of Graphics Interface '94. 190--199.
[17]
Timm Faulwasser, Benjamin Kern, and Rolf Findeisen. 2009. Model predictive path-following for constrained nonlinear systems. In Decision and Control, 2009 held jointly with the 2009 28th Chinese Control Conference. CDC/CCC 2009. Proceedings of the 48th IEEE Conference on. IEEE, 8642--8647.
[18]
Quentin Galvane, Julien Fleureau, Francois-Louis Tariolle, and Philippe Guillotel. 2016. Automated Cinematography with Unmanned Aerial Vehicles. In Eurographics Workshop on Intelligent Cinematography and Editing. The Eurographics Association.
[19]
Christoph Gebhardt, Benjamin Hepp, Tobias Naegeli, Stefan Stevsic, and Otmar Hilliges. 2016. Airways: Optimization-based Planning of Quadrotor Trajectories according to High-Level User Goals. In SIGCHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA.
[20]
T. Geijtenbeek and N. Pronost. 2012. Interactive Character Animation Using Simulated Physics: A State-of-the-Art Review. Comput. Graph. Forum 31, 8 (Dec. 2012), 2492--2515.
[21]
Michael Gleicher and Andrew Witkin. 1992. Through-the-lens camera control. In ACM SIGGRAPH Computer Graphics, Vol. 26. ACM, 331--340.
[22]
Niels Joubert, L. E. Jane, Dan B. Goldman, Floraine Berthouzoz, Mike Roberts, James A. Landay, and Pat Hanrahan. 2016. Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles. CoRR abs/1610.01691 (2016). http://arxiv.org/abs/1610.01691
[23]
Niels Joubert, Mike Roberts, Anh Truong, Floraine Berthouzoz, and Pat Hanrahan. 2015. An Interactive Tool for Designing Quadrotor Camera Shots. ACM Transactions on Graphics (SIGGRAPH Asia 2015) (2015).
[24]
Sertac Karaman and Emilio Frazzoli. 2011. Sampling-based algorithms for optimal motion planning. International Journal of Robotics Research 30, 7 (June 2011), 846--894.
[25]
Eric C Kerrigan and Jan M Maciejowski. 2000. Soft constraints and exact penalty functions in model predictive control. In Control 2000 Conference, Cambridge.
[26]
Denise Lam, Chris Manzie, and Malcolm Good. 2010. Model predictive contouring control. In 49th IEEE Conference on Decision and Control (CDC). IEEE, 6137--6142.
[27]
Alexander Liniger, Alexander Domahidi, and Manfred Morari. 2015. Optimization-based autonomous racing of 1:43 scale RC cars. Optimal Control Applications and Methods 36, 5 (2015), 628--647.
[28]
C Lino and M Christie. 2015. Intuitive and efficient camera control with the toric space. ACM Transactions on Graphics (TOG) (2015).
[29]
Christophe Lino, Marc Christie, Roberto Ranon, and William Bares. 2011. The Director's Lens: An Intelligent Assistant for Virtual Cinematography. In Proceedings of the 19th ACM International Conference on Multimedia (MM '11). ACM, New York, NY, USA, 323--332.
[30]
Brian MacAllister, Jonathan Butzke, Alex Kushleyev, Harsh Pandey, and Maxim Likhachev. 2013. Path planning for non-circular micro aerial vehicles in constrained environments. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3933--3940.
[31]
Daniel Mellinger and Vijay Kumar. 2011. Minimum snap trajectory generation and control for quadrotors. In IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2520--2525.
[32]
Mark W Mueller and Raffaello D'Andrea. 2013. A model predictive controller for quadrocopter state interception. In Control Conference (ECC), 2013 European. IEEE, 1383--1389.
[33]
Tobias Nägeli, Javier Alonso-Mora, Alexander Domahidi, Daniela Rus, and Otmar Hilliges. 2017. Real-time Motion Planning for Aerial Videography with Dynamic Obstacle Avoidance and Viewpoint Optimization. IEEE Robotics and Automation Letters (2017).
[34]
Morgan Quigley, Ken Conley, Brian P. Gerkey, Josh Faust, Tully Foote, Jeremy Leibs, Rob Wheeler, and Andrew Y. Ng. 2009. ROS: an open-source Robot Operating System. In IEEE ICRA Workshop on Open Source Software.
[35]
Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Transactions on Graphics (Proc. SIGGRAPH 2016) 35, 4 (2016).
[36]
Charles Rose, Brian Guenter, Bobby Bodenheimer, and Michael F Cohen. 1996. Efficient generation of motion transitions using spacetime constraints. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques. ACM, 147--154.
[37]
Jeffery B Saunders, Brandon Call, Andrew Curtis, Randal W Beard, and Timothy W McLain. 2005. Static and dynamic obstacle avoidance in miniature air vehicles. AIAA Infotech@ Aerospace 96 (2005).
[38]
Andrew Witkin and Michael Kass. 1988. Spacetime constraints. ACM Siggraph Computer Graphics 22, 4 (1988), 159--168.

Cited By

View all
  • (2024)Generating Flight Summaries Conforming to Cinematographic PrinciplesComputer Graphics Forum10.1111/cgf.15179Online publication date: 17-Oct-2024
  • (2024)Cinematographic Camera Diffusion ModelComputer Graphics Forum10.1111/cgf.1505543:2Online publication date: 27-Apr-2024
  • (2024)CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene CompositionIEEE Transactions on Robotics10.1109/TRO.2024.335355040(1740-1757)Online publication date: 1-Jan-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 36, Issue 4
August 2017
2155 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3072959
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017
Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. aerial videography
  2. collision-avoidance
  3. multi-drone

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)110
  • Downloads (Last 6 weeks)13
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Generating Flight Summaries Conforming to Cinematographic PrinciplesComputer Graphics Forum10.1111/cgf.15179Online publication date: 17-Oct-2024
  • (2024)Cinematographic Camera Diffusion ModelComputer Graphics Forum10.1111/cgf.1505543:2Online publication date: 27-Apr-2024
  • (2024)CineMPC: A Fully Autonomous Drone Cinematography System Incorporating Zoom, Focus, Pose, and Scene CompositionIEEE Transactions on Robotics10.1109/TRO.2024.335355040(1740-1757)Online publication date: 1-Jan-2024
  • (2024)Video presentation based on multiple-flying camera to provide continuous and complementary images for teleoperationAutomation in Construction10.1016/j.autcon.2024.105285159(105285)Online publication date: Mar-2024
  • (2023)Autonomy in UAV Civilian ApplicationsAutonomous Vehicles - Applications and Perspectives10.5772/intechopen.1001969Online publication date: 31-Jul-2023
  • (2023)Perspective Of Vision, Motion Planning, And Motion Control for Quadruped RobotsHighlights in Science, Engineering and Technology10.54097/hset.v38i.597638(902-916)Online publication date: 16-Mar-2023
  • (2023)Probabilistic Multi-Robot Collision Avoidance Using Chance-Constrained Safety Barrier Certificates2023 42nd Chinese Control Conference (CCC)10.23919/CCC58697.2023.10241128(5817-5822)Online publication date: 24-Jul-2023
  • (2023)Aerial roaming aerial photography path planning based on areas of interestProceedings of the 2023 7th International Conference on Electronic Information Technology and Computer Engineering10.1145/3650400.3650689(1731-1737)Online publication date: 20-Oct-2023
  • (2023)BirdViewAR: Surroundings-aware Remote Drone Piloting Using an Augmented Third-person PerspectiveProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580681(1-19)Online publication date: 19-Apr-2023
  • (2023)A Drone Video Clip Dataset and its Applications in Automated CinematographyComputer Graphics Forum10.1111/cgf.1466841:7(189-203)Online publication date: 20-Mar-2023
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media