skip to main content
10.1145/3411764.3445568acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Optimization-based User Support for Cinematographic Quadrotor Camera Target Framing

Published: 07 May 2021 Publication History

Abstract

To create aesthetically pleasing aerial footage, the correct framing of camera targets is crucial. However, current quadrotor camera tools do not consider the 3D extent of actual camera targets in their optimization schemes and simply interpolate between keyframes when generating a trajectory. This can yield videos with aesthetically unpleasing target framing. In this paper, we propose a target framing algorithm that optimizes the quadrotor camera pose such that targets are positioned at desirable screen locations according to videographic compositional rules and entirely visible throughout a shot. Camera targets are identified using a semi-automatic pipeline which leverages a deep-learning-based visual saliency model. A large-scale perceptual study (N ≈ 500) shows that our method enables users to produce shots with a target framing that is closer to what they intended to create and more or as aesthetically pleasing than with the previous state of the art.

Supplementary Material

VTT File (3411764.3445568_videofigurecaptions.vtt)
VTT File (3411764.3445568_videopreviewcaptions.vtt)
MP4 File (3411764.3445568_videofigure.mp4)
Supplemental video
MP4 File (3411764.3445568_videopreview.mp4)
Preview video

References

[1]
APM. 2016. APM Autopilot Suite. Retrieved January 6, 2020 from http://ardupilot.com
[2]
Daniel Arijon. 1976. Grammar of the film language. (1976).
[3]
Amirsaman Ashtari, Stefan Stevsic, Tobias Naegeli, Otmar Hilliges, and Jean-Charles Bazin. 2020. Capturing Subjective First-Person View Shots with Drones for Automated Cinematography. ACM Trans. Graph. 0, ja (2020). https://doi.org/10.1145/3378673
[4]
William H. Bares and James C. Lester. 1998. Intelligent Multi-Shot Visualization Interfaces for Dynamic 3D Worlds. In Proceedings of the 4th International Conference on Intelligent User Interfaces (Los Angeles, California, USA) (IUI ’99). Association for Computing Machinery, New York, NY, USA, 119–126. https://doi.org/10.1145/291080.291101
[5]
Rogerio Bonatti, Cherie Ho, Wenshan Wang, Sanjiban Choudhury, and Sebastian Scherer. 2019. Towards a Robust Aerial Cinematography Platform: Localizing and Tracking Moving Targets in Unstructured Environments. (2019). arXiv:1904.02319
[6]
Rogerio Bonatti, Wenshan Wang, Cherie Ho, Aayush Ahuja, Mirko Gschwindt, Efe Camci, Erdal Kayacan, Sanjiban Choudhury, and Sebastian Scherer. 2019. Autonomous Aerial Cinematography In Unstructured Environments With Learned Artistic Decision-Making. (2019). arXiv:1910.06988
[7]
Zoya Bylinskii, Tilke Judd, Ali Borji, Laurent Itti, Frédo Durand, Aude Oliva, and Antonio Torralba. 2012. MIT Saliency Benchmark.
[8]
Eric Cheng. 2016. Aerial Photography and Videography Using Drones. Vol. 1. Peachpit Press.
[9]
Marc Christie, Patrick Olivier, and Jean-Marie Normand. 2008. Camera Control in Computer Graphics. Computer Graphics Forum 27, 8 (Dec. 2008), 2197–2218. https://doi.org/10.1145/1665817.1665820
[10]
Nicholas Davis, Alexander Zook, Brian O’Neill, Brandon Headrick, Mark Riedl, Ashton Grosz, and Michael Nitsche. 2013. Creativity Support for Novice Digital Filmmaking. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 651–660. https://doi.org/10.1145/2470654.2470747
[11]
DJI. 2020. Ground Station Pro. Retrieved January 6, 2020 from https://www.dji.com/ground-station-pro
[12]
Alexander Domahidi and Juan Jerez. 2017. FORCES Pro: code generation for embedded optimization. Retrieved September 4, 2017 from https://www.embotech.com/FORCES-Pro
[13]
Steven M. Drucker and David Zeltzer. 1994. Intelligent Camera Control in a Virtual Environment. In In Proceedings of Graphics Interface ’94. 190–199.
[14]
David K. Elson and Mark O. Riedl. 2007. A Lightweight Intelligent Virtual Cinematography System for Machinima Production. In Proceedings of the Third AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (Stanford, California) (AIIDE’07). AAAI Press, 8–13.
[15]
Q. Galvane, J. Fleureau, F. L. Tariolle, and P. Guillotel. 2016. Automated Cinematography with Unmanned Aerial Vehicles. In Proceedings of the Eurographics Workshop on Intelligent Cinematography and Editing (Lisbon, Portugal) (WICED ’16). Eurographics Association, Goslar Germany, Germany, 23–30. https://doi.org/10.2312/wiced.20161097
[16]
Quentin Galvane, Christophe Lino, Marc Christie, Julien Fleureau, Fabien Servant, Fran Tariolle, Philippe Guillotel, 2018. Directing cinematographic drones. ACM Transactions on Graphics (TOG) 37, 3 (2018), 34.
[17]
Christoph Gebhardt, Benjamin Hepp, Tobias Nägeli, Stefan Stevšić, and Otmar Hilliges. 2016. Airways: Optimization-Based Planning of Quadrotor Trajectories According to High-Level User Goals. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (Santa Clara, California, USA) (CHI ’16). ACM, New York, NY, USA, 2508–2519. https://doi.org/10.1145/2858036.2858353
[18]
Christoph Gebhardt and Otmar Hilliges. 2018. WYFIWYG: Investigating Effective User Support in Aerial Videography. (2018). arXiv:1801.05972
[19]
Christoph Gebhardt, Stefan Stevsic, and Otmar Hilliges. 2018. Optimizing for Aesthetically Pleasing Quadrotor Camera Motion. 37, 4, Article 90 (2018), 90:1–90:11 pages.
[20]
Michael Gleicher and Andrew Witkin. 1992. Through-the-lens camera control. In Siggraph, Vol. 92. 331–340.
[21]
Mirko Gschwindt, Efe Camci, Rogerio Bonatti, Wenshan Wang, Erdal Kayacan, and Sebastian Scherer. 2019. Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography. (2019). arXiv:1904.02579
[22]
Chong Huang, Fei Gao, Jie Pan, Zhenyu Yang, Weihao Qiu, Peng Chen, Xin Yang, Shaojie Shen, and Kwang-Ting Tim Cheng. 2018. Act: An autonomous drone cinematography system for action scenes. 7039–7046.
[23]
Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human–computer interaction 1, 4 (1985), 311–338.
[24]
Hongda Jiang, Bin Wang, Xi Wang, Marc Christie, and Baoquan Chen. 2020. Example-driven Virtual Cinematography by Learning CameraBehaviors. ACM Transactions on Graphics (Proceedings of ACM SIGGRAPH) XX, X (8 2020), 14 pages.
[25]
Niels Joubert, Dan B Goldman, Floraine Berthouzoz, Mike Roberts, James A Landay, Pat Hanrahan, 2016. Towards a Drone Cinematographer: Guiding Quadrotor Cameras using Visual Composition Principles. (2016). arXiv:1610.01691
[26]
Niels Joubert, Mike Roberts, Anh Truong, Floraine Berthouzoz, and Pat Hanrahan. 2015. An Interactive Tool for Designing Quadrotor Camera Shots. ACM Trans. Graph. 34, 6, Article 238, 11 pages. https://doi.org/10.1145/2816795.2818106
[27]
Matthias Kümmerer. 2016. DeepGaze II. Retrieved September 3, 2020 from https://deepgaze.bethgelab.org/
[28]
Matthias Kummerer, Thomas SA Wallis, Leon A Gatys, and Matthias Bethge. 2017. Understanding low-and high-level contributions to fixation prediction. In Proceedings of the IEEE International Conference on Computer Vision. 4789–4798.
[29]
Tsai-Yen Li and Chung-Chiang Cheng. 2008. Real-Time Camera Planning for Navigation in Virtual Environments. Springer Berlin Heidelberg, Berlin, Heidelberg, 118–129. https://doi.org/10.1007/978-3-540-85412-8_11
[30]
Christophe Lino and Marc Christie. 2012. Efficient Composition for Virtual Camera Control. In Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation (Lausanne, Switzerland) (SCA ’12). Eurographics Association, Aire-la-Ville, Switzerland, Switzerland, 65–70. https://doi.org/10.1145/1409060.1409068
[31]
Christophe Lino and Marc Christie. 2015. Intuitive and Efficient Camera Control with the Toric Space. ACM Trans. Graph. 34, 4, Article 82 (July 2015), 12 pages. https://doi.org/10.1145/2766965
[32]
Christophe Lino, Marc Christie, Roberto Ranon, and William Bares. 2011. The Director’s Lens: An Intelligent Assistant for Virtual Cinematography. In Proceedings of the 19th ACM International Conference on Multimedia (Scottsdale, Arizona, USA) (MM ’11). ACM, New York, NY, USA, 323–332. https://doi.org/10.1145/2072298.2072341
[33]
MathWorks. 2017. Segment point cloud into clusters based on Euclidean distance. Retrieved January 8, 2020 from https://mathworks.com/help/vision/ref/pcsegdist.html
[34]
T. Naegeli, J. Alonso-Mora, A. Domahidi, D. Rus, and O. Hilliges. 2017. Real-time Motion Planning for Aerial Videography with Dynamic Obstacle Avoidance and Viewpoint Optimization. IEEE Robotics and Automation Letters PP, 99 (2017), 1–1. https://doi.org/10.1109/LRA.2017.2665693
[35]
Tobias Nägeli, Lukas Meier, Alexander Domahidi, Javier Alonso-Mora, and Otmar Hilliges. 2017. Real-time Planning for Automated Multi-view Drone Cinematography. ACM Trans. Graph. 36, 4, Article 132 (July 2017), 10 pages. https://doi.org/10.1145/3072959.3073712
[36]
Fabio Poiesi and Andrea Cavallaro. 2015. Distributed vision-based flying cameras to film a moving target. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2453–2459.
[37]
Mike Roberts and Pat Hanrahan. 2016. Generating Dynamically Feasible Trajectories for Quadrotor Cameras. ACM Trans. Graph. 35, 4, Article 61 (July 2016), 11 pages. https://doi.org/10.1145/2897824.2925980
[38]
Gauthier Rousseau, Cristina Stoica Maniu, Sihem Tebbani, Mathieu Babel, and Nicolas Martin. 2018. Quadcopter-performed cinematographic flight plans using minimum jerk trajectories and predictive camera control. In 2018 European Control Conference (ECC). IEEE, 2897–2903.
[39]
Bahareh Sabetghadam, Alfonso Alcántara, Jesús Capitán, Rita Cunha, Aníbal Ollero, and Antonio Pascoal. 2019. Optimal Trajectory Planning for Autonomous Drone Cinematography. In 2019 European Conference on Mobile Robots (ECMR). IEEE, 1–7.
[40]
Ben Shneiderman. 1997. Direct manipulation for comprehensible, predictable and controllable user interfaces. In Proceedings of the 2nd international conference on Intelligent user interfaces. 33–39.
[41]
Angie Taylor. 2011. Composition. In Design Essentials for the Motion Media Artist, Angie Taylor (Ed.). Focal Press, Boston, 99 – 142. https://doi.org/10.1016/B978-0-240-81181-9.00010-8
[42]
VC Technology. 2019. Litchi Tool. Retrieved January 6, 2020 from https://flylitchi.com/
[43]
Ke Xie, Hao Yang, Shengqiu Huang, Dani Lischinski, Marc Christie, Kai Xu, Minglun Gong, Daniel Cohen-Or, and Hui Huang. 2018. Creating and chaining camera moves for quadrotor videography. ACM Transactions on Graphics (TOG) 37, 4 (2018), 88.
[44]
I-Cheng Yeh, Chao-Hung Lin, Hung-Jen Chien, and Tong-Yee Lee. 2011. Efficient camera path planning algorithm for human motion overview. Computer Animation and Virtual Worlds 22, 2-3 (2011), 239–250. https://doi.org/10.1002/cav.398
[45]
YUNEEC. 2018. Data Pilot. Retrieved January 6, 2020 from https://us.yuneec.com/comm-en-datapilot

Cited By

View all
  • (2024)Dragon's Path: Synthesizing User-Centered Flying Creature Animation Paths for Outdoor Augmented Reality ExperiencesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657397(1-11)Online publication date: 13-Jul-2024
  • (2024)OptFlowCam: A 3D‐Image‐Flow‐Based Metric in Camera Space for Camera Paths in Scenes with Extreme Scale VariationsComputer Graphics Forum10.1111/cgf.1505643:2Online publication date: 27-Apr-2024
  • (2024)Embodied Human Activity Recognition2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00632(6433-6443)Online publication date: 3-Jan-2024
  • Show More Cited By

Index Terms

  1. Optimization-based User Support for Cinematographic Quadrotor Camera Target Framing
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
        May 2021
        10862 pages
        ISBN:9781450380966
        DOI:10.1145/3411764
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 07 May 2021

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Aerial Videography
        2. Computational Design
        3. Quadrotor Camera Tools
        4. Trajectory Optimization

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Conference

        CHI '21
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

        Upcoming Conference

        CHI 2025
        ACM CHI Conference on Human Factors in Computing Systems
        April 26 - May 1, 2025
        Yokohama , Japan

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)44
        • Downloads (Last 6 weeks)5
        Reflects downloads up to 16 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Dragon's Path: Synthesizing User-Centered Flying Creature Animation Paths for Outdoor Augmented Reality ExperiencesACM SIGGRAPH 2024 Conference Papers10.1145/3641519.3657397(1-11)Online publication date: 13-Jul-2024
        • (2024)OptFlowCam: A 3D‐Image‐Flow‐Based Metric in Camera Space for Camera Paths in Scenes with Extreme Scale VariationsComputer Graphics Forum10.1111/cgf.1505643:2Online publication date: 27-Apr-2024
        • (2024)Embodied Human Activity Recognition2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV57701.2024.00632(6433-6443)Online publication date: 3-Jan-2024
        • (2023)A Drone Video Clip Dataset and its Applications in Automated CinematographyComputer Graphics Forum10.1111/cgf.1466841:7(189-203)Online publication date: 20-Mar-2023

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media