skip to main content
research-article

Computational Long Exposure Mobile Photography

Published: 26 July 2023 Publication History

Abstract

Long exposure photography produces stunning imagery, representing moving elements in a scene with motion-blur. It is generally employed in two modalities, producing either a foreground or a background blur effect. Foreground blur images are traditionally captured on a tripod-mounted camera and portray blurred moving foreground elements, such as silky water or light trails, over a perfectly sharp background landscape. Background blur images, also called panning photography, are captured while the camera is tracking a moving subject, to produce an image of a sharp subject over a background blurred by relative motion. Both techniques are notoriously challenging and require additional equipment and advanced skills. In this paper, we describe a computational burst photography system that operates in a hand-held smartphone camera app, and achieves these effects fully automatically, at the tap of the shutter button. Our approach first detects and segments the salient subject. We track the scene motion over multiple frames and align the images in order to preserve desired sharpness and to produce aesthetically pleasing motion streaks. We capture an under-exposed burst and select the subset of input frames that will produce blur trails of controlled length, regardless of scene or camera motion velocity. We predict inter-frame motion and synthesize motion-blur to fill the temporal gaps between the input frames. Finally, we composite the blurred image with the sharp regular exposure to protect the sharpness of faces or areas of the scene that are barely moving, and produce a final high resolution and high dynamic range (HDR) photograph. Our system democratizes a capability previously reserved to professionals, and makes this creative style accessible to most casual photographers.

Supplementary Material

ZIP File (papers_305-supplemental.zip)
supplemental material.
MP4 File (papers_305_VOD.mp4)
presentation

References

[1]
Kfir Aberman, Ju He, Yossi Gandelsman, Inbar Mosseri, David E. Jacobs, Kai Kohlhoff, Yael Pritch, and Michael Rubinstein. 2022. Deep Saliency Prior for Reducing Visual Distraction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 19819--19828.
[2]
Sameer Agarwal, Keir Mierle, and The Ceres Solver Team. 2022. Ceres Solver. Google, Inc. https://github.com/ceres-solver/ceres-solver
[3]
Valentin Bazarevsky, Yury Kartynnik, Andrey Vakunov, Karthik Raveendran, and Matthias Grundmann. 2019. BlazeFace: Sub-millisecond Neural Face Detection on Mobile GPUs. ArXiv abs/1907.05047 (2019).
[4]
Tim Brooks and Jonathan T Barron. 2019. Learning to Synthesize Motion Blur. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 6833--6841.
[5]
Gabriel J. Brostow and Irfan Essa. 2001. Image-Based Motion Blur for Stop Motion Animation. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '01). Association for Computing Machinery, New York, NY, USA, 561--566.
[6]
Vincent Dumoulin and Francesco Visin. 2016. A guide to convolution arithmetic for deep learning.
[7]
David S Ebert, F Kenton Musgrave, Darwyn Peachey, Ken Perlin, and Steven Worley. 2003. Texturing & modeling: a procedural approach. Morgan Kaufmann.
[8]
Matthias Grundmann, Vivek Kwatra, and Irfan Essa. 2011. Auto-directed video stabilization with robust L1 optimal camera paths. In 2011 IEEE Conference on Computer Vision and Pattern Recognition. IEEE Computer Society, Los Alamitos, CA, USA, 225--232.
[9]
Monika Gupta. 2021. Google Tensor is a milestone for machine learning. https://blog.google/products/pixel/introducing-google-tensor
[10]
R. I. Hartley and A. Zisserman. 2004. Multiple View Geometry in Computer Vision (second ed.). Cambridge University Press, ISBN: 0521540518.
[11]
Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-Light Imaging on Mobile Cameras. ACM Trans. Graph. 35, 6, Article 192 (dec 2016), 12 pages.
[12]
Kaiming He, Jian Sun, and Xiaoou Tang. 2013. Guided Image Filtering. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 6 (2013), 1397--1409.
[13]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In 2015 IEEE International Conference on Computer Vision (ICCV). IEEE Computer Society, Los Alamitos, CA, USA, 1026--1034.
[14]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 770--778.
[15]
Peter J. Huber. 1964. Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics 35, 1 (1964), 73 -- 101.
[16]
Ming Jiang, Shengsheng Huang, Juanyong Duan, and Qi Zhao. 2015. SALICON: Saliency in Context. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 1072--1080.
[17]
Alexandre Karpenko, David Jacobs, Jongmin Baek, and Marc Levoy. 2011. Digital video stabilization and rolling shutter correction using gyroscopes. Technical Report. Stanford University CS. 13 pages.
[18]
M. Lancelle, P. Dogan, and M. Gross. 2019. Controlling Motion Blur in Synthetic Long Time Exposures. Computer Graphics Forum 38, 2 (2019), 393--403.
[19]
H. Lee, C. H. Lee, and K. Yoon. 2009. Motion based Painterly Rendering. Computer Graphics Forum 28, 4 (2009), 1207--1215.
[20]
Jaakko Lehtinen, Timo Aila, Jiawen Chen, Samuli Laine, and Frédo Durand. 2011. Temporal Light Field Reconstruction for Rendering Distribution Effects. ACM Trans. Graph. 30, 4, Article 55 (jul 2011), 12 pages.
[21]
Orly Liba, Kiran Murthy, Yun-Ta Tsai, Tim Brooks, Tianfan Xue, Nikhil Karnad, Qiurui He, Jonathan T. Barron, Dillon Sharlet, Ryan Geiss, Samuel W. Hasinoff, Yael Pritch, and Marc Levoy. 2019. Handheld Mobile Photography in Very Low Light. ACM Trans. Graph. 38, 6, Article 164 (nov 2019), 16 pages.
[22]
Feng Liu, Michael Gleicher, Hailin Jin, and Aseem Agarwala. 2009. Content-Preserving Warps for 3D Video Stabilization. ACM Trans. Graph. 28, 3, Article 44 (jul 2009), 9 pages.
[23]
Feng Liu, Michael Gleicher, Jue Wang, Hailin Jin, and Aseem Agarwala. 2011. Subspace Video Stabilization. ACM Trans. Graph. 30, 1, Article 4 (feb 2011), 10 pages.
[24]
Shuaicheng Liu, Jue Wang, Sunghyun Cho, and Ping Tan. 2014. TrackCam: 3D-Aware Tracking Shots from Consumer Video. ACM Trans. Graph. 33, 6, Article 198 (Nov 2014), 11 pages.
[25]
Shuaicheng Liu, Lu Yuan, Ping Tan, and Jian Sun. 2013. Bundled Camera Paths for Video Stabilization. ACM Trans. Graph. 32, 4, Article 78 (jul 2013), 10 pages.
[26]
Xuejiao Luo, Nestor Z. Salamon, and Elmar Eisemann. 2018. Adding Motion Blur to Still Images. In Proceedings of the 44th Graphics Interface Conference (Toronto, Canada) (GI '18). Canadian Human-Computer Communications Society, Waterloo, CAN, 108--114.
[27]
Xuejiao Luo, Nestor Z. Salamon, and Elmar Eisemann. 2020. Controllable Motion-Blur Effects in Still Images. IEEE Transactions on Visualization and Computer Graphics 26, 7 (2020), 2362--2372.
[28]
Michihiro Mikamo, Ryo Furukawa, and Hiroshi Kawasaki. 2021. A Method for Adding Motion-Blur on Arbitrary Objects by Using Auto-Segmentation and Color Compensation Techniques. In 2021 IEEE International Conference on Image Processing (ICIP). 1854--1858.
[29]
C. Morimoto and R. Chellappa. 1998. Evaluation of image stabilization algorithms. In Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP '98 (Cat. No.98CH36181), Vol. 5. 2789--2792 vol.5.
[30]
Fernando Navarro, Francisco J. Serón, and Diego Gutierrez. 2011. Motion Blur Rendering: State of the Art. Computer Graphics Forum 30, 1 (2011), 3--26.
[31]
Augustus Odena, Vincent Dumoulin, and Chris Olah. 2016. Deconvolution and Checkerboard Artifacts. Distill (2016).
[32]
Fatih Porikli. 2004. Learning object trajectory patterns by spectral clustering. In 2004 IEEE International Conference on Multimedia and Expo (ICME), Vol. 2. IEEE Computer Society, Los Alamitos, USA, 1171--1174.
[33]
Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. 2022. FILM: Frame Interpolation For Large Motion. In Computer Vision - ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23--27, 2022, Proceedings, Part VII (Tel Aviv, Israel). Springer-Verlag, Berlin, Heidelberg, 250--266.
[34]
Mads J.L. Rønnow, Ulf Assarsson, and Marco Fratarcangeli. 2021. Fast analytical motion blur with transparency. Computers & Graphics 95 (2021), 36--46.
[35]
Nestor Z. Salamon, Markus Billeter, and Elmar Eisemann. 2019. ShutterApp: Spatiotemporal Exposure Control for Videos. Computer Graphics Forum 38, 7 (2019), 675--683.
[36]
Christophe Schlick. 1994. Fast Alternatives to Perlin's Bias and Gain Functions. Academic Press Professional, Inc., USA, 401--403.
[37]
Fuhao Shi, Sung-Fang Tsai, Youyou Wang, and Chia-Kai Liang. 2019. Steadiface: RealTime Face-Centric Stabilization On Mobile Phones. In 2019 IEEE International Conference on Image Processing (ICIP). 4599--4603.
[38]
Spectre. [n. d.]. Spectre app. https://spectre.cam. Accessed: 2023-01-17.
[39]
Michael Stengel, Pablo Bauszat, Martin Eisemann, Elmar Eisemann, and Marcus Magnor. 2015. Temporal Video Filtering and Exposure Control for Perceptual Motion Blur. IEEE Transactions on Visualization and Computer Graphics 21, 5 (2015), 663--671.
[40]
Okihide Teramoto, In Kyu Park, and Takeo Igarashi. 2010. Interactive Motion Photography from a Single Image. Vis. Comput. 26, 11 (nov 2010), 1339--1348.
[41]
Neal Wadhwa, Rahul Garg, David E. Jacobs, Bryan E. Feldman, Nori Kanazawa, Robert Carroll, Yair Movshovitz-Attias, Jonathan T. Barron, Yael Pritch, and Marc Levoy. 2018. Synthetic Depth-of-Field with a Single-Camera Mobile Phone. ACM Trans. Graph. 37, 4, Article 64 (jul 2018), 13 pages.
[42]
Ben Weiss. 2006. Fast Median and Bilateral Filtering. ACM Trans. Graph. 25, 3 (jul 2006), 519--526.
[43]
Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-Kai Liang, Marc Levoy, and Peyman Milanfar. 2019. Handheld MultiFrame Super-Resolution. ACM Trans. Graph. 38, 4, Article 28 (jul 2019), 18 pages.
[44]
Julio Zaragoza, Tat-Jun Chin, Michael S. Brown, and David Suter. 2013. As-Projective-As-Possible Image Stitching with Moving DLT. In 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE Computer Society, Los Alamitos, CA, USA, 2339--2346.
[45]
Congxuan Zhang, Ling ling Zhu, Zhen Chen, Ding ding Kong, and Xuan Shang. 2016. An Improved Evaluation Method for Optical Flow of Endpoint Error. In Proceedings of the International Conference on Computer Networks and Communication Technology (CNCT 2016). Atlantis Press, 312--317.
[46]
Timo Zinßer, Jochen Schmidt, and Heinrich Niemann. 2005. Point set registration with integrated scale estimation. In International conference on pattern recognition and image processing. 116--119.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 42, Issue 4
August 2023
1912 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3609020
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 July 2023
Published in TOG Volume 42, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. machine learning
  2. mobile computing

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 502
    Total Downloads
  • Downloads (Last 12 months)212
  • Downloads (Last 6 weeks)5
Reflects downloads up to 20 Feb 2025

Other Metrics

Citations

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media