skip to main content
research-article

OmniPhotos: casual 360° VR photography

Published: 27 November 2020 Publication History

Abstract

Virtual reality headsets are becoming increasingly popular, yet it remains difficult for casual users to capture immersive 360° VR panoramas. State-of-the-art approaches require capture times of usually far more than a minute and are often limited in their supported range of head motion. We introduce OmniPhotos, a novel approach for quickly and casually capturing high-quality 360° panoramas with motion parallax. Our approach requires a single sweep with a consumer 360° video camera as input, which takes less than 3 seconds to capture with a rotating selfie stick or 10 seconds handheld. This is the fastest capture time for any VR photography approach supporting motion parallax by an order of magnitude. We improve the visual rendering quality of our OmniPhotos by alleviating vertical distortion using a novel deformable proxy geometry, which we fit to a sparse 3D reconstruction of captured scenes. In addition, the 360° input views significantly expand the available viewing area, and thus the range of motion, compared to previous approaches. We have captured more than 50 OmniPhotos and show video results for a large variety of scenes. We will make our code available.

Supplementary Material

MP4 File (a266-bertel.mp4)
MP4 File (3414685.3417770.mp4)
Presentation video

References

[1]
Sameer Agarwal, Keir Mierle, and Others. 2012. Ceres Solver. http://ceres-solver.org.
[2]
Kara-Ali Aliev, Artem Sevastopolsky, Maria Kolos, Dmitry Ulyanov, and Victor Lempitsky. 2020. Neural Point-Based Graphics. In ECCV.
[3]
Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernandez, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual Reality Video. ACM Transactions on Graphics 35, 6 (2016), 198:1--13.
[4]
Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. 2020. MatryODShka: Real-time 6DoF Video View Synthesis using Multi-Sphere Images. In ECCV.
[5]
Lewis Baker, Steven Mills, Stefanie Zollmann, and Jonathan Ventura. 2020. CasualStereo: Casual Capture of Stereo Panoramas with Spherical Structure-from-Motion. In IEEE VR.
[6]
Tobias Bertel, Neill D. F. Campbell, and Christian Richardt. 2019. MegaParallax: Casual 360° Panoramas with Motion Parallax. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1828--1835.
[7]
Tobias Bertel, Moritz Mühlhausen, Moritz Kappel, Paul Maximilian Bittner, Christian Richardt, and Marcus Magnor. 2020. Depth Augmented Omnidirectional Stereo for 6-DoF VR Photography. In IEEE VR Posters.
[8]
Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew DuVall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. 2020. Immersive Light Field Video with a Layered Mesh Representation. ACM Transactions on Graphics 39, 4 (2020), 86:1--15.
[9]
Gaurav Chaurasia, Sylvain Duchêne, Olga Sorkine-Hornung, and George Drettakis. 2013. Depth Synthesis and Local Warps for Plausible Image-based Navigation. ACM Transactions on Graphics 32, 3 (2013), 30:1--12.
[10]
Javier Civera, Andrew J. Davison, and J. M. Martínez Montiel. 2008. Inverse Depth Parametrization for Monocular SLAM. IEEE Transactions on Robotics 24, 5 (2008), 932--945.
[11]
Brian Curless, Steve Seitz, Jean-Yves Bouguet, Paul Debevec, Marc Levoy, and Shree K. Nayar. 2000. 3D Photography. In SIGGRAPH Courses. http://www.cs.cmu.edu/~seitz/course/3DPhoto.html
[12]
Thiago Lopes Trugillo da Silveira and Claudio R Jung. 2019. Dense 3D Scene Reconstruction from Multiple Spherical Images for 3-DoF+ VR Applications. In IEEE VR. 9--18.
[13]
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019. DeepView: View Synthesis With Learned Gradient Descent. In CVPR. 2367--2376.
[14]
Peter Hedman, Suhib Alsisan, Richard Szeliski, and Johannes Kopf. 2017. Casual 3D Photography. ACM Transactions on Graphics 36, 6 (2017), 234:1--15.
[15]
Peter Hedman and Johannes Kopf. 2018. Instant 3D Photography. ACM Transactions on Graphics 37, 4 (2018), 101:1--12.
[16]
Peter Hedman, Julien Philip, True Price, Jan-Michael Frahm, George Drettakis, and Gabriel Brostow. 2018. Deep Blending for Free-Viewpoint Image-Based Rendering. ACM Transactions on Graphics 37, 6 (2018), 257:1--15.
[17]
Peter Hedman, Tobias Ritschel, George Drettakis, and Gabriel Brostow. 2016. Scalable Inside-Out Image-Based Rendering. ACM Transactions on Graphics 35, 6 (2016), 231:1--11.
[18]
Aleksander Holynski and Johannes Kopf. 2018. Fast Depth Densification for Occlusion-aware Augmented Reality. ACM Transactions on Graphics 37, 6 (2018), 194:1--11.
[19]
Ian P. Howard and Brian J. Rogers. 2008. Seeing in Depth. Oxford University Press.
[20]
Jingwei Huang, Zhili Chen, Duygu Ceylan, and Hailin Jin. 2017. 6-DOF VR videos with a single 360-camera. In IEEE VR. 37--44.
[21]
Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2017. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. In CVPR.
[22]
Sunghoon Im, Hyowon Ha, François Rameau, Hae-Gon Jeon, Gyeongmin Choe, and In So Kweon. 2016. All-around Depth from Small Motion with A Spherical Panoramic Camera. In ECCV.
[23]
Robert Konrad, Donald G. Dansereau, Aniq Masood, and Gordon Wetzstein. 2017. SpinVR: Towards Live-Streaming 3D Virtual Reality Video. ACM Transactions on Graphics 36, 6 (2017), 209:1--12.
[24]
Johannes Kopf, Suhib Alsisan, Francis Ge, Yangming Chong, Kevin Matzen, Ocean Quigley, Josh Patterson, Jossie Tirado, Shu Wu, and Michael F. Cohen. 2019. Practical 3D Photography. In CVPR Workshops.
[25]
George Alex Koulieris, Kaan Akşit, Michael Stengel, Rafał K. Mantiuk, Katerina Mania, and Christian Richardt. 2019. Near-Eye Display and Tracking Technologies for Virtual and Augmented Reality. Computer Graphics Forum 38, 2 (2019), 493--519.
[26]
Till Kroeger, Radu Timofte, Dengxin Dai, and Luc Van Gool. 2016. Fast Optical Flow Using Dense Inverse Search. In ECCV. 471--488.
[27]
Jungjin Lee, Bumki Kim, Kyehyun Kim, Younghui Kim, and Junyong Noh. 2016. Rich360: Optimized Spherical Representation from Structured Panoramic Camera Arrays. ACM Transactions on Graphics 35, 4 (2016), 63:1--11.
[28]
Christian Lipski, Felix Klose, and Marcus Magnor. 2014. Correspondence and Depth-Image Based Rendering a Hybrid Approach for Free-Viewpoint Video. IEEE Transactions on Circuits and Systems for Video Technology 24, 6 (2014), 942--951.
[29]
Bicheng Luo, Feng Xu, Christian Richardt, and Jun-Hai Yong. 2018. Parallax360: Stereoscopic 360° Scene Representation for Head-Motion Parallax. IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1545--1553.
[30]
Kevin Matzen, Michael F. Cohen, Bryce Evans, Johannes Kopf, and Richard Szeliski. 2017. Low-cost 360 Stereo Photography and Video Capture. ACM Transactions on Graphics 36, 4 (2017), 148:1--12.
[31]
Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. 2019. Neural Rerendering in the Wild. In CVPR.
[32]
Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines. ACM Transactions on Graphics 38, 4 (2019), 29:1--14.
[33]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. 2020. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV.
[34]
Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. 2019. HoloGAN: Unsupervised learning of 3D representations from natural images. In ICCV.
[35]
Ryan Styles Overbeck, Daniel Erickson, Daniel Evangelakos, Matt Pharr, and Paul Debevec. 2018. A System for Acquiring, Compressing, and Rendering Panoramic Light Field Stills for Virtual Reality. ACM Transactions on Graphics 37, 6 (2018), 197:1--15.
[36]
Albert Parra Pozo, Michael Toksvig, Terry Filiba Schrager, Joyse Hsu, Uday Mathur, Alexander Sorkine-Hornung, Rick Szeliski, and Brian Cabral. 2019. An Integrated 6DoF Video Camera and System Design. ACM Transactions on Graphics 38, 6 (2019), 216:1--16.
[37]
Shmuel Peleg, Moshe Ben-Ezra, and Yael Pritch. 2001. Omnistereo: Panoramic Stereo Imaging. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 3 (2001), 279--290.
[38]
Federico Perazzi, Alexander Sorkine-Hornung, Henning Zimmer, Peter Kaufmann, Oliver Wang, Scott Watson, and Markus Gross. 2015. Panoramic Video from Unstructured Camera Arrays. Computer Graphics Forum 34, 2 (2015), 57--68.
[39]
Christian Richardt. 2020. Omnidirectional Stereo. In Computer Vision: A Reference Guide. Springer, 1--4.
[40]
Christian Richardt, Peter Hedman, Ryan S. Overbeck, Brian Cabral, Robert Konrad, and Steve Sullivan. 2019. Capture4VR: From VR Photography to VR Video. In SIGGRAPH Courses. 1--319.
[41]
Christian Richardt, Yael Pritch, Henning Zimmer, and Alexander Sorkine-Hornung. 2013. Megastereo: Constructing High-Resolution Stereo Panoramas. In CVPR. 1256--1263.
[42]
Christian Richardt, James Tompkin, and Gordon Wetzstein. 2020. Capture, Reconstruction, and Representation of the Visual Real World for Virtual Reality. In Real VR - Immersive Digital Reality: How to Import the Real World into Head-Mounted Immersive Displays. Springer, 3--32.
[43]
Ehsan Sayyad, Pradeep Sen, and Tobias Höllerer. 2017. PanoTrace: Interactive 3D Modeling of Surround-View Panoramic Images in Virtual Reality. In VRST.
[44]
Johannes L. Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In CVPR. 4104--4113.
[45]
Christopher Schroers, Jean-Charles Bazin, and Alexander Sorkine-Hornung. 2018. An Omnistereoscopic Video Pipeline for Capture and Display of Real-World VR. ACM Transactions on Graphics 37, 3 (2018), 37:1--13.
[46]
Ana Serrano, Incheol Kim, Zhili Chen, Stephen DiVerdi, Diego Gutierrez, Aaron Hertzmann, and Belen Masia. 2019. Motion parallax for 360° RGBD video. IEEE Transactions on Visualization and Computer Graphics 25, 5 (2019), 1817--1827.
[47]
Meng-Li Shih, Shih-Yang Su, Johannes Kopf, and Jia-Bin Huang. 2020. 3D Photography using Context-aware Layered Depth Inpainting. In CVPR.
[48]
Heung-Yeung Shum and Li-Wei He. 1999. Rendering with concentric mosaics. In SIGGRAPH. 299--306.
[49]
Vincent Sitzmann, Ana Serrano, Amy Pavel, Maneesh Agrawala, Diego Gutierrez, Belen Masia, and Gordon Wetzstein. 2018. How do people explore virtual environments? IEEE Transactions on Visualization and Computer Graphics 24, 4 (2018), 1633--1642.
[50]
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. 2019a. DeepVoxels: Learning Persistent 3D Feature Embeddings. In CVPR. 2437--2446.
[51]
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019b. Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations. In NeurIPS.
[52]
Mel Slater, Martin Usoh, and Anthony Steed. 1994. Depth of Presence in Virtual Environments. Presence: Teleoperators and Virtual Environments 3, 2 (1994), 130--144.
[53]
Pratul P. Srinivasan, Richard Tucker, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. 2019. Pushing the Boundaries of View Extrapolation With Multiplane Images. In CVPR. 175--184.
[54]
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clarkson, Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueggler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goesele, Steven Lovegrove, and Richard Newcombe. 2019. The Replica Dataset: A Digital Replica of Indoor Spaces. (2019). https://github.com/facebookresearch/Replica-Dataset arXiv:1906.05797.
[55]
Shinya Sumikura, Mikiya Shibuya, and Ken Sakurada. 2019. OpenVSLAM: a Versatile Visual SLAM Framework. In International Conference on Multimedia.
[56]
Richard Szeliski. 2006. Image alignment and stitching: a tutorial. Foundations and Trends in Computer Graphics and Vision 2, 1 (2006), 1--104.
[57]
Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B Goldman, and Michael Zollhöfer. 2020. State of the Art on Neural Rendering. Computer Graphics Forum 39, 2 (2020), 701--727.
[58]
Jayant Thatte, Jean-Baptiste Boin, Haricharan Lakshman, and Bernd Girod. 2016. Depth augmented stereo panorama for cinematic virtual reality with head-motion parallax. In ICME.
[59]
Richard Tucker and Noah Snavely. 2020. Single-View View Synthesis with Multiplane Images. In CVPR.
[60]
Julien Valentin, Adarsh Kowdle, Jonathan T. Barron, Neal Wadhwa, Max Dzitsiuk, Michael Schoenberg, Vivek Verma, Ambrus Csaszar, Eric Turner, Ivan Dryanovski, Joao Afonso, Jose Pascoal, Konstantine Tsotsos, Mira Leung, Mirko Schmidt, Onur Guleryuz, Sameh Khamis, Vladimir Tankovitch, Sean Fanello, Shahram Izadi, and Christoph Rhemann. 2018. Depth from Motion for Smartphone AR. ACM Transactions on Graphics 37, 6 (2018), 193:1--19.
[61]
Michael Waechter, Mate Beljan, Simon Fuhrmann, Nils Moehrle, Johannes Kopf, and Michael Goesele. 2017. Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction. ACM Transactions on Graphics 36, 1 (2017), 8:1--11.
[62]
Fu-En Wang, Yu-Hsuan Yeh, Min Sun, Wei-Chen Chiu, and Yi-Hsuan Tsai. 2020. BiFuse: Monocular 360 Depth Estimation via Bi-Projection Fusion. In CVPR. 462--471.
[63]
Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.
[64]
Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. 2020. SynSin: End-to-end View Synthesis from a Single Image. In CVPR.
[65]
Jianing Zhang, Tianyi Zhu, Anke Zhang, Xiaoyun Yuan, Zihan Wang, Sebastian Beetschen, Lan Xu, Xing Lin, Qionghai Dai, and Lu Fang. 2020. Multiscale-VR: Multiscale Gigapixel 3D Panoramic Videography for Virtual Reality. In ICCP.
[66]
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. 2018. The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In CVPR.
[67]
Ke Colin Zheng, Sing Bing Kang, Michael F. Cohen, and Richard Szeliski. 2007. Layered Depth Panoramas. In CVPR.
[68]
Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo Magnification: Learning View Synthesis using Multiplane Images. ACM Transactions on Graphics 37, 4 (2018), 65:1--12.
[69]
Nikolaos Zioulis, Antonis Karakottas, Dimitrios Zarpalas, Federico Alvarez, and Petros Daras. 2019. Spherical View Synthesis for Self-Supervised 360° Depth Estimation. In 3DV. 690--699.

Cited By

View all
  • (2024)Ladder Safety Assessment Using Head-Mounted 360-Degree Camera-Based Posture Estimation Overlayed Real-Time in Augmented Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00055(1-4)Online publication date: 16-Mar-2024
  • (2024)Avatar360: Emulating 6-DoF Perception in 360°Panoramas through Avatar-Assisted Navigation2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00083(630-638)Online publication date: 16-Mar-2024
  • (2024)Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02067(21886-21896)Online publication date: 16-Jun-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 39, Issue 6
December 2020
1605 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3414685
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 November 2020
Published in TOG Volume 39, Issue 6

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. casual capture
  2. image-based rendering
  3. motion parallax
  4. novel-view synthesis

Qualifiers

  • Research-article

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)133
  • Downloads (Last 6 weeks)18
Reflects downloads up to 03 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Ladder Safety Assessment Using Head-Mounted 360-Degree Camera-Based Posture Estimation Overlayed Real-Time in Augmented Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00055(1-4)Online publication date: 16-Mar-2024
  • (2024)Avatar360: Emulating 6-DoF Perception in 360°Panoramas through Avatar-Assisted Navigation2024 IEEE Conference Virtual Reality and 3D User Interfaces (VR)10.1109/VR58804.2024.00083(630-638)Online publication date: 16-Mar-2024
  • (2024)Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmark2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02067(21886-21896)Online publication date: 16-Jun-2024
  • (2024)OmniSDF: Scene Reconstruction Using Omnidirectional Signed Distance Functions and Adaptive Binoctrees2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01912(20227-20236)Online publication date: 16-Jun-2024
  • (2024)OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00656(6871-6880)Online publication date: 16-Jun-2024
  • (2024)Modeling Practical Multi-Center-of-Projection Using EllipsoidIEEE Access10.1109/ACCESS.2024.345150212(122328-122339)Online publication date: 2024
  • (2024)The effects of digital nature and actual nature on stress reduction: A meta-analysis and systematic reviewInternet Interventions10.1016/j.invent.2024.10077238(100772)Online publication date: Dec-2024
  • (2024)Improved YOLOv7 models based on modulated deformable convolution and swin transformer for object detection in fisheye imagesImage and Vision Computing10.1016/j.imavis.2024.104966144:COnline publication date: 1-Apr-2024
  • (2024)Deep synthesis and exploration of omnidirectional stereoscopic environments from a single surround-view panoramic imageComputers and Graphics10.1016/j.cag.2024.103907119:COnline publication date: 1-Apr-2024
  • (2024)Volumetric Rendering with Baked Quadrature FieldsComputer Vision – ECCV 202410.1007/978-3-031-73036-8_16(275-292)Online publication date: 29-Sep-2024
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media