Michael Waechter
Michael Goesele
Skip Abstract Section
Skip Supplemental Material SectionAbstract
The ultimate goal of many image-based modeling systems is to render photo-realistic novel views of a scene without visible artifacts. Existing evaluation metrics and benchmarks focus mainly on the geometric accuracy of the reconstructed model, which is, however, a poor predictor of visual accuracy. Furthermore, using only geometric accuracy by itself does not allow evaluating systems that either lack a geometric scene representation or utilize coarse proxy geometry. Examples include a light field and most image-based rendering systems. We propose a unified evaluation approach based on novel view prediction error that is able to analyze the visual quality of any method that can render novel views from input images. One key advantage of this approach is that it does not require ground truth geometry. This dramatically simplifies the creation of test datasets and benchmarks. It also allows us to evaluate the quality of an unknown scene during the acquisition and reconstruction process, which is useful for acquisition planning. We evaluate our approach on a range of methods, including standard geometry-plus-texture pipelines as well as image-based rendering techniques, compare it to existing geometry-based benchmarks, demonstrate its utility for a range of use cases, and present a new virtual rephotography-based benchmark for image-based modeling and rendering systems.
Supplemental Material
Available for Download
zip
waechter.zip (250.4 MB)
Supplemental movie and image files for, Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
- Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-scale data for multiple-view stereopsis. IJCV 120, 2, 153--168. Google Scholar
Digital Library
- Tunç Ozan Aydın, Rafał Mantiuk, Karol Myszkowski, and Hans-Peter Seidel. 2008. Dynamic range independent image quality assessment. In SIGGRAPH. Google ScholarDigital Library
- Soonmin Bae, Aseem Agarwala, and Frédo Durand. 2010. Computational rephotography. ACM Transactions on Graphics 29, 3, 24:1--24:15.Google ScholarDigital Library
- Simon Baker, Daniel Scharstein, J. P. Lewis, Stefan Roth, Michael J. Black, and Richard Szeliski. 2011. A database and evaluation methodology for optical flow. IJCV 92, 1, 1--31.Google ScholarDigital Library
- Kai Berger, Christian Lipski, Christian Linz, Anita Sellent, and Marcus Magnor. 2010. A ghosting artifact detector for interpolated image quality assessment. In International Symposium on Consumer Electronics. Google ScholarCross Ref
- Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael F. Cohen. 2001. Unstructured Lumigraph rendering. In SIGGRAPH. Google ScholarDigital Library
- Fatih Calakli, Ali O. Ulusoy, Maria I. Restrepo, Gabriel Taubin, and Joseph L. Mundy. 2012. High resolution surface reconstruction from multi-view aerial imagery. In 3DIMPVT.Google Scholar
- Neill D. Campbell, George Vogiatzis, Carlos Hernández, and Roberto Cipolla. 2008. Using multiple hypotheses to improve depth-maps for multi-view stereo. In ECCV. Google ScholarDigital Library
- Scott Daly. 1993. The visible differences predictor: An algorithm for the assessment of image fidelity. In Digital Images and Human Vision.Google Scholar
- Andrew Fitzgibbon, Yonatan Wexler, and Andrew Zisserman. 2005. Image-based rendering using image-based priors. IJCV 63, 2, 141--151. Google ScholarDigital Library
- Wolfgang Förstner. 1996. 10 pros and cons against performance characterization of vision algorithms. In Workshop on Performance Characteristics of Vision Algorithms.Google Scholar
- Simon Fuhrmann and Michael Goesele. 2011. Fusion of depth maps with multiple scales. In SIGGRAPH Asia. Google ScholarDigital Library
- Simon Fuhrmann, Fabian Langguth, Nils Moehrle, Michael Waechter, and Michael Goesele. 2015. MVE -- An image-based reconstruction environment. Computers 8 Graphics 53, Part A (2015).Google Scholar
- Yasutaka Furukawa, Brian Curless, Steven M. Seitz, and Richard Szeliski. 2010. Towards internet-scale multi-view stereo. In CVPR. Google ScholarCross Ref
- Yasutaka Furukawa and Jean Ponce. 2010. Accurate, dense, and robust multi-view stereopsis. PAMI 32, 8, 1362--1376. Google ScholarDigital Library
- Michael Goesele, Noah Snavely, Brian Curless, Hugues Hoppe, and Steven M. Seitz. 2007. Multi-view stereo for community photo collections. In ICCV. Google ScholarCross Ref
- Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The Lumigraph. In SIGGRAPH. Google ScholarDigital Library
- Stefan Guthe, Douglas Cunningham, Pascal Schardt, and Michael Goesele. 2016. Ghosting and popping detection for image-based rendering. In 3DTV Conference. Google ScholarCross Ref
- Tom Haber, Christian Fuchs, Philippe Bekaert, Hans-Peter Seidel, Michael Goesele, and Hendrik P. A. Lensch. 2009. Relighting objects from image collections. In CVPR. Google ScholarCross Ref
- Christian Hofsetz, Kim Ng, George Chen, Peter McGuinness, Nelson Max, and Yang Liu. 2004. Image-based rendering of range data with estimated depth uncertainty. Computer Graphics and Applications 24, 4, 34--42. Google ScholarDigital Library
- Yuan Hongxing, Guo Li, Yu Li, and Cheng Long. 2010. Multi-view reconstruction using band graph-cuts. Journal of Computer-Aided Design 8 Computer Graphics 4 (2010).Google Scholar
- Christof Hoppe, Manfred Klopschitz, Markus Rumpler, Andreas Wendel, Stefan Kluckner, Horst Bischof, and Gerhard Reitmayr. 2012a. Online feedback for structure-from-motion image acquisition. In BMVC. Google ScholarCross Ref
- Christof Hoppe, Andreas Wendel, Stefanie Zollmann, Katrin Pirker, Arnold Irschara, Horst Bischof, and Stefan Kluckner. 2012b. Photogrammetric camera network design for micro aerial vehicles. In Computer Vision Winter Workshop.Google Scholar
- Michael Kazhdan, Matthew Bolitho, and Hugues Hoppe. 2006. Poisson surface reconstruction. In SGP.Google Scholar
- Johannes Kopf, Michael F. Cohen, and Richard Szeliski. 2014. First-person hyper-lapse videos. In SIGGRAPH. Google ScholarDigital Library
- Yvan G. Leclerc, Quang-Tuan Luong, and Pascal Fua. 2000. Measuring the self-consistency of stereo algorithms. In ECCV. Google ScholarCross Ref
- Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In SIGGRAPH. Google ScholarDigital Library
- Rafał Mantiuk. 2013. Quantifying image quality in graphics: Perspective on subjective and objective metrics and their performance. In SPIE, Vol. 8651.Google ScholarCross Ref
- Rafał Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A calibrated visual metric for visibility and quality predictions in all luminance conditions. In SIGGRAPH.Google Scholar
- Paul Merrell, Amir Akbarzadeh, Liang Wang, Philippos Mordohai, Jan-Michael Frahm, Ruigang Yang, David Nistér, and Marc Pollefeys. 2007. Real-time visibility-based fusion of depth maps. In ICCV. Google ScholarCross Ref
- Ken Perlin. 2002. Improving noise. In SIGGRAPH. Google ScholarDigital Library
- Jens Preiss, Felipe Fernandes, and Philipp Urban. 2014. Color-image quality assessment: From prediction to optimization. IEEE Transactions on Image Processing 23, 3, 1366--1378. Google ScholarDigital Library
- Ganesh Ramanarayanan, James Ferwerda, Bruce Walter, and Kavita Bala. 2007. Visual equivalence: Towards a new standard for image fidelity. In SIGGRAPH.Google Scholar
- Michael Schwarz and Marc Stamminger. 2009. On predicting visual popping in dynamic scenes. In Applied Perception in Graphics and Visualization. Google ScholarDigital Library
- Steven M. Seitz, Brian Curless, James Diebel, Daniel Scharstein, and Richard Szeliski. 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms. In CVPR. Google ScholarDigital Library
- Qi Shan, Riley Adams, Brian Curless, Yasutaka Furukawa, and Steven M. Seitz. 2013. The visual Turing test for scene reconstruction. In 3DV.Google Scholar
- Noah Snavely, Steven M. Seitz, and Richard Szeliski. 2006. Photo tourism: Exploring photo collections in 3D. In SIGGRAPH.Google Scholar
- Christoph Strecha, Wolfgang von Hansen, Luc Van Gool, Pascal Fua, and Ulrich Thoennessen. 2008. On benchmarking camera calibration and multi-view stereo for high resolution imagery. In CVPR. Google ScholarCross Ref
- Richard Szeliski. 1999. Prediction error as a quality metric for motion and stereo. In ICCV. Google ScholarCross Ref
- James Tompkin, Min H. Kim, Kwang In Kim, Jan Kautz, and Christian Theobalt. 2013. Preference and artifact analysis for video transitions of places. ACM Transactions on Applied Perception 10, 3, 13:1--13:19.Google ScholarCross Ref
- Kathleen Tuite, Noah Snavely, Dun-yu Hsiao, Nadine Tabing, and Zoran Popović. 2011. PhotoCity: Training experts at large-scale image acquisition through a competitive game. In SIGCHI.Google Scholar
- Peter Vangorp, Gaurav Chaurasia, Pierre-Yves Laffont, Roland Fleming, and George Drettakis. 2011. Perception of visual artifacts in image-based rendering of façades. In Eurographics Symposium on Rendering. Google ScholarDigital Library
- Peter Vangorp, Christian Richardt, Emily A. Cooper, Gaurav Chaurasia, Martin S. Banks, and George Drettakis. 2013. Perception of perspective distortions in image-based rendering. In SIGGRAPH. Google ScholarDigital Library
- Kenneth Vanhoey, Basile Sauvage, Pierre Kraemer, Frédéric Larue, and Jean-Michel Dischler. 2015. Simplification of meshes with digitized radiance. The Visual Computer 31, 6, 1011--1021. Google ScholarDigital Library
- Michael Waechter, Nils Moehrle, and Michael Goesele. 2014. Let there be color! Large-scale texturing of 3D reconstructions. In ECCV.Google Scholar
- Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4, 600--612. Google ScholarDigital Library
- Robert H. Webb, Diane E. Boyer, and Raymond M. Turner. 2010. Repeat Photography. Island Press.Google Scholar
- Yizhou Yu, Paul Debevec, Jitendra Malik, and Tim Hawkins. 1999. Inverse global illumination: Recovering reflectance models of real scenes from photographs. In SIGGRAPH.Google Scholar
- Ramin Zabih and John Woodfill. 1994. Non-parametric local transforms for computing visual correspondence. In ECCV. Google ScholarCross Ref
- Matthias Zwicker, Hanspeter Pfister, Jeroen van Baar, and Markus Gross. 2001. Surface splatting. In SIGGRAPH. Google ScholarDigital Library
Index Terms
- Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
Recommendations
Virtual Rephotography: Novel View Prediction Error for 3D Reconstruction
The ultimate goal of many image-based modeling systems is to render photo-realistic novel views of a scene without visible artifacts. Existing evaluation metrics and benchmarks focus mainly on the geometric accuracy of the reconstructed model, which is, ...
Image-based rendering for scenes with reflections
We present a system for image-based modeling and rendering of real-world scenes containing reflective and glossy surfaces. Previous approaches to image-based rendering assume that the scene can be approximated by 3D proxies that enable view ...
An introduction to image-based rendering
Integrated image and graphics technologiesIn this chapter, we review the techniques for image-based rendering. Unlike traditional 3D computer graphics in which 3D geometry of the scene is known, image-based rendering (IBR) techniques render novel views directly from input images. IBR techniques ...
Comments