skip to main content
research-article

3DTV at home: eulerian-lagrangian stereo-to-multiview conversion

Published: 20 July 2017 Publication History

Abstract

Stereoscopic 3D (S3D) movies have become widely popular in the movie theaters, but the adoption of S3D at home is low even though most TV sets support S3D. It is widely believed that S3D with glasses is not the right approach for the home. A much more appealing approach is to use automulti-scopic displays that provide a glasses-free 3D experience to multiple viewers. A technical challenge is the lack of native multiview content that is required to deliver a proper view of the scene for every viewpoint. Our approach takes advantage of the abundance of stereoscopic 3D movies. We propose a real-time system that can convert stereoscopic video to a high-quality multiview video that can be directly fed to automultiscopic displays. Our algorithm uses a wavelet-based decomposition of stereoscopic images with per-wavelet disparity estimation. A key to our solution lies in combining Lagrangian and Eulerian approaches for both the disparity estimation and novel view synthesis, which leverages the complementary advantages of both techniques. The solution preserves all the features of Eulerian methods, e.g., subpixel accuracy, high performance, robustness to ambiguous depth cases, and easy integration of inter-view aliasing while maintaining the advantages of Lagrangian approaches, e.g., robustness to large disparities and possibility of performing non-trivial disparity manipulations through both view extrapolation and interpolation. The method achieves real-time performance on current GPUs. Its design also enables an easy hardware implementation that is demonstrated using a field-programmable gate array. We analyze the visual quality and robustness of our technique on a number of synthetic and real-world examples. We also perform a user experiment which demonstrates benefits of the technique when compared to existing solutions.

Supplementary Material

ZIP File (a146-kellnhofer.zip)
Supplemental files.
MP4 File (papers-0172.mp4)

References

[1]
Robert Anderson, David Gallup, Jonathan T. Barron, Janne Kontkanen, Noah Snavely, Carlos Hernández, Sameer Agarwal, and Steven M. Seitz. 2016. Jump: Virtual reality video. ACM Trans. Graph. 35, 6, Article 198 (Nov. 2016), 13 pages.
[2]
Myron Z Brown, Darius Burschka, and Gregory D Hager. 2003. Advances in computational stereo. IEEE Trans. on Pattern Analysis and Machine Intelligence 25, 8 (2003), 993--1008.
[3]
Alexandre Chapiro, Simon Heinzle, Tunç Ozan Aydin, Steven Poulakos, Matthias Zwicker, Aljosa Smolic, and Markus Gross. 2014. Optimizing stereo-to-multiview conversion for autostereoscopic displays. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 63--72.
[4]
Chris Chinnock. 2012. Trends in the 3D TV market. In Handbook of Visual Display Technology. Springer, 2599--2606.
[5]
Piotr Didyk, Tobias Ritschel, Elmar Eisemann, Karol Myszkowski, Hans-Peter Seidel, and Wojciech Matusik. 2012. A luminance-contrast-aware disparity model and applications. ACM Trans. Graph. 31, 6 (2012), 184.
[6]
Piotr Didyk, Pitchaya Sitthi-Amorn, William Freeman, Frédo Durand, and Wojciech Matusik. 2013. Joint view expansion and filtering for automultiscopic 3D displays. ACM Trans. Graph. 32, 6 (2013), 221.
[7]
Song-Pei Du, Piotr Didyk, Frédo Durand, Shi-Min Hu, and Wojciech Matusik. 2014. Improving visual quality of view transitions in automultiscopic displays. ACM Trans. Graph. 33, 6 (2014), 192:1--192:9.
[8]
Ye Fan, Joshua Litven, David IW Levin, and Dinesh K Pai. 2013. Eulerian-on-Lagrangian simulation. ACM Trans. Graph. 32, 3 (2013), 22:1--22:9.
[9]
Miquel Farre, Oliver Wang, Manuel Lang, Nikolce Stefanoski, Alexander Hornung, and Aljoscha Smolic. 2011. Automatic content creation for multiview autostereoscopic displays using image domain warping. In IEEE International Conference on Multimedia and Expo.
[10]
David J Fleet and Allan D Jepson. 1990. Computation of component image velocity from local phase information. International Journal of Computer Vision 5, 1 (1990), 77--104.
[11]
David J Fleet, Allan D Jepson, and Michael RM Jenkin. 1991. Phase-based disparity measurement. CVGIP: Image Understanding 53, 2 (1991), 198--210.
[12]
John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2015. DeepStereo: Learning to predict new views from the world's imagery. arXiv preprint arXiv:1506.06825 (2015).
[13]
Andrea Fusiello, Emanuele Trucco, and Alessandro Verri. 2000. A compact algorithm for rectification of stereo pairs. Machine Vision and Applications 12, 1 (2000), 16--22.
[14]
Samuel W Hasinoff, Sing Bing Kang, and Richard Szeliski. 2006. Boundary matting for view synthesis. Computer Vision and Image Understanding 103, 1 (2006), 22--32.
[15]
Heiko Hirschmuller and Daniel Scharstein. 2007. Evaluation of cost functions for stereo matching. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on. IEEE, 1--8.
[16]
Asmaa Hosni, Christoph Rhemann, Michael Bleyer, Carsten Rother, and Margrit Gelautz. 2013. Fast cost-volume filtering for visual correspondence and beyond. IEEE Trans. on Pattern Analysis and Machine Intelligence 35, 2 (2013), 504--511.
[17]
Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based view synthesis for light field cameras. ACM Trans. Graph. (Proc. of SIGGRAPH Asia 2016) 35, 6 (2016).
[18]
Johannes Kopf, Fabian Langguth, Daniel Scharstein, Richard Szeliski, and Michael Goesele. 2013. Image-based rendering in the gradient domain. ACM Trans. Graph. 32, 6 (2013), 199.
[19]
Manuel Lang, Alexander Hornung, Oliver Wang, Steven Poulakos, Aljoscha Smolic, and Markus Gross. 2010. Nonlinear disparity mapping for stereoscopic 3D. ACM Trans. Graph. 29, 4 (2010), 75:1--75:10.
[20]
Chao-Kang Liao, Hsiu-Chi Yeh, Ke Zhang, Vanmeerbeeck Geert, Tian-Sheuan Chang, and Gauthier Lafruit. 2013. Stereo matching and viewpoint synthesis FPGA implementation. In 3D-TV System with Depth-Image-Based Rendering. Springer, 69--106.
[21]
QH Liu and N Nguyen. 1998. An accurate algorithm for nonuniform fast Fourier transforms (NUFFT's). IEEE Microwave and Guided Wave Letters 8, 1 (1998), 18--20.
[22]
Lytro Inc. 2015. (January 2015). https://www.lytro.com/.
[23]
William R Mark, Leonard McMillan, and Gary Bishop. 1997. Post-rendering 3D warping. In Proc. of the 1997 Symposium on Interactive 3D Graphics. ACM, 7-ff.
[24]
Belen Masia, Gordon Wetzstein, Carlos Aliaga, Ramesh Raskar, and Diego Gutierrez. 2013. Display adaptive 3D content remapping. Computers & Graphics, Special Issue on Advanced Displays 37, 6 (2013), 983--996.
[25]
Takuya Matsuo, Norishige Fukushima, and Yutaka Ishibashi. 2013. Weighted joint bilateral filter with slope depth compensation filter for depth map refinement. In VISAPP (2). 300--309.
[26]
Wojciech Matusik and Hanspeter Pfister. 2004. 3D TV: A scalable system for real-time acquisition, transmission, and autostereoscopic display of dynamic scenes. ACM Trans. Graph. 23, 3 (2004), 814--824.
[27]
Lydia MJ Meesters, Wijnand A IJsselsteijn, and Piter JH Seuntiens. 2004. A survey of perceptual evaluations and requirements of three-dimensional TV. IEEE Trans. on Circuits and Systems for Video Technology 14, 3 (2004), 381--391.
[28]
H Keith Nishihara. 1984. Practical real-time imaging stereo matcher. Optical Engineering 23, 5 (1984), 235536--235536.
[29]
Karl Pauwels and Marc M Van Hulle. 2008. Realtime phase-based optical flow on the GPU. In Computer Vision and Pattern Recognition Workshops, 2008. CVPRW'08. IEEE Computer Society Conference on. IEEE, 1--8.
[30]
Raytrix GmbH. 2015. (January 2015). http://www.raytrix.de/.
[31]
Christian Richardt, Carsten Stoll, Neil A Dodgson, Hans-Peter Seidel, and Christian Theobalt. 2012. Coherent spatiotemporal filtering, upsampling and rendering of RGBZ videos. In Computer Graphics Forum, Vol. 31. Wiley Online Library, 247--256.
[32]
Christian Riechert, Frederik Zilly, Peter Kauff, Jens Güther, and Ralf Schäfer. 2012. Fully automatic stereo-to-multiview conversion in autostereoscopic displays. The Best of IET and IBC 4, 8 (2012), 14.
[33]
Michael Schaffner, Frank Gurkaynak, Pierre Greisen, Hubert Kaeslin, Luca Benini, and Aljosa Smolic. 2015. Hybrid ASIC/FPGA system for fully automatic stereo-to-multiview conversion using IDW. Circuits and Systems for Video Technology, IEEE Trans. on (2015).
[34]
T. Shibata, J. Kim, D.M. Hoffman, and M.S. Banks. 2011. The zone of comfort: Predicting visual discomfort with stereo displays. Journal of Vision 11, 8 (2011), 11:1--11:29.
[35]
Eero P Simoncelli and William T Freeman. 1995. The steerable pyramid: A flexible architecture for multi-scale derivative computation. In Image Processing, International Conference on, Vol. 3. IEEE Computer Society, 3444--3444.
[36]
Eero P Simoncelli, William T Freeman, Edward H Adelson, and David J Heeger. 1992. Shiftable multiscale transforms. IEEE Trans. on Information Theory 38, 2 (1992), 587--607.
[37]
Sudipta N Sinha, Drew Steedly, and Richard Szeliski. 2009. Piecewise planar stereo for image-based rendering. In ICCV. 1881--1888.
[38]
Aljoscha Smolic, Karsten Muller, Kristina Dix, Philipp Merkle, Peter Kauff, and Thomas Wiegand. 2008. Intermediate view interpolation based on multiview video plus depth for advanced 3D video systems. In IEEE International Conference on Image Processing. 2448--2451.
[39]
Nikolce Stefanoski, Oliver Wang, Michael Lang, Pierre Greisen, Simon Heinzle, and Aljoscha Smolic. 2013. Automatic view synthesis by image-domain-warping. Image Processing, IEEE Trans. on 22, 9 (2013), 3329--3341.
[40]
Richard Szeliski, Shai Avidan, and P Anandan. 2000. Layer extraction from multiple images containing reflections and transparency. In Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, Vol. 1. IEEE, 246--253.
[41]
Neal Wadhwa, Michael Rubinstein, Frédo Durand, and William T Freeman. 2013. Phase-based video motion processing. ACM Trans. Graph. 32, 4 (2013), 80:1--80:10.
[42]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. Image Processing, IEEE Trans. on 13, 4 (2004), 600--612.
[43]
Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez, Adam Barth, Andrew Adams, Mark Horowitz, and Marc Levoy. 2005. High performance imaging using large camera arrays. In ACM Trans. Graph., Vol. 24. ACM, 765--776.
[44]
Bennett S Wilburn, Michal Smulski, Hsiao-Heng K Lee, and Mark A Horowitz. 2001. Light field video camera. In Electronic Imaging 2002. International Society for Optics and Photonics, 29--36.
[45]
Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John V Guttag, Frédo Durand, and William T Freeman. 2012. Eulerian video magnification for revealing subtle changes in the world. (2012).
[46]
Zhoutong Zhang, Yebin Liu, and Qionghai Dai. 2015. Light field from micro-baseline image pair. In Proc. of the IEEE Conference on Computer Vision and Pattern Recognition. 3800--3809.
[47]
Jun Zhou, Yi Xu, and Xiaokang Yang. 2007. Quaternion wavelet phase based stereo matching for uncalibrated images. Pattern Recognition Letters 28, 12 (2007), 1509--1522.
[48]
C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2004. High-quality video view interpolation using a layered representation. In ACM Trans. Graph., Vol. 23. ACM, 600--608.
[49]
Matthias Zwicker, Wojciech Matusik, Frédo Durand, and Hanspeter Pfister. 2006. Antialiasing for automultiscopic 3D displays. In Proc. of the 17th Eurographics Conference on Rendering Techniques. Eurographics Association, 73--82.

Cited By

View all
  • (2023)The Embodiment and Innovation of Digital Twin Platform in Modern Interior Environment DesignApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.2.01420Online publication date: 9-Dec-2023
  • (2023)Modified 2D-Ghost-Free Stereoscopic Display with Depth-of-Field EffectsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353496419:1s(1-16)Online publication date: 23-Jan-2023
  • (2022)HiddenGazeStereo: Hiding Gaze-Contingent Disparity Remapping for 2D-Compatible Natural 3D ViewingIEEE Access10.1109/ACCESS.2022.320487410(94778-94796)Online publication date: 2022
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 36, Issue 4
August 2017
2155 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3072959
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017
Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. automultiscopic displays
  2. interperspective antialiasing
  3. view synthesis
  4. view transitions

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)8
  • Downloads (Last 6 weeks)2
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2023)The Embodiment and Innovation of Digital Twin Platform in Modern Interior Environment DesignApplied Mathematics and Nonlinear Sciences10.2478/amns.2023.2.01420Online publication date: 9-Dec-2023
  • (2023)Modified 2D-Ghost-Free Stereoscopic Display with Depth-of-Field EffectsACM Transactions on Multimedia Computing, Communications, and Applications10.1145/353496419:1s(1-16)Online publication date: 23-Jan-2023
  • (2022)HiddenGazeStereo: Hiding Gaze-Contingent Disparity Remapping for 2D-Compatible Natural 3D ViewingIEEE Access10.1109/ACCESS.2022.320487410(94778-94796)Online publication date: 2022
  • (2020)Neural control variatesACM Transactions on Graphics10.1145/3414685.341780439:6(1-19)Online publication date: 27-Nov-2020
  • (2020)PIEACM Transactions on Graphics10.1145/3414685.341780339:6(1-14)Online publication date: 27-Nov-2020
  • (2020)Neural holography with camera-in-the-loop trainingACM Transactions on Graphics10.1145/3414685.341780239:6(1-14)Online publication date: 27-Nov-2020
  • (2020)Bijective projection in a shellACM Transactions on Graphics10.1145/3414685.341776939:6(1-18)Online publication date: 27-Nov-2020
  • (2020)Constraining dense hand surface tracking with elasticityACM Transactions on Graphics10.1145/3414685.341776839:6(1-14)Online publication date: 27-Nov-2020
  • (2020)Mononizing binocular videosACM Transactions on Graphics10.1145/3414685.341776439:6(1-16)Online publication date: 27-Nov-2020
  • (2020)Local Fourier Slice PhotographyACM Transactions on Graphics10.1145/333930739:3(1-16)Online publication date: 21-Apr-2020
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media