ABSTRACT
The commoditization of virtual and augmented reality devices and the availability of inexpensive consumer depth cameras have catalyzed a resurgence of interest in spatiotemporal performance capture. Recent systems like Fusion4D and Holoportation address several crucial problems in the real-time fusion of multiview depth maps into volumetric and deformable representations. Nonetheless, stitching multiview video textures onto dynamic meshes remains challenging due to imprecise geometries, occlusion seams, and critical time constraints. In this paper, we present a practical solution towards real-time seamless texture montage for dynamic multiview reconstruction. We build on the ideas of dilated depth discontinuities and majority voting from Holoportation to reduce ghosting effects when blending textures. In contrast to their approach, we determine the appropriate blend of textures per vertex using view-dependent rendering techniques, so as to avert fuzziness caused by the ubiquitous normal-weighted blending. By leveraging geodesics-guided diffusion and temporal texture fields, our algorithm mitigates spatial occlusion seams while preserving temporal consistency. Experiments demonstrate significant enhancement in rendering quality, especially in detailed regions such as faces. We envision a wide range of applications for Montage4D, including immersive telepresence for business, training, and live entertainment.
Supplemental Material
Available for Download
Supplemental files.
- Cédric Allène, Jean-Philippe Pons, and Renaud Keriven. 2008. Seamless image-based texture atlases using multi-band blending. In 19th International Conference on Pattern Recognition. IEEE, 1--4.Google ScholarCross Ref
- Pierre Alliez, Mark Meyer, and Mathieu Desbrun. 2002. Interactive geometry remeshing. ACM Transactions on Graphics (TOG) 21, 3 (2002), 347--354. Google ScholarDigital Library
- David Bommes and Leif Kobbelt. 2007. Accurate computation of geodesic distance fields for polygonal curves on triangle meshes. In Computer Cision, Graphics and Visualization Workshop, Vol. 7. VMV, 151--160.Google Scholar
- Cedric Cagniart, Edmond Boyer, and Slobodan Ilic. 2010. Probabilistic deformable surface tracking from multiple videos. In ECCV'10 Proceedings of the 11th European Conference on Computer Vision: Part IV. Springer, 326--339. Google ScholarDigital Library
- Dan Casas, Margara Tejera, Jean-Yves Guillemaut, and Adrian Hilton. 2013. Interactive animation of 4D performance capture. IEEE Transactions on Visualization and Computer Graphics (TVCG) 19, 5 (2013), 762--773. Google ScholarDigital Library
- Dan Casas, Marco Volino, John Collomosse, and Adrian Hilton. 2014. 4D video textures for interactive character appearance. In Computer Graphics Forum, Vol. 33. Wiley Online Library, 371--380. Google ScholarDigital Library
- Matthäus G Chajdas, Morgan McGuire, and David Luebke. 2011. Subpixel reconstruction antialiasing for deferred shading. In Proceedings of Symposium on Interactive 3D Graphics and Games (I3D). ACM, 15--22. Google ScholarDigital Library
- Ming Chuang, Linjie Luo, Benedict J Brown, Szymon Rusinkiewicz, and Michael Kazhdan. 2009. Estimating the Laplace-Beltrami Operator by Restricting 3D Functions. In Computer graphics forum, Vol. 28. Wiley Online Library, 1475--1484. Google ScholarDigital Library
- Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. 2015. High-quality streamable free-viewpoint video. ACM Transactions on Graphics (TOG) 34, 4 (2015), 69. Google ScholarDigital Library
- Cyril Crassin, Morgan McGuire, Kayvon Fatahalian, and Aaron Lefohn. 2015. Aggregate G-buffer anti-aliasing. In Proceedings of the 19th Symposium on Interactive 3D Graphics and Games (I3D). ACM, 109--119. Google ScholarDigital Library
- Edilson De Aguiar, Carsten Stoll, Christian Theobalt, Naveed Ahmed, Hans-Peter Seidel, and Sebastian Thrun. 2008. Performance capture from sparse multi-view video. ACM Transactions on Graphics (TOG) 27, 3 (2008), 98. Google ScholarDigital Library
- Paul Debevec, Steven Gortler, Leonard McMillan, Richard Szeliski, and Chris Bregler. 1998a. Image-based modeling and rendering. SIGGRAPH 98 Course Notes for Course 15 (1998).Google Scholar
- Paul Debevec, Yizhou Yu, and George Borshukov. 1998b. Efficient view-dependent image-based rendering with projective texture-mapping. In Rendering Techniques. Springer, 105--116.Google Scholar
- Fernando do Goes, Mathieu Desbrun, and Yiying Tong. 2015. Vector field processing on triangle meshes. In SIGGRAPH Asia 2015 Courses. ACM, 17. Google ScholarDigital Library
- Mingsong Dou, Philip Davidson, Sean Ryan Fanello, Sameh Khamis, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, and Shahram Izadi. 2017. Motion2fusion: real-time volumetric performance capture. ACM Transactions on Graphics (TOG) 36, 6 (2017), 246. Google ScholarDigital Library
- Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al. 2016. Fusion4D: Real-time performance capture of challenging scenes. ACM Transactions on Graphics (TOG) 35, 4 (2016), 114. Google ScholarDigital Library
- Ruofei Du, Sujal Bista, and Amitabh Varshney. 2016. Video Fields: Fusing Multiple Surveillance Videos into a Dynamic Virtual Environment. In Proceedings of the 21st International Conference on Web3D Technology. ACM, 165--172. Google ScholarDigital Library
- Martin Eisemann, Bert De Decker, Marcus Magnor, Philippe Bekaert, Edilson De Aguiar, Naveed Ahmed, Christian Theobalt, and Anita Sellent. 2008. Floating Textures. Computer Graphics Forum 27, 2 (2008), 409--418.Google ScholarCross Ref
- Martin Eisemann and Marcus A Magnor. 2007. Filtered Blending: A new, minimal Reconstruction Filter for Ghosting-Free Projective Texturing with Multiple Images.. In VMV. 119--126.Google Scholar
- Sean Ryan Fanello, Julien Valentin, Adarsh Kowdle, Christoph Rhemann, Vladimir Tankovich, Carlo Ciliberto, Philip Davidson, and Shahram Izadi. 2017a. Low Compute and Fully Parallel Computer Vision with HashMatch. In IEEE International Conference on Computer Vision (ICCV). IEEE, 3894--3903.Google Scholar
- Sean Ryan Fanello, Julien Valentin, Christoph Rhemann, Adarsh Kowdle, Vladimir Tankovich, Philip Davidson, and Shahram Izadi. 2017b. UltraStereo: Efficient Learning-based Matching for Active Stereo Systems. In Computer Vision and Pattern Recognition (CVPR), 2017 IEEE Conference on. IEEE, 6535--6544.Google ScholarCross Ref
- Henry Fuchs, Gary Bishop, Kevin Arthur, Leonard McMillan, Ruzena Bajcsy, Sang Lee, Hany Farid, and Takeo Kanade. 1994. Virtual Space Teleconferencing Using a Sea of Cameras. In Proc. First International Conference on Medical Robotics and Computer Assisted Surgery (MRCAS), Vol. 26. TR94-033, 7. Google ScholarDigital Library
- H Fuchs and U Neumann. 1993. A vision of telepresence for medical consultation and other applications. In Sixth International Symposium of Robotics Research. IFRR, 555--571.Google Scholar
- Yasutaka Furukawa and Jean Ponce. 2008. Dense 3D Motion Capture from Synchronized Video Streams. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 1--8.Google Scholar
- Ran Gal, Yonatan Wexler, Eyal Ofek, Hugues Hoppe, and Daniel Cohen-Or. 2010. Seamless montage for texturing models. Computer Graphics Forum 29, 2 (2010), 479--486.Google ScholarCross Ref
- Bastian Goldluecke and Marcus Magnor. 2004. Space-Time Isosurface Evolution for Temporally Coherent 3D Reconstruction. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Vol. 1. IEEE, I-350.Google ScholarCross Ref
- Eitan Grinspun, Mathieu Desbrun, Konrad Polthier, Peter Schröder, and Ari Stern. 2006. Discrete differential geometry: an applied introduction. ACM SIGGRAPH Course 7 (2006), 1--139.Google Scholar
- Kaiwen Guo, Feng Xu, Yangang Wang, Yebin Liu, and Qionghai Dai. 2015. Robust non-rigid motion tracking and surface reconstruction using l0 regularization. In Proceedings of the IEEE International Conference on Computer Vision. IEEE, 3083--3091. Google ScholarDigital Library
- Kaiwen Guo, Feng Xu, Tao Yu, Xiaoyang Liu, Qionghai Dai, and Yebin Liu. 2017. Real-Time Geometry, Albedo, and Motion Reconstruction Using a Single RGB-D Camera. ACM Transactions on Graphics (TOG) 36, 3 (2017), 32. Google ScholarDigital Library
- Eric Heitz, Derek Nowrouzezahrai, Pierre Poulin, and Fabrice Neyret. 2013. Filtering color mapped textures and surfaces. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games (I3D). ACM, 129--136. Google ScholarDigital Library
- Shahram Izadi, David Kim, Otmar Hilliges, David Molyneaux, Richard Newcombe, Pushmeet Kohli, Jamie Shotton, Steve Hodges, Dustin Freeman, Andrew Davison, et al. 2011. KinectFusion: Real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology (UIST). ACM, 559--568. Google ScholarDigital Library
- Zsolt Janko and Jean-Philippe Pons. 2009. Spatio-temporal image-based texture atlases for dynamic 3-D models. In IEEE 12th International Conference on Computer Vision Workshops. IEEE, 1646--1653.Google ScholarCross Ref
- Takeo Kanade, Peter Rander, and PJ Narayanan. 1997. Virtualized Reality: Constructing Virtual Worlds from Real Scenes. IEEE Multimedia 4, 1 (1997), 34--47. Google ScholarDigital Library
- Takashi Kanai and Hiromasa Suzuki. 2001. Approximate shortest path on a polyhedral surface and its applications. Computer-Aided Design 33, 11 (2001), 801--811.Google ScholarCross Ref
- Sanjiv Kapoor. 1999. Efficient computation of geodesic shortest paths. In Proceedings of the Thirty-First Annual ACM Symposium on Theory of Computing. ACM, 770--779. Google ScholarDigital Library
- Mark Lanthier, Anil Maheshwari, and Jörg-Rüdiger Sack. 1997. Approximating weighted shortest paths on polyhedral surfaces. In Proceedings of the Thirteenth Annual Symposium on Computational Geometry. ACM, 274--283. Google ScholarDigital Library
- Victor Lempitsky and Denis Ivanov. 2007. Seamless mosaicing of image-based texture maps. In 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1--6.Google ScholarCross Ref
- Benjamin Lok. 2001. Online model reconstruction for interactive virtual environments. In Proceedings of the 2001 Symposium on Interactive 3D Graphics (I3D). ACM, 69--72. Google ScholarDigital Library
- Dimas Martinez, Luiz Velho, and Paulo Cezar Carvalho. 2004. Geodesic paths on triangular meshes. In Proceedings of Computer Graphics and Image Processing. IEEE, 8. Google ScholarDigital Library
- Pavlos Mavridis and Georgios Papaioannou. 2011. High quality elliptical texture filtering on GPU. In Symposium on Interactive 3D Graphics and Games (I3D). ACM, 23--30. Google ScholarDigital Library
- Joseph SB Mitchell. 2000. Geometric shortest paths and network optimization. Handbook of Computational Geometry 334 (2000), 633--702. http://www.ams.sunysb.edu/~jsbm/papers/survey.ps.gzGoogle ScholarCross Ref
- Joseph SB Mitchell, David M Mount, and Christos H Papadimitriou. 1987. The discrete geodesic problem. SIAM J. Comput. 16, 4 (1987), 647--668. Google ScholarDigital Library
- Karthik S Narayan and Pieter Abbeel. 2015. Optimized color models for high-quality 3D scanning. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2503--2510.Google ScholarCross Ref
- PJ Narayanan, Peter W Rander, and Takeo Kanade. 1998. Constructing Virtual Worlds Using Dense Stereo. In Sixth International Conference on Computer Vision (ICCV). IEEE, 3--10. Google ScholarDigital Library
- Richard A Newcombe, Dieter Fox, and Steven M Seitz. 2015. DynamicFusion: Reconstruction and Tracking of Non-Rigid Scenes in Real-Time. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 343--352.Google Scholar
- Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. 2011. KinectFusion: Real-time dense surface mapping and tracking. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 127--136. Google ScholarDigital Library
- Sergio Orts-Escolano, Christoph Rhemann, Sean Fanello, Wayne Chang, Adarsh Kowdle, Yury Degtyarev, David Kim, Philip L Davidson, Sameh Khamis, Mingsong Dou, et al. 2016. Holoportation: Virtual 3D Teleportation in Real-time. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST). ACM, 741--754. Google ScholarDigital Library
- Rob Patro, Cheuk Yiu Ip, Sujal Bista, and Amitabh Varshney. 2011. Social Snapshot: A system for temporally coupled social photography. IEEE Computer Graphics and Applications 31, 1 (2011), 74--84. Google ScholarDigital Library
- Patrick Pérez, Michel Gangnet, and Andrew Blake. 2003. Poisson image editing. ACM Transactions on graphics (TOG) 22, 3 (2003), 313--318. Google ScholarDigital Library
- Fabián Prada, Misha Kazhdan, Ming Chuang, Alvaro Collet, and Hugues Hoppe. 2016. Motion graphs for unstructured textured meshes. ACM Transactions on Graphics (TOG) 35, 4 (2016), 108. Google ScholarDigital Library
- Fabián Prada, Misha Kazhdan, Ming Chuang, Alvaro Collet, and Hugues Hoppe. 2017a. Spatiotemporal atlas parameterization for evolving meshes. ACM Transactions on Graphics (TOG) 36, 4 (2017), 58. Google ScholarDigital Library
- Fabián Prada, Misha Kazhdan, Ming Chuang, Alvaro Collet, and Hugues Hoppe. 2017b. Spatiotemporal atlas parameterization for evolving meshes. ACM Transactions on Graphics (TOG) 36, 4 (2017), 58. Google ScholarDigital Library
- Yipeng Qin, Xiaoguang Han, Hongchuan Yu, Yizhou Yu, and Jianjun Zhang. 2016. Fast and exact discrete geodesic computation based on triangle-oriented wavefront propagation. ACM Transactions on Graphics (TOG) 35, 4 (2016), 13. Google ScholarDigital Library
- Lijun Qu and Gary W Meyer. 2006. Perceptually driven interactive geometry remeshing. In Proceedings of the 2006 Symposium on Interactive 3D Graphics and Games (I3D). ACM, 199--206. Google ScholarDigital Library
- Peter Shirley, Timo Aila, Jonathan Cohen, Eric Enderton, Samuli Laine, David Luebke, and Morgan McGuire. 2011. A local image reconstruction algorithm for stochastic rendering. In Symposium on Interactive 3D Graphics and Games (I3D). ACM, PAGE-5. Google ScholarDigital Library
- Vitaly Surazhsky, Tatiana Surazhsky, Danil Kirsanov, Steven J Gortler, and Hugues Hoppe. 2005. Fast exact and approximate geodesics on meshes. ACM transactions ongraphics (TOG) 24, 3 (2005), 553--560. Google ScholarDigital Library
- Herman Towles, Wei-Chao Chen, Ruigang Yang, Sang-Uok Kum, Henry Fuchs Nikhil Kelshikar, Jane Mulligan, Kostas Daniilidis, Henry Fuchs, Carolina Chapel Hill, Nikhil Kelshikar Jane Mulligan, et al. 2002. 3D tele-collaboration over Internet2. In International Workshop on Immersive Telepresence. Juan Les Pins, France, 6.Google Scholar
- Vibhav Vineet, Jonathan Warrell, and Philip HS Torr. 2014. Filter-based mean-field inference for random fields with higher-order terms and product label-spaces. International Journal of Computer Vision 110, 3 (2014), 290--307. Google ScholarDigital Library
- Daniel Vlasic, Ilya Baran, Wojciech Matusik, and Jovan Popović. 2008. Articulated mesh animation from multi-view silhouettes. ACM Transactions on Graphics (TOG) 27, 3 (2008), 97. Google ScholarDigital Library
- Marco Volino, Dan Casas, John Collomosse, and Adrian Hilton. 2014. Optimal Representation of Multiple View Video. In Proceedings of the British Machine Vision Conference. BMVA Press.Google ScholarCross Ref
- Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612. Google ScholarDigital Library
- Feng Xu, Yebin Liu, Carsten Stoll, James Tompkin, Gaurav Bharaj, Qionghai Dai, Hans-Peter Seidel, Jan Kautz, and Christian Theobalt. 2011. Video-based characters: creating new human performances from a multi-view video database. ACM Transactions on Graphics (TOG) 30, 4 (2011), 32. Google ScholarDigital Library
- Weipeng Xu, Mathieu Salzmann, Yongtian Wang, and Yue Liu. 2015. Deformable 3D Fusion: From Partial Dynamic 3D Observations to Complete 4D Models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV). IEEE, 2183--2191. Google ScholarDigital Library
- Mao Ye and Ruigang Yang. 2014. Real-Time Simultaneous Pose and Shape Estimation for Articulated Objects Using a Single Depth Camera. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2345--2352. Google ScholarDigital Library
- Tao Yu, Kaiwen Guo, Feng Xu, Yuan Dong, Zhaoqi Su, Jianhui Zhao, Jianguo Li, Qionghai Dai, and Yebin Liu. 2017. BodyFusion: Real-time Capture of Human Motion and Surface Geometry Using a Single Depth Camera. In The IEEE International Conference on Computer Vision (ICCV). ACM.Google ScholarCross Ref
- Qing Zhang, Bo Fu, Mao Ye, and Ruigang Yang. 2014. Quality dynamic human body modeling using a single low-cost depth camera. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 676--683. Google ScholarDigital Library
- Kun Zhou, Xi Wang, Yiying Tong, Mathieu Desbrun, Baining Guo, and Heung-Yeung Shum. 2005. TextureMontage: Seamless texturing of arbitrary surfaces from multiple images. ACM Transactions on Graphics (TOG) 24, 3 (2005), 1148--1155. Google ScholarDigital Library
- Qian-Yi Zhou and Vladlen Koltun. 2014. Color map optimization for 3D reconstruction with consumer depth cameras. ACM Transactions on Graphics (TOG) 33, 4 (2014), 155. Google ScholarDigital Library
- C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2004. High-quality video view interpolation using a layered representation. In ACM Transactions on Graphics (TOG), Vol. 23. ACM, 600--608. Google ScholarDigital Library
Index Terms
- Montage4D: interactive seamless fusion of multiview video textures
Recommendations
Rendering fake soft shadows with smoothies
EGRW '03: Proceedings of the 14th Eurographics workshop on RenderingWe present a new method for real-time rendering of shadows in dynamic scenes. Our approach builds on the shadow map algorithm by attaching geometric primitives that we call "smoothies" to the objects' silhouettes. The smoothies give rise to fake shadows ...
Reflection reprojection using temporal coherence
A powerful approach for rendering high-quality images at low cost is to exploit temporal coherence by projecting already computed images into a novel view. However, conventional temporal coherence projection methods assume pixel values remain almost ...
Rendering intricately-shaped objects using view-dependent microfacets
When a real object is displayed in a virtual space by using computer graphics, the general approach is to draw the image on the basis of the geometrical information and the texture information about the object. However, when the object has a complex and ...
Comments