Shuai Guo
ABSTRACT
Free viewpoint video (FVV) has drawn great attention in recent years, which provides viewers with strong interactive and immersive experience. Despite the developments made, further progress of FVV research is limited by existing datasets that mostly have too few number of camera views, or static scenes. To overcome the limitations, in this paper, we present a new dynamic RGB-D video dataset with up to 12 views. Our dataset consists of 13 groups of dynamic video sequences that are taken at the same scene, and a group of video sequences of the empty scene. Each group has 12 HD video sequences taken by synchronized cameras and 12 correspondingly estimated depth video sequences. Moreover, we also introduce a FVV synthesis benchmark on the basis of depth image based rendering (DIBR) to help researchers validate their data-driven methods. We hope our work will inspire more FVV synthesis methods with enhanced robustness, improved performance and deeper understanding.
- Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. 2016. Large-scale data for multiple-view stereopsis. International Journal of Computer Vision 120, 2 (2016), 153--168.Google Scholar
Digital Library
- Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan. 2020. 4d visualization of dynamic events from unconstrained multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5366--5375.Google ScholarCross Ref
- M Domański, A Dziembowski, M Kurc, A Łuczak, D Mieloch, J Siast, O Stankiewicz, and K Wegner. 2015. Poznan University of Technology test multiview video sequences acquired with circular camera arrangement'Poznan Team'and'Poznan Blocks' sequences. ISO/IEC JTC1/SC29/WG11, Doc. MPEG M 35846 (2015).Google Scholar
- Marek Domaski, T. Grajek, K. Klimaszewski, M. Kurc, and K. Wegner. 2009. Contribution Poznań Multiview Video Test Sequences and Camera Parameters. (2009).Google Scholar
- M Domański, A. Dziembowski, T. Grajek, A. Grzelka, and K. Wegner. 2015. [FTV AHG] Video and depth multiview test sequences acquired with circular camera arrangement - "Poznan Service" and "Poznan People". (2015).Google Scholar
- M Domański, A. Dziembowski, A. Grzelka, D. Mieloch, and K. Wegner. 2016. Multiview test video sequences for free navigation exploration obtained using pairs of cameras. In ISO/IEC JTC1/SC29/WG11 MPEG2016/ m38247.Google Scholar
- Christoph Fehn. 2004. Depth-image-based rendering (DIBR), compression, and transmission for a new approach on 3D-TV. In Stereoscopic Displays and Virtual Reality Systems XI, Vol. 5291. International Society for Optics and Photonics, 93--104.Google ScholarCross Ref
- John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. 2019. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2367--2376.Google ScholarCross Ref
- Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. 2017. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. ACM Transactions on Graphics 36, 4 (2017).Google ScholarDigital Library
- Chuen-Chien Lee, Ali Tabatabai, and Kenji Tashiro. 2015. Free viewpoint video (FVV) survey and future research direction. APSIPA Transactions on Signal and Information Processing 4 (2015).Google Scholar
- Chuen-Chien Lee, Ali Tabatabai, and Kenji Tashiro. 2015. Free viewpoint video (FVV) survey and future research direction. APSIPA Transactions on Signal and Information Processing 4 (2015).Google Scholar
- Qinbo Li and Nima Khademi Kalantari. 2020. Synthesizing light field from a single image with variable MPI and two network fusion. ACM Trans. Graph. 39, 6 (2020), 229--1.Google ScholarDigital Library
- Kai-En Lin, Lei Xiao, Feng Liu, Guowei Yang, and Ravi Ramamoorthi. 2021. Deep 3D Mask Volume for View Synthesis of Dynamic Scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 1749--1758.Google ScholarCross Ref
- Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. 2019. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) 38, 4 (2019), 1--14.Google ScholarDigital Library
- OpenCV. 2022. Camera Calibration. https://docs.opencv.org/3.4/dc/dbb/tutorial_py_calibration.html.Google Scholar
- Néill O'Dwyer, Jan Ondřej, Rafael Pagés, Konstantinos Amplianitis, and Aljoša Smolić. 2018. Jonathan Swift: augmented reality application for Trinity library's long room. In International Conference on Interactive Digital Storytelling. Springer, 348--351.Google ScholarDigital Library
- Rafael Pagés, Konstantinos Amplianitis, David Monaghan, Jan Ondřej, and Aljosa Smolić. 2018. Affordable content creation for free-viewpoint video and VR/AR applications. Journal of Visual Communication and Image Representation 53 (2018), 192--201.Google ScholarCross Ref
- Gernot Riegler and Vladlen Koltun. 2020. Free view synthesis. In European Conference on Computer Vision. Springer, 623--640.Google ScholarDigital Library
- Johannes Lutz Schönberger and Jan-Michael Frahm. 2016. Structure-from-Motion Revisited. In Conference on Computer Vision and Pattern Recognition (CVPR).Google Scholar
- Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys, and Jan-Michael Frahm. 2016. Pixelwise View Selection for Unstructured Multi-View Stereo. In European Conference on Computer Vision (ECCV).Google Scholar
- Thomas Schops, Johannes L Schonberger, Silvano Galliani, Torsten Sattler, Konrad Schindler, Marc Pollefeys, and Andreas Geiger. 2017. A multi-view stereo benchmark with high-resolution images and multi-camera videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3260--3269.Google ScholarCross Ref
- Jangwoo Son, Serhan Gül, Gurdeep Singh Bhullar, Gabriel Hege, Wieland Morgenstern, Anna Hilsmann, Thomas Ebner, Sven Bliedung, Peter Eisert, Thomas Schierl, et al. 2020. Split Rendering for Mixed Reality: Interactive Volumetric Video in Action. In SIGGRAPH Asia 2020 XR. 1--3.Google Scholar
- Olgierd Stankiewicz, Marek Domański, Adrian Dziembowski, Adam Grzelka, Dawid Mieloch, and Jarosław Samelak. 2018. A free-viewpoint television system for horizontal virtual navigation. IEEE Transactions on Multimedia 20, 8 (2018), 2182--2195.Google ScholarCross Ref
- Yanru Wang, Zhihao Huang, Hao Zhu, Wei Li, Xun Cao, and Ruigang Yang. 2020. Interactive free-viewpoint video generation. Virtual Reality & Intelligent Hardware 2, 3 (2020), 247--260.Google ScholarCross Ref
- Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. 2021. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 9421--9431.Google ScholarCross Ref
- Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. 2020. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5336--5345.Google Scholar
- Liang Zhang, Wa James Tam, and Demin Wang. 2004. Stereoscopic image generation based on depth images. In 2004 International Conference on Image Processing, 2004. ICIP'04., Vol. 5. IEEE, 2993--2996.Google ScholarCross Ref
- Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. 2018. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 (2018).Google Scholar
- C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2004. High-quality video view interpolation using a layered representation. ACM transactions on graphics (TOG) 23, 3 (2004), 600--608.Google Scholar
Index Terms
- A new free viewpoint video dataset and DIBR benchmark
Recommendations
Real-Time Layered View Synthesis for Free-Viewpoint Video from Unreliable Depth Information
MMVE '23: Proceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment SystemsIn this work we present a novel approach for the generation in real time of synthetic views for free-viewpoint video. Our system is based on purely passive stereo cameras which, under the constraints of real-time operation yield unreliable depth maps, ...
Free viewpoint video generation for walk-through experience using image-based rendering
MM '08: Proceedings of the 16th ACM international conference on MultimediaThis paper presents a novel method to represent a real 3D world using IBR (Image-based Rendering) technology. The major achievement is realization of "walk-through" experience, in which audiences can see the scene of sport games as if they were the ...
Free-viewpoint video rendering for mobile devices
MIRAGE '13: Proceedings of the 6th International Conference on Computer Vision / Computer Graphics Collaboration Techniques and ApplicationsFree-viewpoint video renderers (FVVR) allow a user to view captured video footage from any position and direction. Despite the obvious appeal of such systems, they have yet to make a major impact on digital entertainment. Current FVVR implementations ...
Comments