Javier Usón
ABSTRACT
The recent development of new video capture systems has led to the adoption of volumetric video technologies to replace 2D video in use cases such as videoconference, where this enhancement promises to solve videoconference fatigue. In particular, volumetric capture allows the content to be viewed from different points of view, enabling more natural interaction during the videoconference. One of the solutions proposed for this scenario is Free Viewpoint Video (FVV). It makes use of a set of calibrated cameras that allows the use of real life information to generate a synthetic view from any arbitrary point in space. Although there are real-time capture developments of FVV systems, they make use of 2D displays and joysticks to control the point of view. In our opinion, this undermines the possibilities of volumetric video for the videoconferencing use case. Building on a previously developed FVV system, we present a novel untethered HMD-based immersive visualization system that enables point of view control with the user's natural position and visualization of live volumetric content in a 3D environment. Synthetic views are generated in real-time by the FVV system, and streamed with low latency protocols to a Meta quest 3 HMD using a WebRTC-based server. This work discusses the architecture of the end-to-end system and describes the bitrate, framerate and latency values at which the system works.
- Andrew A Bennett, Emily D Campion, Kathleen R Keeler, and Sheila K Keener. 2021. Videoconference fatigue? Exploring changes in fatigue after videoconference meetings during COVID-19. Journal of Applied Psychology 106, 3 (2021), 330.Google Scholar
Cross Ref
- Pablo Carballeira, Carlos Carmona, César Díaz, Daniel Berjón, Daniel Corregidor, Julián Cabrera, Francisco Morán, Carmen Doblado, Sergio Arnaldo, María del Mar Martín, and Narciso García. 2022. FVV Live: A Real-Time Free-Viewpoint Video System With Consumer Electronics Hardware. IEEE Transactions on Multimedia 24 (2022), 2378--2391. https://doi.org/10.1109/TMM.2021.3079711Google ScholarDigital Library
- Robert Gruen, Eyal Ofek, Anthony Steed, Ran Gal, Mike Sinclair, and Mar Gonzalez-Franco. 2020. Measuring System Visual Latency through Cognitive Latency on Video See-Through AR devices. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 791--799. https://doi.org/10.1109/VR46266.2020.00103Google ScholarCross Ref
- Richard Hartley and Andrew Zisserman. 2003. Multiple view geometry in computer vision. Cambridge university press.Google Scholar
- Teresa Hernando, Daniel Berjón, Francisco Morán, Javier Usón, Cesar Díaz, Julián Cabrera, and Narciso García. 2023. Real-Time Layered View Synthesis for Free-Viewpoint Video from Unreliable Depth Information. In Proceedings of the 15th International Workshop on Immersive Mixed and Virtual Environment Systems (Vancouver, BC, Canada) (MMVE '23). Association for Computing Machinery, New York, NY, USA, 7--11. https://doi.org/10.1145/3592834.3592881Google ScholarDigital Library
- IETF. 2006. Session Description Protocol (SDP). https://tools.ietf.org/html/rfc4566.Google Scholar
- IETF. 2018. Interactive Connectivity Establishment (ICE). https://tools.ietf.org/html/rfc8445.Google Scholar
- Yili Jin, Kaiyuan Hu, Junhua Liu, Fangxin Wang, and Xue Liu. 2023. From Capture to Display: A Survey on Volumetric Video. arXiv:2309.05658 [cs.MM]Google Scholar
- Jeremy Lainé. 2024. aiortc. https://github.com/aiortc/aiortc.Google Scholar
- Pablo Pérez, Daniel Corregidor, Emilio Garrido, Ignacio Benito, Ester González-Sosa, Julián Cabrera, Daniel Berjón, César Díaz, Francisco Morán, Narciso García, Josué Igual, and Jaime Ruiz. 2022. Live Free-Viewpoint Video in Immersive Media Production Over 5G Networks. IEEE Transactions on Broadcasting 68, 2 (2022), 439--450. https://doi.org/10.1109/TBC.2022.3154612Google ScholarCross Ref
- Pablo Pérez, Ester Gonzalez-Sosa, Jesús Gutiérrez, and Narciso García. 2022. Emerging Immersive Communication Systems: Overview, Taxonomy, and Good Practices for QoE Assessment. Frontiers in Signal Processing 2 (2022). https://doi.org/10.3389/frsip.2022.917684Google ScholarCross Ref
- Moritz Queisner, Michael Pogorzhelskiy, Christopher Remde, Johann Pratschke, and Igor M Sauer. 2022. VolumetricOR: A New Approach to Simulate Surgical Interventions in Virtual Reality for Training and Education. Surg Innov 29, 3 (Feb. 2022), 406--415.Google ScholarCross Ref
- Oliver Schreer, Ingo Feldmann, Sylvain Renault, Marcus Zepp, Markus Worchel, Peter Eisert, and Peter Kauff. 2019. Capture and 3D Video Processing of Volumetric Video. In 2019 IEEE International Conference on Image Processing (ICIP). 4310--4314. https://doi.org/10.1109/ICIP.2019.8803576Google ScholarCross Ref
- Janto Skowronek, Alexander Raake, Gunilla H. Berndtsson, Olli S. Rummukainen, Paolino Usai, Simon N. B. Gunkel, Mathias Johanson, Emanuël A. P. Habets, Ludovic Malfait, David Lindero, and Alexander Toet. 2022. Quality of Experience in Telemeetings and Videoconferencing: A Comprehensive Survey. IEEE Access 10 (2022), 63885--63931. https://doi.org/10.1109/ACCESS.2022.3176369Google ScholarCross Ref
- A. Smolic, K. Mueller, P. Merkle, C. Fehn, P. Kauff, P. Eisert, and T. Wiegand. 2006. 3D Video and Free Viewpoint Video - Technologies, Applications and MPEG Standards. In 2006 IEEE International Conference on Multimedia and Expo. 2161--2164. https://doi.org/10.1109/ICME.2006.262683Google ScholarCross Ref
- Irene Viola, Jack Jansen, Shishir Subramanyam, Ignacio Reimat, and Pablo Cesar. 2023. VR2Gather: A Collaborative, Social Virtual Reality System for Adaptive, Multiparty Real-Time Communication. IEEE MultiMedia 30, 2 (2023), 48--59. https://doi.org/10.1109/MMUL.2023.3263943Google ScholarDigital Library
- WebRTC Working Group. 2021. Web Real-Time Communication (WebRTC). https://www.w3.org/TR/webrtc/.Google Scholar
- Yuko Yoshida and Tetsuya Kawamoto. 2014. [DEMO] Displaying free-viewpoint video with user controlable head mounted display DEMO. In 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 389--390. https://doi.org/10.1109/ISMAR.2014.6948503Google ScholarCross Ref
- Gareth W. Young, Néill O'Dwyer, and Aljosa Smolic. 2023. Chapter 21 - Volumetric video as a novel medium for creative storytelling. In Immersive Video Technologies, Giuseppe Valenzise, Martin Alain, Emin Zerman, and Cagri Ozcinar (Eds.). Academic Press, 591--607. https://doi.org/10.1016/B978-0-32-391755-1.00027-4Google ScholarCross Ref
Index Terms
- Untethered Real-Time Immersive Free Viewpoint Video
Recommendations
Free-viewpoint video rendering for mobile devices
MIRAGE '13: Proceedings of the 6th International Conference on Computer Vision / Computer Graphics Collaboration Techniques and ApplicationsFree-viewpoint video renderers (FVVR) allow a user to view captured video footage from any position and direction. Despite the obvious appeal of such systems, they have yet to make a major impact on digital entertainment. Current FVVR implementations ...
Free-viewpoint video of human actors
In free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its ...
Free-viewpoint video of human actors
SIGGRAPH '03: ACM SIGGRAPH 2003 PapersIn free-viewpoint video, the viewer can interactively choose his viewpoint in 3-D space to observe the action of a dynamic real-world scene from arbitrary perspectives. The human body and its motion plays a central role in most visual media and its ...
Comments