Yuan-Chun Sun
ABSTRACT
We address the problem of quantifying the perceived quality in 6DoF (Degree-of-Freedom) live immersive video in two steps. First, we develop a set of tools to generate (or collect) datasets in a photorealistic simulator, AirSim. Using these tools, we get to change diverse settings of live immersive videos, such as scenes, trajectories, camera placements, and encoding parameters. Second, we develop objective and subjective evaluation procedures, and carry out evaluations on a sample immersive video codec, MPEG MIV, using our own dataset. Several insights were found through our experiments: (1) the two synthesizers in TMIV produce comparable target view quality, but RVS runs 2 times faster; (2) Quantization Parameter (QP) is a good control knob to exercise target view quality and bitrate, but camera placements (or trajectories) also impose significant impacts; and (3) overall subjective quality has strong linear/rank correlation with subjective similarity, sharpness, and color. These findings shed some light on the future research problems for the development of emerging applications relying on immersive interactions.
Supplemental Material
- Federica Battisti, Emilie Bosc, Marco Carli, Patrick Le Callet, and Simone Perugia. 2015. Objective image quality assessment of 3D synthesized views. Signal Processing: Image Communication 30 (2015), 78--88. https://doi.org/10.1016/j. image.2014.10.005Google Scholar
Cross Ref
- Jacob Benesty, Jingdong Chen, Yiteng Huang, and Israel Cohen. 2009. Pearson correlation coefficient. In Noise reduction in speech processing. Springer, 1--4.Google ScholarDigital Library
- Gisle Bjontegaard. 2001. Calculation of average PSNR differences between RDcurves. VCEG-M33 (2001).Google Scholar
- J. Boyce, R. Dore, A. Dziembowski, J. Fleureau, J. Jung, B. Kroon, B. Salahieh, V. Vadakital, and L. Yu. 2021. MPEG Immersive Video Coding Standard. Proc. IEEE (2021).Google Scholar
- Yangang Cai, Ronggang Wang, Ke Qiu, Rui Peng, Zhipeng Cheng, and Qi Wang. 2021. Depth Map Video Compression Performance Evaluation For Ieee 1857.9. In Proc. of IEEE International Conference on Multimedia Expo Workshops (ICMEW'21). Shenzhen, China, 1--6. https://doi.org/10.1109/ICMEW53276.2021.9455977Google ScholarCross Ref
- Epic Games. 2019. Unreal Engine. Retrieved July 7, 2021 from https://www. unrealengine.comGoogle Scholar
- Epic Games. 2021. Unreal Engine marketplace. Retrieved Sep 3, 2021 from https://www.unrealengine.com/marketplace/en-US/storeGoogle Scholar
- Sarah Fachada, Daniele Bonatto, Arnaud Schenkel, and Gauthier Lafruit. 2018. Free Navigation in Natural Scenery With DIBR: RVS and VSRS in MPEG-I Standardization. In Proc. of IEEE International Conference on 3D Immersion (IC3D). 1--6. https://doi.org/10.1109/IC3D.2018.8657912Google ScholarCross Ref
- FFmpeg. 2021. FFmpeg. Retrieved Sep 3, 2021 from https://www.ffmpeg.org/Google Scholar
- Julien Fleureau, Bertrand Chupeau, Franck Thudor, Gerard Briand, Thierry Tapie, and Renaud Dore. 2020. An Immersive Video Experience with Real-Time View Synthesis Leveraging the Upcoming MIV Distribution Standard. In Proc. of IEEE International Conference on Multimedia Expo Workshops (ICMEW). 1--2. https: //doi.org/10.1109/ICMEW46912.2020.9105948Google ScholarCross Ref
- Fortune Business Insights. 2022. Virtual Reality Market Size, Share & COVID-19 Impact Analysis, By Component (Hardware, Software, Content), By Device Type (Head Mounted Display, VR Simulator, VR Glasses, Treadmills & Haptic Gloves, Others), By Industry (Gaming, Entertainment, Automotive, Retail, Healthcare, Education, Aerospace & Defense, Manufacturing, Others), and Regional Forecast, 2022--2029. Retrieved May. 29, 2022 from https://www.fortunebusinessinsights.com/industryreports/virtual-reality-market-101378Google Scholar
- J. Hooft, M. Vega, T. Wauters, C. Timmerer, A. Begen, F. Turck, and R. Schatz. 2020. From Capturing to Rendering: Volumetric Media Delivery with Six Degrees of Freedom. IEEE Communications Magazine 58, 10 (2020), 49--55. https://doi. org/10.1109/MCOM.001.2000242Google ScholarCross Ref
- Chih-Fan Hsu, Tse-Hou Hung, and Cheng-Hsin Hsu. 2022. Optimizing Immersive Video Coding Configurations Using Deep Learning: A Case Study on TMIV. ACM Trans. Multimedia Comput. Commun. Appl. 18, 1, Article 19 (jan 2022), 25 pages. https://doi.org/10.1145/3471191Google ScholarDigital Library
- J. Jeong, S. Lee, I. Ryu, T. Le, and E. Ryu. 2020. Towards Viewport-Dependent 6DoF 360 Video Tiled Streaming for Virtual Reality Systems. In Proc. of the ACM International Conference on Multimedia (MM '20). New York, NY, USA, 3687--3695. https://doi.org/10.1145/3394171.3413712Google ScholarDigital Library
- Jong-Beom Jeong, Soonbin Lee, and Eun-Seok Ryu. 2021. DWS-BEAM: DecoderWise Subpicture Bitstream Extracting and Merging for MPEG Immersive Video. In 2021 International Conference on Visual Communications and Image Processing (VCIP). 1--5. https://doi.org/10.1109/VCIP53242.2021.9675419Google Scholar
- Jong-Beom Jeong, Soonbin Lee, and Eun-Seok Ryu. 2021. Sub-bitstream packing based lightweight tiled streaming for 6 degree of freedom immersive video. Electronics Letters 57, 25 (2021), 973--976. https://doi.org/10.1049/ell2.12329 arXiv:https://ietresearch.onlinelibrary.wiley.com/doi/pdf/10.1049/ell2.12329Google ScholarCross Ref
- Jong-Beom Jeong, Soonbin Lee, and Eun-Seok Ryu. 2022. Rethinking FatigueAware 6DoF Video Streaming: Focusing on MPEG Immersive Video. In 2022 International Conference on Information Networking (ICOIN). 304--309. https: //doi.org/10.1109/ICOIN53446.2022.9687247Google ScholarCross Ref
- Chongchong Jin, Zongju Peng, Fen Chen, and Gangyi Jiang. 2022. Subjective and Objective Video Quality Assessment for Windowed-6DoF Synthesized Videos. IEEE Transactions on Broadcasting (2022), 1--15. https://doi.org/10.1109/TBC. 2022.3165473Google ScholarCross Ref
- Joël Jung and Patrick Boissonade. 2020. VVS: Versatile View Synthesizer for 6-DoF Immersive Video. working paper or preprint.Google Scholar
- J Jung, B Kroon, and J Boyce. 2020. Common Test Conditions for MPEG Immersive Video. ISO/IEC JTC 1/SC 29/WG 11 N19484 (2020).Google Scholar
- Sangwoon Kwak, Joungil Yun, Jun-Young Jeong, Youngwook Kim, Insung Ihm, Won-Sik Cheong, and Jeongil Seo. 2022. View synthesis with sparse light field for 6DoF immersive video. ETRI Journal 44, 1 (2022), 24--37. https://doi.org/10.4218/ etrij.2021-0205 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.4218/etrij.2021- 0205Google ScholarCross Ref
- Heesub Lee, Alan H. Rowberg, Mark S. Frank M.D., Hyung-Sik Choi M.D., and Yongmin Kim. 1992. Subjective evaluation of compressed image quality. In Medical Imaging VI: Image Capture, Formatting, and Display, Yongmin Kim (Ed.), Vol. 1653. International Society for Optics and Photonics, SPIE, 241 -- 251. https: //doi.org/10.1117/12.59503Google Scholar
- Soonbin Lee, Jong-Beom Jeong, and Eun-Seok Ryu. 2022. Efficient Group-Based Packing Strategy for 6DoF Immersive Video Streaming. In 2022 International Conference on Information Networking (ICOIN). 310--314. https://doi.org/10.1109/ ICOIN53446.2022.9687139Google ScholarCross Ref
- Leida Li, Yipo Huang, Jinjian Wu, Ke Gu, and Yuming Fang. 2021. Predicting the Quality of View Synthesis With Color-Depth Image Fusion. IEEE Transactions on Circuits and Systems for Video Technology 31, 7 (2021), 2509--2521. https: //doi.org/10.1109/TCSVT.2020.3024882Google ScholarCross Ref
- Leida Li, Yu Zhou, Ke Gu, Weisi Lin, and Shiqi Wang. 2018. Quality Assessment of DIBR-Synthesized Images by Measuring Local Geometric Distortions and Global Sharpness. IEEE Transactions on Multimedia 20, 4 (2018), 914--926. https: //doi.org/10.1109/TMM.2017.2760062Google ScholarDigital Library
- Xiangkai Liu, Yun Zhang, Sudeng Hu, Sam Kwong, C-C Jay Kuo, and Qiang Peng. 2015. Subjective and objective video quality assessment of 3D synthesized views with texture/depth compression distortion. IEEE Transactions on Image Processing 24, 12 (2015), 4847--4861.Google ScholarDigital Library
- Dawid Mieloch, Patrick Garus, Marta Milovanovi?, Joël Jung, Jun Young Jeong, Smitha Lingadahalli Ravi, and Basel Salahieh. 2022. Overview and Efficiency of Decoder-Side Depth Estimation in MPEG Immersive Video. IEEE Transactions on Circuits and Systems for Video Technology (2022).Google Scholar
- Sebastian Möller, Marcel Wältermann, and Marie-Neige Garcia. 2014. Features of Quality of Experience. Springer International Publishing, Cham, 73--84. https: //doi.org/10.1007/978--3--319-02681--7_5Google Scholar
- MPEG. 2019. The gitlab of MPEG test model for immersive video. Retrieved Sep 3, 2021 from https://gitlab.com/mpeg-i-visual/tmiv/-/tree/v10.0.1Google Scholar
- Netflix. 2021. VMAF - Video Multi-Method Assessment Fusion. Retrieved Sep 3, 2021 from https://github.com/Netflix/vmafGoogle Scholar
- K. Nevelsteen. 2018. Virtual world, defined from a technological perspective and applied to video games, mixed reality, and the Metaverse. Computer Animation and Virtual Worlds 29, 1 (2018), e1752. https://doi.org/10.1002/cav.1752Google ScholarCross Ref
- B. Ray, J. Jung, and M. Larabi. 2018. On the possibility to achieve 6-DoF for 360 video using divergent multi-view content. In Proc. of European Signal Processing Conference (EUSIPCO'18). Rome, Italy, 211--215. https://doi.org/10.23919/ EUSIPCO.2018.8553397Google Scholar
- Sadbhawna, Vinit Jakhetiya, Deebha Mumtaz, Badri Narayan Subudhi, and Sharath Chandra Guntuku. 2022. Stretching Artifacts Identification for Quality Assessment of 3D-Synthesized Views. IEEE Transactions on Image Processing 31 (2022), 1737--1750. https://doi.org/10.1109/TIP.2022.3145997Google ScholarCross Ref
- Basel Salahieh, Sumit Bhatia, and Jill Boyce. 2019. Multi-Pass Renderer in MPEG Test Model for Immersive Video. In Proc. of IEEE International Conference on Picture Coding Symposium (PCS). 1--5. https://doi.org/10.1109/PCS48520.2019. 8954515Google ScholarCross Ref
- Sebastian Schwarz and M. Hannuksela. 2017. Perceptual quality assessment of HEVC main profile depth map compression for six degrees of freedom virtual reality video. In Proc. of IEEE International Conference on Image Processing (ICIP'17). Beijing, China, 181--185. https://doi.org/10.1109/ICIP.2017.8296267Google ScholarDigital Library
- BT Series. 2012. Methodology for the subjective assessment of the quality of television pictures. Recommendation ITU-R BT (2012), 500--13.Google Scholar
- Shital Shah, Debadeepta Dey, Chris Lovett, and Ashish Kapoor. 2018. AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehicles. In Field and Service Robotics. Springer, 621--635.Google Scholar
- Jakub Szekieda, Adrian Dziembowski, and Dawid Mieloch. 2021. The Influence of Coding Tools on Immersive Video Coding. In Proc. of WSCG International Conference on Computer Graphics, Visualization and Computer Vision. Pilsen (Plzen), Czech Republic. https://doi.org/10.24132/CSRN.2021.3002.21Google Scholar
- Shishun Tian, Lu Zhang, Wenbin Zou, Xia Li, Ting Su, Luce Morin, and Olivier Deforges. 2019. Quality Assessment of DIBR-synthesized views: An Overview. https://doi.org/10.48550/ARXIV.1911.07036Google Scholar
- Xuejin Wang, Feng Shao, Qiuping Jiang, Randi Fu, and Yo-Sung Ho. 2019. Quality Assessment of 3D Synthesized Images via Measuring Local Feature Similarity and Global Sharpness. IEEE Access 7 (2019), 10242--10253. https://doi.org/10. 1109/ACCESS.2019.2891070Google ScholarCross Ref
- Jiebin Yan, Jing Li, Yuming Fang, Zhaohui Che, Xue Xia, and Yang Liu. 2022. Subjective and Objective Quality of Experience of Free Viewpoint Videos. IEEE Transactions on Image Processing 31 (2022), 3896--3907. https://doi.org/10.1109/ TIP.2022.3177127Google ScholarDigital Library
- Huiqing Zhang, Donghao Li, Zhifang Xia, Zichen Wang, and Guangchen Wang. 2020. Energy Loss Estimation Based Reference-Free Quality Assessment of DIBRSynthesized Views. In 2020 39th Chinese Control ConfGoogle Scholar
Index Terms
- On Objective and Subjective Quality of 6DoF Synthesized Live Immersive Videos
Recommendations
Towards Viewport-dependent 6DoF 360 Video Tiled Streaming for Virtual Reality Systems
MM '20: Proceedings of the 28th ACM International Conference on MultimediaPrevious studies of 360-degree video streaming with regard to virtual reality allowed users to move their head freely, while their position is fixed according to the camera's location in virtual reality. One of the approaches to overcome the problem is ...
Subjective Evaluation of an Olfaction Enhanced Immersive Virtual Reality Environment
AltMM '17: Proceedings of the 2nd International Workshop on Multimedia Alternate RealitiesRecent research efforts have reported findings on user Quality of Experience (QoE) of immersive virtual reality (VR) experiences. Truly immersive multimedia experiences also include multisensory components such as factional, tactile etc., in addition to ...
Subjective quality evaluation of compressed digital compound images
We build a new Compound Image Quality Assessment Database (CIQAD).Subjective test is conducted to get the MOS values of images in CIQAD.Five compression methods are compared with respect to the obtained MOS values.We study the applicability of existing ...
Comments