research-article

Error Concealment of Dynamic 3D Point Cloud Streaming

Authors:

Samuel Rhys Cox,

Cheng-Hsin HsuAuthors Info & Claims

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

Pages 3134 - 3142

https://doi.org/10.1145/3503161.3548384

Published: 10 October 2022 Publication History

Abstract

Recently standardized MPEG Video-based Point Cloud Compression (V-PCC) codec has shown promise in achieving a good rate-distortion ratio of dynamic 3D point cloud compression. Current error concealment methods of V-PCC, however, lead to significantly distorted 3D point cloud frames under imperfect network conditions. To address this problem, we propose a general framework for concealing distorted and lost 3D point cloud frames due to packet loss. We also design, implement, and evaluate a suite of tools for each stage of our framework, which can be combined into multiple variants of error concealment algorithms. We conduct extensive experiments using seven dynamic 3D point cloud sequences with diverse characteristics to understand the strengths and limitations of our proposed error concealment algorithms. Our experiment results show that our algorithms outperform: (i) the method employed by V-PCC by at least 3.58 dB in Geometry Peak Signal-to-Noise Ratio (GPSNR) and 10.68 in Video Multi-Method Assessment Fusion (VMAF) and (ii) point cloud frame copy method by at most 5.8 dB in (3D) GPSNR and 12.0 in (2D) VMAF. Further, the proposed error concealment framework and algorithms work in the 3D domain, and thus are agnostic to the codecs and are applicable to future point cloud compression standards

Supplementary Material

MP4 File (mmfp2931.mp4)

Presentation video - short version.

Download
345.82 MB

References

[1]

Anne Aaron, Zhi Li, Megha Manohara, Joe Yuchieh Lin, Eddy Chi-Hao Wu, and C.-C Jay Kuo. Challenges in cloud based ingest and encoding for high quality streaming media. In 2015 IEEE International Conference on Image Processing (ICIP), pages 1732--1736, Quebec City, QC, Canada, 2015.

Digital Library

[2]

Bing-Yu Chen and Tomoyuki Nishita. Multiresolution streaming mesh with shape preserving and QoS-like controlling. In Proc. of ACM International Conference on 3D Web Technology (Web3D'02), pages 35--42, Tempe, Arizona, February 2002.

Digital Library

[3]

Jingdao Chen, John Seon Keun Yi, Mark Kahoush, Erin S Cho, and Yong K Cho. Point cloud scene completion of obstructed building facades with generative adversarial inpainting. Sensors, 20(18):5029, 2020.

[4]

Wei Cheng and Wei Tsang Ooi. Receiver-driven view-dependent streaming of progressive mesh. In Proc. of ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'08), pages 9--14, Braunschweig, Germany, May 2008.

Digital Library

[5]

Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Cal- abrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan. High-quality streamable free-viewpoint video. ACM Transactions on Graphics, 34(4):1--13, 2015.

Digital Library

[6]

Eugene d'Eon, Bob Harrison, Taos Myers, and Philip A. Chou. 8i voxelized full bodies - a voxelized point cloud dataset. ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) input document WG11M40059/WG1M74006, 2017. Geneva, CH.

[7]

Thomas Forgione, Axel Carlier, Géraldine Morin, Wei Tsang Ooi, Vincent Charvil- lat, and Praveen Kumar Yadav. DASH for 3D networked virtual environment. In Proc. of ACM International Conference on Multimedia (MM'18), pages 1910--1918, Seoul, Republic of Korea, October 2018.

Digital Library

[8]

Zeqing Fu, Wei Hu, and Zongming Guo. 3d dynamic point cloud inpainting via temporal consistency on graphs. In 2020 IEEE International Conference on Multimedia and Expo (ICME), pages 1--6. IEEE, 2020.

[9]

D Graziosi, O Nakagami, S Kuma, A Zaghetto, T Suzuki, and A Tabatabai. An overview of ongoing point cloud compression standardization activities: video- based (V-PCC) and geometry-based (G-PCC). APSIPA Transactions on Signal and Information Processing, 9, 2020.

[10]

André Guéziec, Gabriel Taubin, Bill Horn, and Francis Lazarus. A framework for streaming geometry in VRML. IEEE Computer Graphics and Applications, 19(2):68--78, 1999.

Digital Library

[11]

Bo Han, Yu Liu, and Feng Qian. Vivo: Visibility-aware mobile volumetric video streaming. In Proc. of ACM Annual International Conference on Mobile Computing and Networking (MobiCom'18), pages 1--13, New Delhi, India, October 2020.

[12]

Ju He, Zeqing Fu, Wei Hu, and Zongming Guo. Point cloud attribute inpainting in graph spectral domain. In 2019 IEEE International Conference on Image Processing (ICIP), pages 4385--4389. IEEE, 2019.

[13]

Mohammad Hosseini and Christian Timmerer. Dynamic adaptive point cloud streaming. In Proc. of ACM Packet Video Workshop (PV'18), pages 25--30, Amsterdam, The Netherlands, June 2018.

Digital Library

[14]

S-Y Hu, T-H Huang, S-C Chang, W-L Sung, J-R Jiang, and B-Y Chen. Flod: A framework for peer-to-peer 3D streaming. In Proc. of IEEE Conference on Computer Communications (INFOCOM'08), pages 1373--1381, Phoenix, AZ, April 2008.

[15]

Wei Hu, Zeqing Fu, and Zongming Guo. Local frequency interpretation and non-local self-similarity on graph for point cloud inpainting. IEEE Transactions on Image Processing, 28(8):4087--4100, 2019.

[16]

E. S. Jang, M. Preda, K. Mammou, A. M. Tourapis, J. Kim, D. B. Graziosi, S. Rhyu, and M. Budagavi. Video-based point-cloud-compression standard in MPEG: From evidence collection to committee draft [standards in a nutshell]. IEEE Signal Processing Magazine, 36(3):118--123, 2019.

[17]

Mohammad Kazemi, Mohammad Ghanbari, and Shervin Shirmohammadi. A review of temporal video error concealment techniques and their suitability for HEVC and VVC. Multimedia Tools and Applications, 80:12685--12730, March 2021.

Digital Library

[18]

Kyungjin Lee, Juheon Yi, Youngki Lee, Sunghyun Choi, and Young Min Kim. GROOT: a real-time streaming system of high-fidelity volumetric videos. In Proc. of ACM Annual International Conference on Mobile Computing and Networking (MobiCom'20), pages 1--14, London, United Kingdom, September 2020.

[19]

L. Li, Z. Li, V. Zakharchenko, J. Chen, and H. Li. Advanced 3D motion prediction for video-based dynamic point cloud compression. IEEE Transactions on Image Processing, 29:289--302, 2020.

Digital Library

[20]

Common test conditions for V3C and V-PCC. Document, ISO/IEC JTC1/SC29/WG7 MPEG 3D Graphics Coding, 2020. Meeting held online.

[21]

V-PCC test model v11. Document, ISO/IEC JTC1/SC29/WG11 MPEG 3DG, 2020. Meeting held Online.

[22]

Szymon Rusinkiewicz and Marc Levoy. Streaming QSplat: A viewer for networked visualization of large, dense models. In Proc. of ACM Symposium on Interactive 3D Graphics (I3D'01), pages 63--68, Chapel Hill, NC, March 2001.

[23]

Radu Bogdan Rusu and Steve Cousins. 3D is here: Point cloud library (PCL). In IEEE International Conference on Robotics and Automation (ICRA), pages 1--4, Shanghai, China, 2011.

[24]

The Image Magick Development Team. Image Magick. https://imagemagick.org.

[25]

Socrates Varakliotis, Jörn Ostermann, and Vicky Hardman. Coding of animated 3-d wireframe models for internet streaming applications. In Proc. of IEEE International Conference on Multimedia and Expo (ICME'01), pages 61--61, Tokyo, Japan, August 2001.

[26]

Irene Viola, Jelmer Mulder, Francesca De Simone, and Pablo Cesar. Temporal interpolation of dynamic digital humans using convolutional neural networks. In 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pages 90--907, 2019.

[27]

Xin Wen, Tianyang Li, Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip-attention network with hierarchical folding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1939--1948, 2020.

[28]

Cheng-Hao Wu, Chih-Fan Hsu, Tzu-Kuan Hung, Carsten Griwodz, Wei Tsang Ooi, and Cheng-Hsin Hsu. Quantitative comparison of point cloud compression algorithms with PCC Arena. IEEE Transactions on Multimedia, pages 1--16, February 2022. Accepted to Appear.

[29]

Cheng-Hao Wu, Xiner Li, Rahul Rajesh, Wei Tsang Ooi, and Cheng-Hsin Hsu. Dynamic 3D point cloud streaming: Distortion and concealment. In Proc. of ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV'21), Istanbul, Turkey, September 2021.

[30]

Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun. GRNet: Gridding residual network for dense point cloud completion. arXiv preprint arXiv:2006.03761, 2020.

[31]

Zhenyu Yang, Klara Nahrstedt, Yi Cui, Bin Yu, Jin Liang, Sang-hack Jung, and Ruzena Bajscy. Teeve: The next generation architecture for teleimmersive environments. In Proc. of IEEE International Symposium on Multimedia (ISM'05), pages 8--pp, Irvine, CA, December 2005.

[32]

Yiming Zeng, Yue Qian, Qijian Zhang, Junhui Hou, Yixuan Yuan, and Ying He. Idea-net: Dynamic 3d point cloud interpolation via deep embedding alignment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6338--6347, 2022.

[33]

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847, 2018.

Cited By

Dumic Eda Silva Cruz L(2023)Subjective Quality Assessment of V-PCC-Compressed Dynamic Point Clouds Degraded by Packet LossesSensors10.3390/s2312562323:12(5623)Online publication date: 15-Jun-2023
https://doi.org/10.3390/s23125623
Yang MWu DWang ZHu MZhou Y(2023)Understanding and Improving Perceptual Quality of Volumetric Video Streaming2023 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME55011.2023.00339(1979-1984)Online publication date: Jul-2023
https://doi.org/10.1109/ICME55011.2023.00339
Makiyah EKliamees N(2023)Point Cloud Streaming over HTTP/22023 6th International Conference on Advanced Communication Technologies and Networking (CommNet)10.1109/CommNet60167.2023.10365247(1-5)Online publication date: 11-Dec-2023
https://doi.org/10.1109/CommNet60167.2023.10365247

Index Terms

Error Concealment of Dynamic 3D Point Cloud Streaming

Recommendations

Dynamic 3D point cloud streaming: distortion and concealment
NOSSDAV '21: Proceedings of the 31st ACM Workshop on Network and Operating Systems Support for Digital Audio and Video

We present a study on the impact of packet loss on dynamic 3D point cloud streaming, encoded with MPEG Video-based Point Cloud Compression (V-PCC) standard. We show the distortion when different channels of V-PCC bitstream are lost, with the loss of ...
Error resilient coding and error concealment in scalable video coding

Scalable video coding (SVC), which is the scalable extension of the H.264/AVC standard, was developed by the Joint Video Team (JVT) of ISO/IEC MPEG (Moving Picture Experts Group) and ITU-T VCEG (Video Coding Experts Group). SVC is designed to provide ...
Error concealment aware streaming video system over packet-based mobile networks

A compressed video bitstream is extremely sensitive to errors when it is transmitted via networks with bursty packet loss such as 3G and mobile WiMAX. Error concealment techniques have been used to improve the recovered picture quality in cases in which ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '22: Proceedings of the 30th ACM International Conference on Multimedia

October 2022

7537 pages

ISBN:9781450392037

DOI:10.1145/3503161

General Chairs:
João Magalhães
NOVA University of Lisbon, Portugal
,
Alberto del Bimbo
University of Florence, Italy
,
Shin'ichi Satoh
National Institute of Informatics, Japan
,
Nicu Sebe
University of Trento, Italy
,
Program Chairs:
Xavier Alameda-Pineda
Inria, Grenoble, France
,
Qin Jin
Renmin University of China, China
,
Vincent Oria
New Jersey Institute of Technology, USA
,
Laura Toni
University College London, UK

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 October 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

The Singapore Ministry of Education Academic Research Fund Tier 1
The MOST of Taiwan

Conference

MM '22

Sponsor:

SIGMM

MM '22: The 30th ACM International Conference on Multimedia

October 10 - 14, 2022

Lisboa, Portugal

Acceptance Rates

Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
350
Total Downloads

Downloads (Last 12 months)98
Downloads (Last 6 weeks)11

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dumic Eda Silva Cruz L(2023)Subjective Quality Assessment of V-PCC-Compressed Dynamic Point Clouds Degraded by Packet LossesSensors10.3390/s2312562323:12(5623)Online publication date: 15-Jun-2023
https://doi.org/10.3390/s23125623
Yang MWu DWang ZHu MZhou Y(2023)Understanding and Improving Perceptual Quality of Volumetric Video Streaming2023 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME55011.2023.00339(1979-1984)Online publication date: Jul-2023
https://doi.org/10.1109/ICME55011.2023.00339
Makiyah EKliamees N(2023)Point Cloud Streaming over HTTP/22023 6th International Conference on Advanced Communication Technologies and Networking (CommNet)10.1109/CommNet60167.2023.10365247(1-5)Online publication date: 11-Dec-2023
https://doi.org/10.1109/CommNet60167.2023.10365247

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten