Upsampling Algorithm for V-PCC-Coded 3D Point Clouds
Authors: 
Ting-Lan Lin, Bing-Wei Su, Po-Cheng Shen, Ding-Yuan Chen, Chi-Fu Liang, Yan-Cheng Chen, Yangming Wen, Mohammad ShahidAuthors Info & Claims
Ting-Lan Lin, Bing-Wei Su, Po-Cheng Shen, Ding-Yuan Chen, Chi-Fu Liang, Yan-Cheng Chen, Yangming Wen, Mohammad ShahidAuthors Info & ClaimsArticle No.: 367, Pages 1 - 23
Abstract
Point cloud (PC) compression is crucial to immersive visual applications such as autonomous vehicles to classify objects on the roads. The Motion Picture Experts Group (MPEG) standardization group has achieved a notable compression efficiency, called video-based PC compression (V-PCC), which consists of an encoder-decoder. The V-PCC encoder takes original 3D PC data and projects them onto multiple 2D planes to generate several 2D feature images. These images are then compressed using the well-established High-Efficiency Video Coding (HEVC) method. The V-PCC decoder uses compressed information and decoding techniques to reconstruct the 3D PC. However, the PCs produced by V-PCC are often sparse, non-uniform, and contain artifacts. In many practical applications, it is necessary to recover complete PCs from partial ones in real time. This article presents a method for enhancing decoded PCs as a post-processing step in the V-PCC with reduced computational time. Our approach involves a 2D upsampling for the V-PCC occupancy image, which increases the density of the PC, and a 2D high-resolution auxiliary information modification algorithm for the 2D-3D conversion of high-resolution 3D PCs, which improves the uniformity and reduces the noise in the PC. The 3D high-resolution PC has been further enhanced using the developed 3D outlier removal and point regeneration algorithm. Our proposed work can significantly simplify the state-of-the-art super resolution methods for PCs and reduce the time complexity of 61–75% while maintaining a high level of quality in PCs.
References
[1]
Siheng Chen, Baoan Liu, Chen Feng, Carlos Vallespi-Gonzalez, and Carl Wellington. 2020. 3d point cloud processing and learning for autonomous driving: Impacting map creation, localization, and perception. IEEE Signal Processing Magazine 38, 1 (2020), 68–86.
[2]
Anthony Scavarelli, Ali Arya, and Robert J. Teather. 2021. Virtual reality and augmented reality in social learning spaces: A literature review. Virtual Reality 25 (2021), 257–277.
[3]
Evangelos Alexiou, Nanyang Yang, and Touradj Ebrahimi. 2020. PointXR: A toolbox for visualization and subjective evaluation of point clouds in virtual reality. In Proceedings of the 2020 12th International Conference on Quality of Multimedia Experience (QoMEX ’20). IEEE, 1–6.
[4]
Qiangqiang Cheng, Pengyu Sun, Chunsheng Yang, Yubin Yang, and Peter Xiaoping Liu. 2020. A morphing-based 3D point cloud reconstruction framework for medical image processing. Computer Methods and Programs in Biomedicine 193 (2020), 105495.
[5]
Euee S. Jang, Marius Preda, Khaled Mammou, Alexis M. Tourapis, Jungsun Kim, Danillo B. Graziosi, Sungryeul Rhyu, and Madhukar Budagavi. 2019. Video-based point-cloud-compression standard in MPEG: From evidence collection to committee draft [standards in a nutshell]. IEEE Signal Processing Magazine 36, 3 (2019), 118–123.
[6]
Sebastian Schwarz, Marius Preda, Vittorio Baroncini, Madhukar Budagavi, Pablo Cesar, Philip A. Chou, Robert A. Cohen, Maja Krivokuća, Sébastien Lasserre, Zhu Li, Joan Llach, Khaled Mammou, Rufael Mekuria, Ohji Nakagami, Ernestasia Siahaan, Ali Tabatabai, Alexis M. Tourapis, and Vladyslav Zakharchenko. 2018. Emerging MPEG standards for point cloud compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 9, 1 (2018), 133–148.
[7]
Li Cui, Rufael Mekuria, Marius Preda, and Euee S. Jang. 2019. Point-cloud compression: Moving picture experts group’s new standard in 2020. IEEE Consumer Electronics Magazine 8, 4 (2019), 17–21.
[8]
Ruwen Schnabel and Reinhard Klein. 2006. Octree-based point-cloud compression. In PBG@ SIGGRAPH 8, 3 (2006), 111–121.
[9]
Julius Kammerl, Nico Blodow, Radu Bogdan Rusu, Suat Gedikli, Michael Beetz, and Eckehard Steinbach. 2012. Real-time compression of point cloud streams. In Proceedings of the 2012 IEEE International Conference on Robotics and Automation. IEEE, 778–785.
[10]
Jyun-Yuan Chen, Chao-Hung Lin, Po-Chi Hsu, and Chung-Hao Chen. 2013. Point cloud encoding for 3D building model retrieval. IEEE Transactions on Multimedia 16, 2 (2013), 337–345.
[11]
Yuxue Fan, Yan Huang, and Jingliang Peng. 2013. Point cloud compression based on hierarchical point clustering. In Proceedings of the 2013 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference. IEEE, 1–7.
[12]
Ricardo L. De Queiroz and Philip A. Chou. 2016. Compression of 3D point clouds using a region-adaptive hierarchical transform. IEEE Transactions on Image Processing 25, 8 (2016), 3947–3956.
[13]
Ricardo L. de Queiroz and Philip A. Chou. 2017. Motion-compensated compression of dynamic voxelized point clouds. IEEE Transactions on Image Processing 26, 8 (2017), 3886–3895. DOI:
[14]
Paulo de Oliveira Rente, Catarina Brites, Joao Ascenso, and Fernando Pereira. 2018. Graph-based static 3D point clouds geometry coding. IEEE Transactions on Multimedia 21, 2 (2018), 284–299.
[15]
Enrico Mattei and Alexey Castrodad. 2017. Point cloud denoising via moving RPCA. In Computer Graphics Forum, Vol. 36. Wiley Online Library, 123–137.
[16]
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. 2017. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 652–660.
[17]
Yue Wu, Xidao Hu, Yue Zhang, Maoguo Gong, Wenping Ma, and Qiguang Miao. 2023. SACF-Net: Skip-attention based correspondence filtering network for point cloud registration. IEEE Transactions on Circuits and Systems for Video Technology 33 (2023), 3585–3595.
[18]
Yue Wu, Yue Zhang, Wenping Ma, Maoguo Gong, Xiaolong Fan, Mingyang Zhang, A. K. Qin, and Qiguang Miao. 2023b. Rornet: Partial-to-partial registration network with reliable overlapping representations. IEEE Transactions on Neural Networks and Learning Systems (2023), 1–4. DOI:
[19]
Yongzhe Yuan, Yue Wu, Xiaolong Fan, Maoguo Gong, Wenping Ma, and Qiguang Miao. 2023. EGST: Enhanced geometric structure transformer for point cloud registration. IEEE Transactions on Visualization and Computer Graphics 30 (2023), 6222–6234.
[20]
Soonjo Kwon, Ji-Hyeon Hur, and Hyungki Kim. 2023. Deep learning-based point cloud upsampling: A review of recent trends. JMST Advances 5, 4 (2023), 105–111.
[21]
Li Li, Zhu Li, Shan Liu, and Houqiang Li. 2020. Efficient projected frame padding for video-based point cloud compression. IEEE Transactions on Multimedia 23 (2020), 2806–2819.
[22]
Qi Liu, Hui Yuan, Junhui Hou, Raouf Hamzaoui, and Honglei Su. 2020. Model-based joint bit allocation between geometry and color for video-based 3D point cloud compression. IEEE Transactions on Multimedia 23 (2020), 3278–3291.
[23]
Junsik Kim, Jiheon Im, Sungryeul Rhyu, and Kyuheon Kim. 2020. 3D motion estimation and compensation method for video-based point cloud compression. IEEE Access 8 (2020), 83538–83547.
[24]
Tianyu Dong, Kyutae Kim, and Euee S. Jang. 2021. Performance evaluation of the codec agnostic approach in MPEG-I video-based point cloud compression. IEEE Access 9 (2021), 167990–168003.
[25]
Keming Cao and Pamela Cosman. 2021. Denoising and inpainting for point clouds compressed by V-PCC. IEEE Access 9 (2021), 107688–107700.
[26]
David Stutz and Andreas Geiger. 2018. Learning 3d shape completion from laser scan data with weak supervision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1955–1964.
[27]
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2018. Pu-net: Point cloud upsampling network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2790–2799.
[28]
Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine-Hornung. 2019. Patch-based progressive 3d point set upsampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 5958–5967.
[29]
Ruihui Li, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. 2019. Pu-gan: A point cloud upsampling adversarial network. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 7203–7212.
[30]
Shuquan Ye, Dongdong Chen, Songfang Han, Ziyu Wan, and Jing Liao. 2021. Meta-pu: An arbitrary-scale upsampling network for point cloud. IEEE Transactions on Visualization and Computer Graphics 28, 9 (2021), 3206–3218.
[31]
Dandan Ding, Chi Qiu, Fuchang Liu, and Zhigeng Pan. 2021. Point cloud upsampling via perturbation learning. IEEE Transactions on Circuits and Systems for Video Technology 31, 12 (2021), 4661–4672.
[32]
Pedro Hermosilla, Tobias Ritschel, and Timo Ropinski. 2019. Total denoising: Unsupervised learning of 3D point cloud cleaning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 52–60.
[33]
Shitong Luo and Wei Hu. 2021. Score-based point cloud denoising. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 4583–4592.
[34]
John Bowers, Rui Wang, Li-Yi Wei, and David Maletz. 2010. Parallel Poisson disk sampling with spectrum analysis on surfaces. ACM Transactions on Graphics 29, 6 (2010), 1–10.
[35]
Robert Keys. 1981. Cubic convolution interpolation for digital image processing. IEEE Transactions on Acoustics, Speech, and Signal Processing 29, 6 (1981), 1153–1160.
[36]
JPEG. 2019. Common test conditions for PCC. ISO/IEC JTC1/SC29/WG11 N18883.
[37]
The Moving Picture Experts Group (MPEG). 2020. VPCC-mpeg-pcc-tmc2-realese8.0. Retrieved from https://github.com/MPEGGroup/mpeg-pcc-tmc2/tree/release-v8.0
[38]
High Efficiency Video Coding (HEVC). 2017. HEVC HM-16.20+SCM-8.8. Retrieved from https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.20+SCM-8.8/
[39]
Ting-Lan Lin, Hong-Bin Bu, Yan-Cheng Chen, Jun-Rui Yang, Chi-Fu Liang, Kun-Hu Jiang, Ching-Hsuan Lin, and Xiao-Feng Yue. 2021. Efficient quadtree search for HEVC coding units for V-PCC. IEEE Access 9 (2021), 139109–139121.
[40]
Eugene d’Eon, Bob Harrison, Taos Myers, and Philip A. Chou. 2017. 8i voxelized full bodies-a voxelized point cloud dataset. ISO/IEC JTC1/SC29 Joint WG11/WG1 (MPEG/JPEG) input document WG11M40059/WG1M74006 7, 8 (2017), 11.
[41]
Y. Xu, Y. Lu, and Z. Wen. 2017. Owlii dynamic human mesh sequence dataset. ISO/IEC JTC1/SC29/WG11 m41658, 120th MPEG Meeting.
[42]
Gabriel Meynet, Yana Nehmé, Julie Digne, and Guillaume Lavoué. 2020. PCQM: A full-reference quality metric for colored 3D point clouds. In Proceedings of the 2020 12th International Conference on Quality of Multimedia Experience (QoMEX ’42). IEEE, 1–6.
Index Terms
- Upsampling Algorithm for V-PCC-Coded 3D Point Clouds
Recommendations
Transcoding V-PCC Point Cloud Streams in Real-time
Dynamic Point Clouds are a representation for 3D immersive media that allows users to freely navigate a scene while consuming the content. However, this comes at the cost of substantial data size, requiring efficient compression techniques to make point ...
Rate conversion of MPEG coded video by re-quantization process
ICIP '95: Proceedings of the 1995 International Conference on Image Processing (Vol. 3)-Volume 3 - Volume 3We propose rate conversion method by re-quantization in which MPEG coded video at high bit rate is converted into the MPEG bitstream at a lower bit rate without decoding to obtain the reconstructed picture. The quantization step required for re-...
Enabling Low Bit-Rate MPEG V-PCC-encoded Volumetric Video Streaming with 3D Sub-sampling
MMSys '23: Proceedings of the 14th ACM Multimedia Systems ConferenceMPEG's Video-based Point Cloud Compression (V-PCC) is a recent new standard for volumetric video compression. By mapping a 3D dynamic point cloud to a 2D image sequence, V-PCC can rely on state-of-the-art video codecs to achieve high compression rate ...
Comments
Information & Contributors
Information
Published In
December 2024
721 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 November 2024
Online AM: 31 August 2024
Accepted: 22 August 2024
Revised: 11 August 2024
Received: 13 March 2024
Published in TOMM Volume 20, Issue 12
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Ministry of Science and Technology of Taiwan
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 217Total Downloads
- Downloads (Last 12 months)217
- Downloads (Last 6 weeks)23
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in