skip to main content
10.1145/3503161.3548215acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Rate-Distortion-Guided Learning Approach with Cross-Projection Information for V-PCC Fast CU Decision

Published: 10 October 2022 Publication History

Abstract

In video-based point cloud compression (V-PCC), the 3D dynamic point cloud sequence is projected into 2D sequences for compression by utilizing the mature 2D video encoder. It is noted the encoding of attribute sequence is extremely time-consuming, and the applicable fast algorithms are still lacking because of the uniqueness of video content and coding structure in V-PCC. This paper proposes a novel rate-distortion-guided fast attribute coding unit (CU) partitioning approach with cross-projection information in V-PCC all-intra (AI) coding. By analyzing the guidance effectiveness of cross-projection information for attribute CU partition, we first propose to combine the occupancy, geometry and attribute features for CU division determination. Afterward, considering that different CUs have different rate-distortion costs and the influences on coding performances by inaccurate different CU predictions are also dissimilar, we devise a rate-distortion-guided learning approach to reduce the coding loss generated by the mispredictions of CU partition. Moreover, we carefully design an overall decision framework for CU partition in V-PCC AI coding structure. Experimental results prove the advantages of our approach, where the coding time is saved by 62.41%, and the End-to-End BD-TotalRate loss only is 0.27%. To the best of our knowledge, the proposed fast attribute CU decision approach achieves the state-of-the-art performance in V-PCC AI coding.

Supplementary Material

MP4 File (MM22-fp1900.mp4)
Presentation video

References

[1]
Sangsoo Ahn, Bumshik Lee, and Munchurl Kim. 2015. A Novel Fast CU Encoding Scheme Based on Spatiotemporal Encoding Parameters for HEVC Inter Coding. IEEE Transactions on Circuits and Systems for Video Technology 25, 3 (2015), 422--435. https://doi.org/10.1109/TCSVT.2014.2360031
[2]
G. Bjøntegaard. VCEG, Austin, TX, USA, Apr. VCEG-M33, 2001. Calculation of Average PSNR Differences between RD-Curves. (VCEG, Austin, TX, USA, Apr. VCEG-M33, 2001).
[3]
Yangang Cai, Xufeng Li, Yueming Wang, and Ronggang Wang. 2022. An Overview of Panoramic Video Projection Schemes in the IEEE 1857.9 Standard for Immersive Visual Content Coding. IEEE Transactions on Circuits and Systems for Video Technology (2022).
[4]
Yangang Cai, Ronggang Wang, Zhenyu Wang, Bingjie Han, and Xufeng Li. 2021. An efficient and open source encoder uavs3e for video compression. In 2021 IEEE International Conference on Multimedia and Expo (ICME). IEEE, 1--6.
[5]
Chao Cao, Marius Preda, Vladyslav Zakharchenko, Euee S. Jang, and Titus Zaharia. 2021. Compression of Sparse and Dense Dynamic Point Clouds-Methods and Standards. Proc. IEEE 109, 9 (2021), 1537--1558. https://doi.org/10.1109/JPROC.2021.3085957
[6]
Jiawei Chen and Lu Yu. 2016. Effective HEVC Intra Coding Unit Size Decision based on Online Progressive Bayesian Classification. In 2016 IEEE International Conference on Multimedia and Expo (ICME). 1--6. https://doi.org/10.1109/ICME.2016.7552970
[7]
Jing Cui, Tao Zhang, Chenchen Gu, Xinfeng Zhang, and Siwei Ma. 2020. Gradient-Based Early Termination of CU Partition in VVC Intra Coding. In 2020 Data Compression Conference (DCC). 103--112. https://doi.org/10.1109/DCC47342.2020.00018
[8]
James H. Donnelly and Robert E. Shannon. 1981. Minimum Mean-Squared-Error Estimators for Simulation Experiments. Commun. ACM 24, 4 (apr 1981), 253--259. https://doi.org/10.1145/358598.358637
[9]
Fanyi Duanmu, Zhan Ma, and Yao Wang. 2016. Fast Mode and Partition Decision Using Machine Learning for Intra-Frame Coding in HEVC Screen Content Coding Extension. IEEE Journal on Emerging and Selected Topics in Circuits and Systems 6, 4 (2016), 517--531. https://doi.org/10.1109/JETCAS.2016.2597698
[10]
Chang-Hong Fu, Yui-Lam Chan, Hong-Bin Zhang, Sik Ho Tsang, and Meng- Ting Xu. 2021. Efficient Depth Intra Frame Coding in 3D-HEVC by Corner Points. IEEE Transactions on Image Processing 30 (2021), 1608--1622. https://doi.org/10.1109/TIP.2020.3046866
[11]
Hamza Hamout and Abderrahmane Elyousfi. 2020. Fast Depth Map Intra Coding for 3D Video Compression-Based Tensor Feature Extraction and Data Analysis. IEEE Transactions on Circuits and Systems for Video Technology 30, 7 (2020), 1933--1945. https://doi.org/10.1109/TCSVT.2019.2918770
[12]
Euee S. Jang, Marius Preda, Khaled Mammou, Alexis M. Tourapis, Jungsun Kim, Danillo B. Graziosi, Sungryeul Rhyu, and Madhukar Budagavi. 2019. Video- Based Point-Cloud-Compression Standard in MPEG: From Evidence Collection to Committee Draft [Standards in a Nutshell]. IEEE Signal Processing Magazine 36, 3 (2019), 118--123. https://doi.org/10.1109/MSP.2019.2900721
[13]
Kyungah Kim and Won Woo Ro. 2019. Fast CU Depth Decision for HEVC Using Neural Networks. IEEE Transactions on Circuits and Systems for Video Technology 29, 5 (2019), 1462--1473. https://doi.org/10.1109/TCSVT.2018.2839113
[14]
Wei Kuang, Yui-Lam Chan, Sik-Ho Tsang, and Wan-Chi Siu. 2020. Online-Learning-Based Bayesian Decision Rule for Fast Intra Mode and CU Partitioning Algorithm in HEVC Screen Content Coding. IEEE Transactions on Image Processing 29 (2020), 170--185. https://doi.org/10.1109/TIP.2019.2924810
[15]
Yao-Tsung Kuo, Pei-Yin Chen, and Hong-Cheng Lin. 2020. A Spatiotemporal Content-Based CU Size Decision Algorithm for HEVC. IEEE Transactions on Broadcasting 66, 1 (2020), 100--112. https://doi.org/10.1109/TBC.2019.2960938
[16]
Jianjun Lei, Jinhui Duan, Feng Wu, Nam Ling, and Chunping Hou. 2018. Fast Mode Decision Based on Grayscale Similarity and Inter-View Correlation for Depth Map Coding in 3D-HEVC. IEEE Transactions on Circuits and Systems for Video Technology 28, 3 (2018), 706--718. https://doi.org/10.1109/TCSVT.2016.2617332
[17]
Jianjun Lei, Dongyang Li, Zhaoqing Pan, Zhenyan Sun, Sam Kwong, and Chunping Hou. 2017. Fast Intra Prediction Based on Content Property Analysis for Low Complexity HEVC-Based Screen Content Coding. IEEE Transactions on Broadcasting 63, 1 (2017), 48--58. https://doi.org/10.1109/TBC.2016.2623241
[18]
Li Li, Zhu Li, Shan Liu, and Houqiang Li. 2021. Occupancy-Map-Based Rate Distortion Optimization and Partition for Video-Based Point Cloud Compression. IEEE Transactions on Circuits and Systems for Video Technology 31, 1 (2021), 326--338. https://doi.org/10.1109/TCSVT.2020.2966118
[19]
Tianyi Li, Mai Xu, Runzhi Tang, Ying Chen, and Qunliang Xing. 2021. DeepQTMT: A Deep Learning Approach for Fast QTMT-Based CU Partition of Intra-Mode VVC. IEEE Transactions on Image Processing 30 (2021), 5377--5390. https://doi.org/10.1109/TIP.2021.3083447
[20]
Jie-Ru Lin, Mei-Juan Chen, Chia-Hung Yeh, Yong-Ci Chen, Lih-Jen Kau, Chuan-Yu Chang, and Min-Hui Lin. 2021. Visual Perception Based Algorithm for Fast Depth Intra Coding of 3D-HEVC. IEEE Transactions on Multimedia (2021), 1--1. https://doi.org/10.1109/TMM.2021.3070106
[21]
Ting-Lan Lin, Hong-Bin Bu, Yan-Cheng Chen, Jun-Rui Yang, Chi-Fu Liang, Kun-Hu Jiang, Ching-Hsuan Lin, and Xiao-Feng Yue. 2021. Efficient Quadtree Search for HEVC Coding Units for V-PCC. IEEE Access 9 (2021), 139109--139121. https://doi.org/10.1109/ACCESS.2021.3118806
[22]
Xingang Liu, Yayong Li, Deyuan Liu, Peicheng Wang, and Laurence T. Yang. 2019. An Adaptive CU Size Decision Algorithm for HEVC Intra Prediction Based on Complexity Classification Using Machine Learning. IEEE Transactions on Circuits and Systems for Video Technology 29, 1 (2019), 144--155. https://doi.org/10.1109/TCSVT.2017.2777903
[23]
MPEG. 2021. HEVC Test Model Version 16.20 Screen Content Model Version 8.8, HM-16.20+SCM-8.8. https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.20+SCM-8.8/.
[24]
MPEG. 2021. Point Cloud Compression Category 2 Reference Software, TMC2--12.0. http://mpegx.int-evry.fr/software/MPEG/PCC/TM/mpeg-pcc-tmc2.git.
[25]
Karsten Müller, Heiko Schwarz, Detlev Marpe, Christian Bartnik, Sebastian Bosse, Heribert Brust, Tobias Hinz, Haricharan Lakshman, Philipp Merkle, Franz Hunn Rhee, Gerhard Tech, Martin Winken, and Thomas Wiegand. 2013. 3D High- Efficiency Video Coding for Multi-View Video and Depth Data. IEEE Transactions on Image Processing 22, 9 (2013), 3366--3378. https://doi.org/10.1109/TIP.2013.2264820
[26]
Sang-hyo Park and Je-Won Kang. 2021. Fast Multi-Type Tree Partitioning for Versatile Video Coding Using a Lightweight Neural Network. IEEE Transactions on Multimedia 23 (2021), 4388--4399. https://doi.org/10.1109/TMM.2020.3042062
[27]
Schwarz S and Flynn D. Standard ISO/IEC JTC1/SC29/WG11 MPEG2020/N19324, Apr. 2020. Common test conditions for point cloud compression.
[28]
Liquan Shen, Ping An, Zhaoyang Zhang, Qianqian Hu, and Zhengchuan Chen. 2015. A 3D-HEVC Fast Mode Decision Algorithm for Real-Time Applications. ACM Trans. Multimedia Comput. Commun. Appl. 11, 3, Article 34 (feb 2015), 23 pages. https://doi.org/10.1145/2700298
[29]
Liquan Shen, Zhaoyang Zhang, and Zhi Liu. 2014. Effective CU Size Decision for HEVC Intracoding. IEEE Transactions on Image Processing 23, 10 (2014), 4232--4241. https://doi.org/10.1109/TIP.2014.2341927
[30]
G.J. Sullivan and T. Wiegand. 1998. Rate-Distortion Optimization for Video Compression. IEEE Signal Processing Magazine 15, 6 (1998), 74--90. https://doi.org/10.1109/79.733497
[31]
Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, and Thomas Wiegand. 2012. Overview of the High Efficiency Video Coding (HEVC) Standard. IEEE Trans- actions on Circuits and Systems for Video Technology 22, 12 (2012), 1649--1668. https://doi.org/10.1109/TCSVT.2012.2221191
[32]
Jian Xiong, Hao Gao, Miaohui Wang, Hongliang Li, and Weisi Lin. 2021. Occupancy Map Guided Fast Video-based Dynamic Point Cloud Coding. IEEE Transactions on Circuits and Systems for Video Technology (2021), 1--1. https://doi.org/10.1109/TCSVT.2021.3063501
[33]
Jian Xiong, Hongliang Li, Fanman Meng, Shuyuan Zhu, Qingbo Wu, and Bing Zeng. 2014. MRF-Based Fast HEVC Inter CU Decision With the Variance of Absolute Differences. IEEE Transactions on Multimedia 16, 8 (2014), 2141--2153. https://doi.org/10.1109/TMM.2014.2356795
[34]
Yali Xue, Xu Wang, Linwei Zhu, Zhaoqing Pan, and Sam Kwong. 2019. Fast Coding Unit Decision for Intra Screen Content Coding Based on Ensemble Learning. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 8543--8547. https://doi.org/10.1109/ICASSP.2019.8682707
[35]
Hang Yuan, Wei Gao, and Junle Wang. 2021. Dynamic Computational Resource Allocation for Fast Inter Frame Coding in Video Conferencing Applications. In 2021 IEEE International Conference on Multimedia and Expo (ICME). 1--6. https://doi.org/10.1109/ICME51207.2021.9428275
[36]
Yun Zhang, Sam Kwong, Xu Wang, Hui Yuan, Zhaoqing Pan, and Long Xu. 2015. Machine Learning-Based Coding Unit Depth Decisions for Flexible Complexity Allocation in High Efficiency Video Coding. IEEE Transactions on Image Processing 24, 7 (2015), 2225--2238. https://doi.org/10.1109/TIP.2015.2417498

Cited By

View all
  • (2024)Improved Patch Packing and Refining Segmentation for the V-PCC StandardApplied Sciences10.3390/app1404140514:4(1405)Online publication date: 8-Feb-2024
  • (2024)Improved Just Noticeable Difference Model Based Algorithm for Fast CU Partition in V-PCCIEICE Transactions on Information and Systems10.1587/transinf.2023EDL8090E107.D:8(1101-1104)Online publication date: 1-Aug-2024
  • (2024)Rethinking Feature Mining for Light Field Salient Object DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367696720:10(1-24)Online publication date: 8-Jul-2024
  • Show More Cited By

Index Terms

  1. Rate-Distortion-Guided Learning Approach with Cross-Projection Information for V-PCC Fast CU Decision

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '22: Proceedings of the 30th ACM International Conference on Multimedia
    October 2022
    7537 pages
    ISBN:9781450392037
    DOI:10.1145/3503161
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 10 October 2022

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. V-PCC
    2. coding unit partition
    3. fast algorithm
    4. machine learning

    Qualifiers

    • Research-article

    Funding Sources

    • Natural Science Foundation of China
    • Guangdong Basic and Applied Basic Research Foundation
    • Shenzhen Fundamental Research Program
    • Major Key Project of PCL
    • National Key R&D Program of China
    • Natural Science Foundation of China
    • Shenzhen Science and Technology Plan Basic Research Project

    Conference

    MM '22
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)72
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 14 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Improved Patch Packing and Refining Segmentation for the V-PCC StandardApplied Sciences10.3390/app1404140514:4(1405)Online publication date: 8-Feb-2024
    • (2024)Improved Just Noticeable Difference Model Based Algorithm for Fast CU Partition in V-PCCIEICE Transactions on Information and Systems10.1587/transinf.2023EDL8090E107.D:8(1101-1104)Online publication date: 1-Aug-2024
    • (2024)Rethinking Feature Mining for Light Field Salient Object DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/367696720:10(1-24)Online publication date: 8-Jul-2024
    • (2024)Divide-and-conquer-based RDO-free CU Partitioning for 8K Video CompressionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363470520:4(1-20)Online publication date: 11-Jan-2024
    • (2024)Bandwidth-Efficient Mobile Volumetric Video Streaming by Exploiting Inter-Frame CorrelationIEEE Transactions on Mobile Computing10.1109/TMC.2024.336775023:10(9410-9423)Online publication date: Oct-2024
    • (2024)Low Complexity Coding Unit Decision for Video-Based Point Cloud CompressionIEEE Transactions on Image Processing10.1109/TIP.2023.333763733(149-162)Online publication date: 1-Jan-2024
    • (2024)Leveraging occupancy map to accelerate video-based point cloud compressionJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104292104(104292)Online publication date: Oct-2024
    • (2024)Open-Source Projects for 3D Point CloudsDeep Learning for 3D Point Clouds10.1007/978-981-97-9570-3_9(255-272)Online publication date: 10-Oct-2024
    • (2024)Point Cloud-Language Multi-modal LearningDeep Learning for 3D Point Clouds10.1007/978-981-97-9570-3_8(227-254)Online publication date: 10-Oct-2024
    • (2024)Point Cloud Pre-trained Models and Large ModelsDeep Learning for 3D Point Clouds10.1007/978-981-97-9570-3_7(195-225)Online publication date: 10-Oct-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media