Skip to main content
SpringerLink
Log in

Deep Learning Geometry Compression Artifacts Removal for Video-Based Point Cloud Compression

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Point cloud is an essential format for three-dimensional (3-D) object modelling and interaction in Augmented Reality and Virtual Reality applications. In the current state of the art video-based point cloud compression (V-PCC), a dynamic point cloud is projected onto geometry and attribute videos patch by patch, each represented by its texture, depth, and occupancy map for reconstruction. To deal with occlusion, each patch is projected onto near and far depth fields in the geometry video. Once there are artifacts on the compressed two-dimensional (2-D) geometry video, they would be propagated to the 3-D point-cloud frames. In addition, in the lossy compression, there always exists a tradeoff between the rate of bitstream and distortion. Although some geometry-related methods were proposed to attenuate these artifacts and improve the coding efficiency, the interactive correlation between projected near and far depth fields has been ignored. Moreover, the non-linear representation ability of Convolutional Neural Network has not been fully considered. Therefore, we propose a learning-based approach to remove the geometry artifacts and improve the compressing efficiency. We have the following contributions. We devise a two-step method working on the near and far depth fields decomposed from geometry. The first stage is learning-based Pseudo-Motion Compensation. The second stage exploits the potential of the strong correlations between near and far depth fields. Our proposed algorithm is embedded in the V-PCC reference software. To the best of our knowledge, this is the first learning-based solution of the geometry artifacts removal in V-PCC. The extensive experimental results show that the proposed approach achieves significant gains on geometry artifacts removal and quality improvement of 3-D point-cloud reconstruction compared to the state-of-the-art schemes.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Andrivon, P., Ricard, J., Guede, C., Nakagami, O., Graziosi, D., & Tabatabai, A. (2020). Patch border filtering specification in V-PCC. Document ISO/IEC JTC1/SC29/WG11 m51501, Geneva, CH.

  • Biswas, S., Liu, J., Wong, K., Wang, S., Urtasun, R. (2020). Muscle: Multi sweep compression of lidar using deep entropy models. In: Larochelle, H., Ranzato, M., Hadsell, R. Balcan, M.F. Lin, H. (eds.) Advances in Neural Information Processing Systems, vol. 33, pp. 22170–22181. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2020/file/fc152e73692bc3c934d248f639d9e963-Paper.pdf.

  • Bross, B., Chen, J., Liu, S. (2019). Versatile Video Coding (Draft 4). Document ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 JVET-M1001-v6, Marrakech, MA.

  • Bruder, G., Steinicke, F., Nüchter, A. (2014). Poster: Immersive point cloud virtual environments. In: 2014 IEEE symposium on 3D user interfaces (3DUI), pp. 161–162. IEEE.

  • Budagavi, M., Faramarzi, E., Ho, T., Najaf-Zadeh, H., Sinharoy, I. (2017). Samsungs response to cfp for point cloud compression (category 2). Document ISO/IEC JTC1/SC29/WG11 m41808, Macau, China.

  • Cai, K., & Ricard, J., C.G., Llach, J., Chevet, J.C. (2018). Geometry image coding improvements. Document ISO/IEC JTC1/SC29/WG11 m42111, Gwangju, Korea.

  • Chen, X., Ma, H., Wan, J., Li, B., Xia, T. (2017). Multi-view 3d object detection network for autonomous driving. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1907–1915.

  • Chen, J., Lin, C., Hsu, P., & Chen, C. (2014). Point cloud encoding for 3d building model retrieval. IEEE Transactions on Multimedia, 16(2), 337–345.

    Article  Google Scholar 

  • Choy, C., Gwak, J., Savarese, S. (2019). 4d spatio-temporal convnets: Minkowski convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3075–3084.

  • Committee, M. (2020). V-PCC codec description. Document ISO/IEC JTC1/SC29/WG11 w19526, Italy.

  • Culture 3D cloud. http://c3dc.fr/.

  • Dawar, N., Najaf-Zadeh, H., Joshi, R., Budagavi, M. (2018). PCC TMC2 Interleaving in geometry and texture layers. Document ISO/IEC JTC1/SC29/WG11 m43723, Ljubljana, Slovenia.

  • de Queiroz, R. L., & Chou, P. A. (2017). Motion-compensated compression of dynamic voxelized point clouds. IEEE Transactions on Image Processing, 26(8), 3886–3895.

    Article  MathSciNet  Google Scholar 

  • d’Eon, E., Harrison, B., Myers, T., Chou, P. (2017). Input to ad hoc groups on mpeg point cloud compression and jpeg pleno. Document ISO/IEC JTC1/SC29/WG11 m40059, Geneva, Switzerland.

  • Fuchs, H., State, A., & Bazin, J. C. (2014). Immersive 3d telepresence. Computer, 47(7), 46–52.

    Article  Google Scholar 

  • Gojcic, Z., Zhou, C., Wegner, J.D., Guibas, L.J., Birdal, T. (2020). Learning multiview 3d point cloud registration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1759–1769.

  • Graziosi, D., Tabatabai, A. (2019). [V-PCC] New contribution on geometry padding. Document ISO/IEC JTC1/SC29/WG11 m47496, Geneva, CH.

  • Guo, K., Xu, F., Yu, T., Liu, X., Dai, Q., & Liu, Y. (2017). Real-time geometry, albedo, and motion reconstruction using a single rgb-d camera. ACM Transactions on Graphics (ToG), 36(4), 1.

    Article  Google Scholar 

  • He, L., Zhu, W., Xu, Y. (2017). Best-effort projection based attribute compression for 3d point cloud. In: 2017 23rd Asia-Pacific conference on communications (APCC), pp. 1–6. IEEE.

  • Huang, T., Liu, Y. (2019). 3d point cloud geometry compression on deep learning. In: Proceedings of the 27th ACM international conference on multimedia, pp. 890–898.

  • Huang, L., Wang, S., Wong, K., Liu, J., Urtasun, R. (2020). Octsqueeze: Octree-structured entropy model for lidar compression. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1313–1323.

  • Jang, E. S., Preda, M., Mammou, K., Tourapis, A. M., Kim, J., Graziosi, D. B., et al. (2019). Video-based point-cloud-compression standard in mpeg: from evidence collection to committee draft [standards in a nutshell]. IEEE Signal Processing Magazine, 36(3), 118–123.

    Article  Google Scholar 

  • Kammerl, J., Blodow, N., Rusu, R.B., Gedikli, S., Beetz, M., Steinbach, E. (2012). Real-time compression of point cloud streams. In: 2012 IEEE international conference on robotics and automation, pp. 778–785. IEEE.

  • Kingma, D.P., Ba, J. (2014). Adam: A method for stochastic optimization. Preprint arXiv:1412.6980

  • Lasserre, S., Llach, J., Guede, C., Ricard, J. (2017). Technicolor’s response to the cfpp for point cloud compression. Document ISO/IEC JTC1/SC29/WG11 m41822, Macau, China.

  • Li, Y., Liu, S., Kawamura, K. (2019). Methodology and reporting template for neural network coding tool testing. Document ITU-T SG 16 WP 3 and ISO/IEC JTC 1/SC 29/WG 11 JVET-M1006-v1, Marrakech, MA.

  • Li, L., Li, Z., Zakharchenko, V., Chen, J., & Li, H. (2019). Advanced 3d motion prediction for video-based dynamic point cloud compression. IEEE Transactions on Image Processing, 29, 289–302.

    Article  MathSciNet  Google Scholar 

  • Mammou, K., Tourapis, A.M., Singer, D., Su, Y. (2017). Video-based and hierarchical approaches point cloud compression. Document ISO/IEC JTC1/SC29/WG11 m41649, Macau, China.

  • Mekuria, R., Blom, K., & Cesar, P. (2016). Design, implementation, and evaluation of a point cloud codec for tele-immersive video. IEEE Transactions on Circuits and Systems for Video Technology, 27(4), 828–842.

    Article  Google Scholar 

  • Mobile Mapping System. http://www.mitsubishielectric.com/bu/mms/index.html.

  • Nakagami, O. (2018). PCC TMC2 low complexity geometry smoothing. Document ISO/IEC JTC1/SC29/WG11 m43501, Ljubjana, SI.

  • Olivier, Y., Llach, J. (2018). Per patch projection optimization for TMC2. Document ISO/IEC JTC1/SC29/WG11 m43723, San Diego, CA, US.

  • Point Cloud Compression Category 2 Reference Software TMC2-8.0. http://mpegx.int-evry.fr/software/MPEG/PCC/TM/mpeg-pcc-tmc2.

  • Preda, M. (2017). Report on pcc cfp answers. Document ISO/IEC JTC1/SC29/WG11 w17251, Macau, China.

  • Qi, C.R., Su, H., Mo, K., Guibas, L.J.(2017) Pointnet: Deep learning on point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652–660.

  • Quach, M., Valenzise, G., Dufaux, F.(2019). Learning convolutional transforms for lossy point cloud geometry compression. In: 2019 IEEE international conference on image processing (ICIP), pp. 4320–4324. IEEE.

  • Rhyu, S., Oh, Y., Woo, J.(2018). PCC CE2.13 report on texture and depth padding improvement. Document ISO/IEC JTC1/SC29/WG11 m43667, Ljubjana, SI.

  • Schwarz, S., Martin-Cocher, G., Flynn, D., Budagavi, M. (2018). Common test conditions for point cloud compression. Document ISO/IEC JTC1/SC29/WG11 w17766, Ljubljana, Slovenia.

  • Schwarz, S., Preda, M., Baroncini, V., Budagavi, M., Cesar, P., Chou, P. A., et al. (2018). Emerging mpeg standards for point cloud compression. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 9(1), 133–148.

    Article  Google Scholar 

  • Sportillo, D., Paljic, A., Boukhris, M., Fuchs, P., Ojeda, L., Roussarie, V.(2017). An immersive virtual reality system for semi-autonomous driving simulation: A comparison between realistic and 6-dof controller-based interaction. In: Proceedings of the 9th international conference on computer and automation engineering, pp. 6–10.

  • Sullivan, G. J., Ohm, J. R., Han, W. J., & Wiegand, T. (2012). Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology, 22(12), 1649–1668.

    Article  Google Scholar 

  • Sun, Y., Liu, M., & Meng, M. Q. H. (2017). Improving rgb-d slam in dynamic environments: A motion removal approach. Robotics and Autonomous Systems, 89, 110–122.

    Article  Google Scholar 

  • Sun, X., Ma, H., Sun, Y., & Liu, M. (2019). A novel point cloud compression algorithm based on clustering. IEEE Robotics and Automation Letters, 4(2), 2132–2139.

  • Thanou, D., Chou, P. A., & Frossard, P. (2016). Graph-based compression of dynamic 3d point cloud sequences. IEEE Transactions on Image Processing, 25(4), 1765–1778.

    Article  MathSciNet  Google Scholar 

  • Tian, D., Ochimizu, H., Feng, C., Cohen, R., Vetro, A. (2017). Geometric distortion metrics for point cloud compression. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 3460–3464. IEEE.

  • Tu, C., Takeuchi, E., Carballo, A., Takeda, K. (2019). Point cloud compression for 3d lidar sensor using recurrent neural network with residual blocks. In: 2019 International Conference on Robotics and Automation (ICRA), pp. 3274–3280. IEEE.

  • Tu, C., Takeuchi, E., Miyajima, C., Takeda, K. (2016). Compressing continuous point cloud data using image compression methods. In: 2016 IEEE 19th international conference on intelligent transportation systems (ITSC), pp. 1712–1719. IEEE.

  • Tulvan, C., Mekuria, R., Li, Z., Laserre, S. (2016). Use cases for point cloud compression (pcc).

  • Voulodimos, A., Doulamis, N., Doulamis, A., & Protopapadakis, E. (2018). Deep learning for computer vision: A brief review. Computational Intelligence and Neuroscience,2018.

  • Wiegand, T., Sullivan, G. J., Bjontegaard, G., & Luthra, A. (2003). Overview of the H. 264/AVC video coding standard. IEEE Transactions on Circuits and Systems for Video Technology, 13(7), 560–576.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Missouri-Kansas City, Kansas City, MO, 64110, USA

    Wei Jia & Zhu Li

  2. University of Science and Technology of China, Hefei, 230026, China

    Li Li

  3. Tencent Media Lab, Palo Alto, CA, 94306, USA

    Shan Liu

Authors
  1. Wei Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Li Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shan Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zhu Li.

Additional information

Communicated by Dong Xu.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jia, W., Li, L., Li, Z. et al. Deep Learning Geometry Compression Artifacts Removal for Video-Based Point Cloud Compression. Int J Comput Vis 129, 2947–2964 (2021). https://doi.org/10.1007/s11263-021-01503-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-021-01503-6

Keywords

Search

Navigation