Skip to main content
SpringerLink
Log in

Learning Geometric Transformation for Point Cloud Completion

  • Manuscript
  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Point cloud completion aims to estimate the missing shape from a partial point cloud. Existing encoder-decoder based generative models usually reconstruct the complete point cloud from the learned distribution of the shape prior, which may lead to distortion of geometric details (such as sharp structures and structures without smooth surfaces) due to the information loss of the latent space embedding. To address this problem, we formulate point cloud completion as a geometric transformation problem and propose a simple yet effective geometric transformation network (GTNet). It exploits the repetitive geometric structures in common 3D objects to recover the complete shapes, which contains three sub-networks: geometric patch network, structure transformation network, and detail refinement network. Specifically, the geometric patch network iteratively discovers repetitive geometric structures that are related or similar to the missing parts. Then, the structure transformation network uses the discovered geometric structures to complete the corresponding missing parts by learning their spatial transformations such as symmetry, rotation, translation, and uniform scaling. Finally, the detail refinement network performs global optimization to eliminate unnatural structures. Extensive experiments demonstrate that the proposed method outperforms the state-of-the-art methods on the Shape-Net55-34, MVP, PCN, and KITTI datasets. Models and code will be available at https://github.com/ivislabhit/GTNet.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

All datasets generated or analyzed during the current study are included in the published articles Yu et al. (2021); Pan et al. (2021). These datasets can be derived from the following public-domain resources: https://github.com/paul007pl/VRCNet, https://github.com/yuxumin/PoinTr/blob/master/DATASET.md.

References

  • Achlioptas, P., Diamanti, O., Mitliagkas, I., & Guibas, L. (2018). Learning representations and generative models for 3D point clouds. In International conference on machine learning (pp. 40–49).

  • Arjovsky, M., Chintala, S., & Bottou, L. (2017). Wasserstein generative adversarial networks. In International conference on machine learning (pp. 214–223).

  • Buades, A., Duran, J., & Navarro, J. (2019). Motion-compensated spatio-temporal filtering for multi-image and multimodal super-resolution. International Journal of Computer Vision, 127(10), 1474–1500.

    Article  Google Scholar 

  • Chang, A. X., Funkhouser, T., Guibas, L. J., Hanrahan, P., Huang, Q., Li, Z., Savarese, S., Savva, M., Song, S., Su, H., Xiao, J., Yi, L., & Yu, F. (2015). Shapenet: An information-rich 3D model repository. arXiv:1512.03012

  • Chen, X., Chen, B., & Mitra, N. J. (2020). Unpaired point cloud completion on real scans using adversarial training. In International conference on learning representations.

  • Chrysos, G. G., Kossaifi, J., & Zafeiriou, S. (2020). RoCGAN: Robust conditional GAN. International Journal of Computer Vision, 128(10), 2665–2683.

    Article  MathSciNet  MATH  Google Scholar 

  • Dai, A., Ruizhongtai Qi, C., Nießner, M. (2017). Shape completion using 3d-encoder-predictor cnns and shape synthesis. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5868–5877).

  • Fei, B., Yang, W., Chen, W. M., & Ma, L. (2022b). VQ-DcTr: Vector-quantized autoencoder with dual-channel transformer points splitting for 3D point cloud completion. In Proceedings of the 30th ACM international conference on multimedia (pp. 4769–4778).

  • Fei, B., Yang, W., Chen, W., Li, Z., Li, Y., Ma, T., Hu, X., & Ma, L. (2022a). Comprehensive review of deep learning-based 3D point clouds completion processing and analysis. arXiv preprint arXiv:2203.03311

  • Fu, Y., Lam, A., Sato, I., & Sato, Y. (2017). Adaptive spatial-spectral dictionary learning for hyperspectral image restoration. International Journal of Computer Vision, 122(2), 228–245.

    Article  MathSciNet  MATH  Google Scholar 

  • Geiger, A., Lenz, P., Stiller, C., & Urtasun, R. (2013). Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11), 1231–1237.

    Article  Google Scholar 

  • Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672–2680).

  • Graham, B., Engelcke, M., & van der Maaten, L. (2018). 3D semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 9224–9232).

  • Groueix, T., Fisher, M., Kim, V. G., Russell, B. C., & Aubry, M. (2018). AtlasNet: A Papier-Mâché Approach to Learning 3D Surface Generation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 216–224).

  • Guo, M. H., Cai, J. X., Liu, Z. N., Mu, T. J., Martin, R. R., & Hu, S. M. (2021). Pct: Point cloud transformer. Computational Visual Media, 7(2), 187–199.

    Article  Google Scholar 

  • Han, X., Li, Z., Huang, H., Kalogerakis, E., & Yu, Y. (2017) High-resolution shape completion using deep neural networks for global structure and local geometry inference. In Proceedings of the IEEE international conference on computer vision (pp. 85–93).

  • Huang, Z., Yu, Y., Xu, J., Ni, F., Le, X. (2020). PF-Net: Point fractal network for 3D point cloud completion. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7662–7670).

  • Hui, L., Xu, R., Xie, J., Qian, J., & Yang, J. (2020). Progressive point cloud deconvolution generation network. In European conference on computer vision (pp. 397–413).

  • Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. In International conference on learning representations.

  • Klokov, R., & Lempitsky, V. (2017). Escape from cells: Deep KD-networks for the recognition of 3D point cloud models. In Proceedings of the IEEE international conference on computer vision (pp. 863–872).

  • Li, R., Li, X., Fu, C. W., Cohen-Or, D., & Heng, P. A. (2019). PU-GAN: A point cloud upsampling adversarial network. In Proceedings of the IEEE international conference on computer vision (pp. 7203–7212).

  • Liu, M., Sheng, L., Yang, S., Shao, J., & Hu, S. M. (2020) Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAI conference on artificial intelligence (pp. 11,596–11,603).

  • Lyu, Z., Kong, Z., Xu, X., Pan, L., & Lin, D. (2022). A conditional point diffusion-refinement paradigm for 3D point cloud completion. In International conference on learning representations.

  • Mark, P., Niloy, J. M., Johannes, W., Helmut, P., & Leonidas, J. G. (2008). Discovering structural regularity in 3D geometry. ACM Transactions on Graphics, 27(3), 43.

    Google Scholar 

  • Niloy, J. M., Leonidas, J. G., & Mark, P. (2006). Partial and approximate symmetry detection for 3D geometry. ACM Transactions on Graphics, 25(3), 560–568.

    Article  Google Scholar 

  • Pan, L., Chen, X., Cai, Z., Zhang, J., Zhao, H., Yi, S., & Liu, Z. (2021). Variational relational point completion network. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8524–8533).

  • Pan, L. (2020). ECG: Edge-aware point cloud completion with graph convolution. IEEE Robotics and Automation Letters, 5(3), 4392–4398.

    Article  Google Scholar 

  • Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., & Chintala, S. (2019) Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems (pp. 8024–8035).

  • Qi, C. R., Su, H., Mo, K., Guibas, L. J. (2017a). PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 652–660).

  • Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017b). Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems (pp. 5099–5108).

  • Riegler, G., Osman Ulusoy, A., & Geiger, A. (2017). OctNet: Learning deep 3D representations at high resolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3577–3586).

  • Sarmad, M., Lee, H. J., Kim, Y. M. (2019). RL-GAN-Net: A reinforcement learning agent controlled gan network for real-time point cloud shape completion. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 5898–5907).

  • Schiebener, D., Schmidt, A., Vahrenkamp, N., & Asfour, T. (2016). Heuristic 3D object shape completion based on symmetry and scene context. In Proceedings of the IEEE/RSJ international conference on intelligent robots and systems (pp. 74–81).

  • Shu, D. W., Park, S. W., & Kwon, J. (2019) 3D point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3859–3868).

  • Sipiran, I., Mendoza, A., Apaza, A., & Lopez, C. (2022). Data-driven restoration of digital archaeological pottery with point cloud analysis. International Journal of Computer Vision, 130, 1–17.

    Article  Google Scholar 

  • Stutz, D., & Geiger, A. (2018). Learning 3D shape completion from laser scan data with weak supervision. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1955–1964).

  • Stutz, D., & Geiger, A. (2020). Learning 3D shape completion under weak supervision. International Journal of Computer Vision, 128(5), 1162–1181.

    Article  MATH  Google Scholar 

  • Su, H., Jampani, V., Sun, D., Maji, S., Kalogerakis, E., Yang, M. H., & Kautz, J. (2018). Splatnet: Sparse lattice networks for point cloud processing. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2530–2539).

  • Sung, M., Kim, V. G., Angst, R., & Guibas, L. (2015). Data-driven structural priors for shape completion. ACM Transactions on Graphics (TOG), 34(6), 1–11.

    Article  Google Scholar 

  • Tatarchenko, M., Dosovitskiy, A., & Brox, T. (2017). Octree generating networks: Efficient convolutional architectures for high-resolution 3D outputs. In Proceedings of the IEEE international conference on computer vision (pp. 2088–2096).

  • Tchapmi, L. P., Kosaraju, V., Rezatofighi, H., Reid, I., & Savarese, S. (2019). Topnet: Structural point cloud decoder. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 383–392).

  • Thrun, S., & Wegbreit, B. (2005). Shape from symmetry. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 1824–1831).

  • Wang, X., Ang, Jr. M. H, & Lee, G. H. (2020b). Cascaded refinement network for point cloud completion. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 790–799).

  • Wang, X., Ang, Jr. M. H., & Lee, G. H. (2020a). Point cloud completion by learning shape priors. In 2020 IEEE/RSJ international conference on intelligent robots and systems (pp. 10,719–10,726).

  • Wang, Y., Tan, D. J., Navab, N., & Tombari, F. (2019). Forknet: Multi-branch volumetric semantic completion from a single depth image. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 8608–8617).

  • Wang, Y., Tan, D. J., Navab, N., & Tombari, F. (2020c). Softpoolnet: Shape descriptor for point cloud completion and classification. In European conference on computer vision (pp. 70–85). Springer.

  • Wang, Y., Tan, D. J., Navab, N., & Tombari, F. (2022a). Learning local displacements for point cloud completion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1568–1577).

  • Wang, X., Ang, M. H., & Lee, G. H. (2021). Cascaded refinement network for point cloud completion with self-supervision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11), 8139–8150.

    Google Scholar 

  • Wang, P. S., Sun, C. Y., Liu, Y., & Tong, X. (2018). Adaptive ocnn: A patch-based deep representation of 3d shapes. ACM Transactions on Graphics, 37(6), 1–11.

    Google Scholar 

  • Wang, Y., Tan, D. J., Navab, N., & Tombari, F. (2022). Softpool++: An encoder-decoder network for point cloud completion. International Journal of Computer Vision, 130(5), 1145–1164.

    Article  Google Scholar 

  • Wen, X., Han, Z., Cao, Y. P., Wan, P., Zheng, W., & Liu, Y. S, (2021a). Cycle4completion: Unpaired point cloud completion using cycle transformation with missing region coding. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 13,080–13,089).

  • Wen, X., Li, T., Han, Z., & Liu, Y. S. (2020). Point cloud completion by skip-attention network with hierarchical folding. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1939–1948).

  • Wen, X., Xiang, P., Han, Z., Cao, Y. P., Wan, P., Zheng, W., & Liu, Y. S. (2021b). Pmp-net: Point cloud completion by learning multi-step point moving paths. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7443–7452).

  • Wen, X., Xiang, P., Han, Z., Cao, Y. P., Wan, P., Zheng, W., & Liu, Y. S. (2022). Pmp-net++: Point cloud completion by transformer-enhanced multi-step point moving paths. IEEE Transactions on Pattern Analysis and Machine Intelligence.

  • Wu, R., Chen, X., Zhuang, Y., & Chen, B. (2020). Multimodal shape completion via conditional generative adversarial networks. In European conference on computer vision (pp. 281–296).

  • Wu, T., Pan, L., Zhang, J., Wang, T., Liu, Z., & Lin, D. (2021). Density-aware chamfer distance as a comprehensive metric for point cloud completion. In Advances in neural information processing systems.

  • Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., & Xiao, J. (2015). 3D shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1912–1920).

  • Xia, Y., Xia, Y., Li, W., Song, R., Cao, K., & Stilla, U. (2021). Asfm-net: Asymmetrical siamese feature matching network for point completion. In Proceedings of the 29th ACM international conference on multimedia (pp. 1938–1947).

  • Xiang, P., Wen, X., Liu, Y. S., Cao, Y. P., Wan, P., Zheng, W., & Han, Z. (2021). Snowflakenet: Point cloud completion by snowflake point deconvolution with skip-transformer. In Proceedings of the IEEE international conference on computer vision (pp. 5499–5509).

  • Xie, C., Wang, C., Zhang, B., Yang, H., Chen, D., & Wen, F. (2021). Style-based point generator with adversarial rendering for point cloud completion. In Proceedings of the IEEE Conference on computer vision and pattern recognition (pp. 4619–4628).

  • Xie, H., Yao, H., Sun, X., Zhou, S., & Zhang, S. (2019). Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 2690–2698).

  • Xie, H., Yao, H., Zhou, S., Mao, J., Zhang, S., & Sun, W. (2020b) GRNet: Gridding residual network for dense point cloud completion. In European conference on computer vision (pp. 365–381).

  • Xie, H., Yao, H., Zhang, S., Zhou, S., & Sun, W. (2020). Pix2Vox++: Multi-scale context-aware 3D object reconstruction from single and multiple images. International Journal of Computer Vision, 128(12), 2919–2935.

  • Yan, W., Zhang, R., Wang, J., Liu, S., Li, T. H., & Li, G. (2020). Vaccine-style-net: Point cloud completion in implicit continuous function space. In Proceedings of the 28th ACM international conference on multimedia (pp. 2067–2075).

  • Yang Y., Feng C., Shen Y., & Tian, D. (2018) Foldingnet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 206–215).

  • Yang, Y., Cao, Q., Zhang, J., & Tao, D. (2022). Codon: On orchestrating cross-domain attentions for depth super-resolution. International Journal of Computer Vision, 130(2), 267–284.

    Article  Google Scholar 

  • Yu X., Rao Y., Wang Z., Liu Z., Lu J., & Zhou, J. (2021). PoinTr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE international conference on computer vision (pp. 12,498–12,507).

  • Yuan, W., Khot, T., Held, D., Mertz, C., & Hebert, M. (2018). PCN: Point completion network. In International conference on 3D vision (3DV) (pp. 728–737).

  • Zhang, W., Yan, Q., & Xiao, C. (2020a). Detail preserved point cloud completion via separated feature aggregation. In European conference on computer vision (pp. 512–528).

  • Zhang, X., Dong, H., Hu, Z., Lai, W. S., Wang, F., & Yang, M. H. (2020). Gated fusion network for degraded image super resolution. International Journal of Computer Vision, 128(6), 1699–1721.

    Article  Google Scholar 

  • Zhang, H., Li, Y., Chen, H., Gong, C., Bai, Z., & Shen, C. (2022). Memory-efficient hierarchical neural architecture search for image restoration. International Journal of Computer Vision, 130(1), 157–178.

    Article  Google Scholar 

  • Zhao, Y., Zhou, Y., Chen, R., Hu, B., & Ai, X. (2021). MM-Flow: Multi-modal flow network for point cloud completion. In Proceedings of the 29th ACM international conference on multimedia (pp. 3266–3274).

  • Zheng, C., Cham, T. J., & Cai, J. (2021). Pluralistic free-form image completion. International Journal of Computer Vision, 129(10), 2786–2805.

    Article  Google Scholar 

  • Zhou, H., Cao, Y., Chu, W., Zhu, J., Lu, T., Tai, Y., & Wang, C. (2022). Seedformer: Patch seeds based point cloud completion with upsample transformer. In European conference on computer vision (pp. 416–432).

  • Zhou, L., Du, Y., & Wu, J. (2021). 3D shape generation and completion through point-voxel diffusion. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 5826–5835).

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Nos. 62272134 and 62236003), in part by the Taishan Scholars Program of Shandong Province (No. tsqn201812106), in part by the Shenzhen Colleges and Universities Stable Support Program (No. GXWD20220817144428005), in part by the National Key R &D Program of China (No. 2021ZD0110901).

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, Weihai, China

    Shengping Zhang & Xianzhu Liu

  2. Tencent AI Lab, Shenzhen, China

    Haozhe Xie

  3. Harbin Institute of Technology, Shenzhen, China

    Liqiang Nie

  4. University of Leicester, Leicester, UK

    Huiyu Zhou

  5. The University of Sydney, Camperdown, Australia

    Dacheng Tao

  6. Northwestern Polytechnical University, Xi’an, China

    Xuelong Li

Authors
  1. Shengping Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xianzhu Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haozhe Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Liqiang Nie
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Huiyu Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Dacheng Tao
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xuelong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Liqiang Nie.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, S., Liu, X., Xie, H. et al. Learning Geometric Transformation for Point Cloud Completion. Int J Comput Vis 131, 2425–2445 (2023). https://doi.org/10.1007/s11263-023-01820-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-023-01820-y

Keywords

Search

Navigation