Skip to main content
Springer Nature Link
Log in

Retrieval-and-alignment based large-scale indoor point cloud semantic segmentation

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Current methods for point cloud semantic segmentation depend on the extraction of descriptive features. However, unlike images, point clouds are irregular and often lack texture information, making it demanding to extract discriminative features. In addition, noise, outliers, and uneven point distribution are commonly present in point clouds, which further complicates the segmentation task. To address these problems, a novel architecture is proposed for direct and accurate large-scale point cloud segmentation based on point cloud retrieval and alignment. The proposed approach involves using a feature-based point cloud retrieval method for searching for reference point clouds with annotations from a dataset. In the following segmentation stage, an overlap-based point cloud registration method has been developed to align the target and reference point clouds. For accurate and robust alignment, an overlap region estimation module is trained to locate the optimal overlap region between two pieces of point clouds in a coarse-to-fine manner. In the detected overlap region, the global and local features of the points are extracted and combined for feature-metric registration to obtain accurate transformation parameters between the target and reference point clouds. After alignment, the annotated segmentation of the reference is transferred to the target point clouds to obtain accurate segmentation results. Extensive experiments are conducted to show that the developed method outperforms the state-of-the-art approaches in terms of both accuracy and robustness against noise and outliers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Li J L, Dai H, Ding Y. Self-distillation for robust LiDAR semantic segmentation in autonomous driving. In: Proceedings of the European Conference on Computer Vision, 2022. 659–676

  2. Liu B S, Chen X M, Han Y H, et al. Accelerating DNN-based 3D point cloud processing for mobile computing. Sci China Inf Sci, 2019, 62: 212102

    Article  MathSciNet  Google Scholar 

  3. Moyano J, León J, Nieto-Julián J E, et al. Semantic interpretation of architectural and archaeological geometries: point cloud segmentation for HBIM parameterisation. Automat Constr, 2021, 130: 103856

    Article  Google Scholar 

  4. Xia T, Yang J, Chen L. Automated semantic segmentation of bridge point cloud based on local descriptor and machine learning. Automat Constr, 2022, 133: 103992

    Article  Google Scholar 

  5. Ni H, Lin X G, Zhang J X. Classification of ALS point cloud with improved point cloud segmentation and random forests. Remote Sens, 2017, 9: 288

    Article  Google Scholar 

  6. Chiang Y, Hsu C, Tsai A. Fast multi-resolution spatial clustering for 3D point cloud data. In: Proceedings of the IEEE International Conference on Systems, Man and Cybernetics (SMC), 2019. 1678–1683

  7. Schmidt A, Rottensteiner F, Sörgel U. Classification of airborne laser scanning data in Wadden sea areas using conditional random fields. Int Arch Photogramm Remote Sens Spatial Inf Sci, 2012, 39: 161–166

    Article  Google Scholar 

  8. Ren D Y, Wu Z Y, Li J W, et al. Point attention network for point cloud semantic segmentation. Sci China Inf Sci, 2022, 65: 192104

    Article  Google Scholar 

  9. Thomas H, Qi C R, Deschaud J, et al. KPConv: flexible and deformable convolution for point clouds. In: Proceedings of the IEEE International Conference on Computer Vision, 2019. 6411–6420

  10. Qi C R, Su H, Mo K C, et al. PointNet: deep learning on point sets for 3D classification and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 652–660

  11. Qi C R, Yi L, Su H, et al. PointNet++: deep hierarchical feature learning on point sets in a metric space. In: Proceedings of the Advances in Neural Information Processing Systems, 2017. 5099–5108

  12. Tang L Y, Zhan Y B, Chen Z, et al. Contrastive boundary learning for point cloud segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, 2022. 8489–8499

  13. Yang B, Luo W J, Urtasun R. PIXOR: real-time 3D object detection from point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 7652–7660

  14. Meng H, Gao L, Lai Y K, et al. VV-Net: voxel VAE net with group convolutions for point cloud segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 8500–8508

  15. Choy C, Gwak J, Savarese S. 4D spatio-temporal ConvNets: Minkowski convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 3075–3084

  16. Zhang C, Luo W J, Urtasun R. Efficient convolutions for real-time semantic segmentation of 3D point clouds. In: Proceedings of the International Conference on 3D Vision (3DV), 2018. 399–408

  17. Huang X S, Qu W T, Zuo Y F, et al. GMF: general multimodal fusion framework for correspondence outlier rejection. IEEE Robot Autom Lett, 2022, 7: 12585–12592

    Article  Google Scholar 

  18. Huang X S, Wang Y F, Li S, et al. Robust real-world point cloud registration by inlier detection. Comput Vision Image Understanding, 2022, 224: 103556

    Article  Google Scholar 

  19. Xu Z Y, Yuan B, Zhao S S, et al. Hierarchical point-based active learning for semi-supervised point cloud semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. 18098–18108

  20. Hu Q Y, Yang B, Xie L H, et al. RandLA-Net: efficient semantic segmentation of large-scale point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 11108–11117

  21. Wang P S. OctFormer: octree-based transformers for 3D point clouds. 2023. ArXiv:2305.03045

  22. Yang Z T, Jiang L, Sun Y N, et al. A unified query-based paradigm for point cloud understanding. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 8541–8551

  23. Lai X, Liu J H, Jiang L, et al. Stratified transformer for 3D point cloud segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 8500–8509

  24. Wu X Y, Lao Y X, Jiang L, et al. Point transformer V2: grouped vector attention and partition-based pooling. In: Proceedings of the Advances in Neural Information Processing Systems, 2022. 33330–33342

  25. Landrieu L, Simonovsky M. Large-scale point cloud semantic segmentation with superpoint graphs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018. 4558–4567

  26. Shi W J, Rajkumar R. Point-GNN: graph neural network for 3D object detection in a point cloud. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 1711–1719

  27. Bardera A, Feixas M, Boada I, et al. Registration-based segmentation using the information bottleneck method. In: Proceedings of the Iberian Conference on Pattern Recognition and Image Analysis, 2007. 130–137

  28. Rueckert D, Schnabel J. A. Registration and segmentation in medical imaging. In: Registration and Recognition in Images and Videos. Berlin: Springer, 2014. 137–156

  29. Choy C, Park J, Koltun V. Fully convolutional geometric features. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 8958–8966

  30. Ronneberger O, Fischer P, Brox T. U-Net: convolutional networks for biomedical image segmentation. In: Proceedings of the International Conference on Medical Image Computing and Computer-assisted Intervention, 2015. 234–241

  31. Huang S Y, Gojcic Z, Usvyatsov M, et al. PREDATOR: registration of 3D point clouds with low overlap. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 4267–4276

  32. Huang X S, Mei G F, Zhang J. Feature-metric registration: a fast semi-supervised approach for robust point cloud registration without correspondences. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 11366–11374

  33. Armeni I, Sax S, Zamir A R, et al. Joint 2D-3D-semantic data for indoor scene understanding. 2017. ArXiv:1702.01105

  34. Dai A, Chang A X, Savva M, et al. ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017. 5828–5839

  35. Zhao H S, Jiang L, Fu C, et al. PointWeb: enhancing local neighbourhood features for point cloud processing. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 5565–5573

  36. Li Y Y, Bu R, Sun M C, et al. PointCNN: convolution on X-transformed points. In: Proceedings of the Advances in Neural Information Processing Systems, 2018. 820–830

  37. Yan X, Zheng C D, Li Z, et al. PointASNL: robust point clouds processing using nonlocal neural networks with adaptive sampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 5589–5598

  38. Zhao H S, Jiang L, Jia J Y, et al. Point transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 16259–16268

  39. Li G H, Muller M, Thabet A, et al. DeepGCNs: can GCNs go as deep as CNNs? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019. 9267–9276

  40. Wu W X, Qi Z G, Fuxin L. PointConv: deep convolutional networks on 3D point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019. 9621–9630

  41. Kundu A, Yin X Q, Fathi A, et al. Virtual multi-view fusion for 3D semantic segmentation. In: Proceedings of the European Conference on Computer Vision, 2020. 518–535

  42. Hu Z Y, Bai X Y, Shang J X, et al. VMNet: voxel-mesh network for geodesic-aware 3D semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 15488–15498

  43. Luo S T, Hu W. Score-based point cloud denoising. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021. 4583–4592

  44. Choy C, Gwak J, Savarese S. 4D spatiotemporal convnets: minkowski convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019. 3075–3084

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant Nos. 62206033, 62221005, U22A2096), Natural Science Foundation of Chongqing (Grant Nos. cstc2020jcyj-msxmX0855, cstc2021ycjh-bgzxm0339), and Chongqing Postdoctoral Research Special Funding Project (Grant No. 2021XM2044).

Author information

Author notes

  1. Xu Z Y and Huang X S have the same contribution to this work.

Authors and Affiliations

  1. School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China

    Zongyi Xu, Bo Yuan, Yangfu Wang, Weisheng Li & Xinbo Gao

  2. Guangyang Bay Laboratory, Chongqing Institute for Brain and Intelligence, Chongqing, 400064, China

    Zongyi Xu & Xinbo Gao

  3. Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China

    Xiaoshui Huang

  4. School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E1 4NS, UK

    Qianni Zhang

Authors
  1. Zongyi Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoshui Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Bo Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yangfu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Qianni Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Weisheng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Xinbo Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xinbo Gao.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xu, Z., Huang, X., Yuan, B. et al. Retrieval-and-alignment based large-scale indoor point cloud semantic segmentation. Sci. China Inf. Sci. 67, 142104 (2024). https://doi.org/10.1007/s11432-022-3928-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-022-3928-x

Keywords

Search

Navigation