Skip to main content
Springer Nature Link
Log in

ML-SemReg: Boosting Point Cloud Registration with Multi-level Semantic Consistency

  • Conference paper
  • First Online:
Computer Vision – ECCV 2024 (ECCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15102))

Included in the following conference series:

  • 265 Accesses

Abstract

Recent advances in point cloud registration mostly leverage geometric information. Although these methods have yielded promising results, they still struggle with problems of low overlap, thus limiting their practical usage. In this paper, we propose ML-SemReg, a plug-and-play point cloud registration framework that fully exploits semantic information. Our key insight is that mismatches can be categorized into two types, i.e., inter- and intra-class, after rendering semantic clues, and can be well addressed by utilizing multi-level semantic consistency. We first propose a Group Matching module to address inter-class mismatching, outputting multiple matching groups that inherently satisfy Local Semantic Consistency. For each group, a Mask Matching module based on Scene Semantic Consistency is then introduced to suppress intra-class mismatching. Benefit from those two modules, ML-SemReg generates correspondences with a high inlier ratio. Extensive experiments demonstrate excellent performance and robustness of ML-SemReg, e.g., in hard-cases of the KITTI dataset, the Registration Recall of MAC increases by almost 34% points when our ML-SemReg is equipped. Code is available at https://github.com/Laka-3DV/ML-SemReg.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Aiger, D., Mitra, N.J., Cohen-Or, D.: 4-points congruent sets for robust pairwise surface registration. In: ACM SIGGRAPH 2008 Papers, pp. 1–10 (2008)

    Google Scholar 

  2. Ao, S., Hu, Q., Yang, B., Markham, A., Guo, Y.: SpinNet: learning a general surface descriptor for 3D point cloud registration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11753–11762 (2021)

    Google Scholar 

  3. Arce, J., Vödisch, N., Cattaneo, D., Burgard, W., Valada, A.: PADLoC: LiDAR-based deep loop closure detection and registration using panoptic attention. IEEE Robot. Autom. Lett. 8(3), 1319–1326 (2023)

    Article  Google Scholar 

  4. Azuma, R.T.: A survey of augmented reality. Presence Teleoper. Virtual Environ. 6(4), 355–385 (1997)

    Article  Google Scholar 

  5. Besl, P.J., McKay, N.D.: Method for registration of 3-D shapes. In: Sensor fusion IV: Control Paradigms and Data Structures, vol. 1611, pp. 586–606. International Society for Optics and Photonics (1992)

    Google Scholar 

  6. Bouaziz, S., Tagliasacchi, A., Pauly, M.: Sparse iterative closest point. In: Computer Graphics Forum, vol. 32, pp. 113–123. Wiley Online Library (2013)

    Google Scholar 

  7. Chen, H., Bhanu, B.: 3D free-form object recognition in range images using local surface patches. Pattern Recogn. Lett. 28(10), 1252–1262 (2007)

    Article  Google Scholar 

  8. Chen, X., Milioto, A., Palazzolo, E., Giguère, P., Behley, J., Stachniss, C.: SuMa++: efficient LiDAR-based semantic SLAM. In: Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4530–4537 (2019). https://www.ipb.uni-bonn.de/wp-content/papercite-data/pdf/chen2019iros.pdf

  9. Chen, Z., Sun, K., Yang, F., Tao, W.: SC2-PCR: a second order spatial compatibility for efficient and robust point cloud registration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13221–13231, June 2022 (2022)

    Google Scholar 

  10. Choi, S., Zhou, Q.Y., Koltun, V.: Robust reconstruction of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015 (2015)

    Google Scholar 

  11. Choy, C., Dong, W., Koltun, V.: Deep global registration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2514–2523 (2020)

    Google Scholar 

  12. Choy, C., Park, J., Koltun, V.: Fully convolutional geometric features. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 8958–8966 (2019)

    Google Scholar 

  13. Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: richly-annotated 3D reconstructions of indoor scenes. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5828–5839 (2017)

    Google Scholar 

  14. Deng, H., Birdal, T., Ilic, S.: PPF-FoldNet: unsupervised learning of rotation invariant 3D local descriptors. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11209, pp. 620–638. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01228-1_37

    Chapter  Google Scholar 

  15. Deng, H., Birdal, T., Ilic, S.: PPFNet: global context aware local features for robust 3D point matching. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 195–205 (2018)

    Google Scholar 

  16. Durrant-Whyte, H., Bailey, T.: Simultaneous localization and mapping: part i. IEEE Robot. Autom. Mag. 13(2), 99–110 (2006)

    Article  Google Scholar 

  17. Fischler, M.A., Bolles, R.C.: Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun. ACM 24(6), 381–395 (1981)

    Article  MathSciNet  Google Scholar 

  18. Frome, A., Huber, D., Kolluri, R., Bülow, T., Malik, J.: Recognizing objects in range data using regional point descriptors. In: Pajdla, T., Matas, J. (eds.) ECCV 2004. LNCS, vol. 3023, pp. 224–237. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24672-5_18

    Chapter  Google Scholar 

  19. Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The KITTI vision benchmark suite. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2012)

    Google Scholar 

  20. Guo, Y., Sohel, F., Bennamoun, M., Wan, J., Lu, M.: A novel local surface feature for 3D object recognition under clutter and occlusion. Inf. Sci. 293, 196–213 (2015)

    Article  Google Scholar 

  21. Guo, Y., Sohel, F.A., Bennamoun, M., Wan, J., Lu, M.: RoPS: a local feature descriptor for 3D rigid objects based on rotational projection statistics. In: 2013 1st International Conference on Communications, Signal Processing, and their Applications (ICCSPA), pp. 1–6. IEEE (2013)

    Google Scholar 

  22. He, Y., Ma, L., Jiang, Z., Tang, Y., Xing, G.: VI-eye: semantic-based 3D point cloud registration for infrastructure-assisted autonomous driving. In: Proceedings of the 27th Annual International Conference on Mobile Computing and Networking, pp. 573–586 (2021)

    Google Scholar 

  23. Huang, S., Gojcic, Z., Usvyatsov, M., Wieser, A., Schindler, K.: PREDATOR: registration of 3D point clouds with low overlap. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4267–4276 (2021)

    Google Scholar 

  24. Johnson, A.E., Hebert, M.: Using spin images for efficient object recognition in cluttered 3D scenes. IEEE Trans. Pattern Anal. Mach. Intell. 21(5), 433–449 (1999)

    Article  Google Scholar 

  25. Li, Y., Harada, T.: Lepard: learning partial point cloud matching in rigid and deformable scenes. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2022)

    Google Scholar 

  26. Lindeberg, T.: Scale invariant feature transform (2012)

    Google Scholar 

  27. Liu, C., Guo, J., Yan, D.M., Liang, Z., Zhang, X., Cheng, Z.: SARNet: semantic augmented registration of large-scale urban point clouds. arXiv preprint arXiv:2206.13117 (2022)

  28. Liu, S., et al.: Deep semantic graph matching for large-scale outdoor point cloud registration. IEEE Trans. Geosci. Remote Sens. 62, 1–12 (2024)

    Google Scholar 

  29. Milioto, A., Vizzo, I., Behley, J., Stachniss, C.: RangeNet++: fast and accurate LiDAR semantic segmentation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2019)

    Google Scholar 

  30. Pais, G.D., Ramalingam, S., Govindu, V.M., Nascimento, J.C., Chellappa, R., Miraldo, P.: 3DRegNet: a deep neural network for 3d point registration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7193–7203 (2020)

    Google Scholar 

  31. Poiesi, F., Boscaini, D.: Learning general and distinctive 3d local deep descriptors for point cloud registration. IEEE Trans. Pattern Anal. Mach. Intell. 45, 3979–3985 (2022)

    Google Scholar 

  32. Qiao, Z., Yu, Z., Jiang, B., Yin, H., Shen, S.: G3Reg: pyramid graph-based global registration using gaussian ellipsoid model. arXiv preprint arXiv:2308.11573 (2023)

  33. Qin, Z., Yu, H., Wang, C., Guo, Y., Peng, Y., Xu, K.: Geometric transformer for fast and robust point cloud registration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2022, pp. 11143–11152 (2022)

    Google Scholar 

  34. Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings Third International Conference on 3-D Digital Imaging and Modeling, pp. 145–152. IEEE (2001)

    Google Scholar 

  35. Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: 2009 IEEE International Conference on Robotics and Automation, pp. 3212–3217. IEEE (2009)

    Google Scholar 

  36. Rusu, R.B., Blodow, N., Marton, Z.C., Beetz, M.: Aligning point cloud views using persistent feature histograms. In: 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3384–3391. IEEE (2008)

    Google Scholar 

  37. Segal, A., Haehnel, D., Thrun, S.: Generalized-ICP. In: Robotics: Science and Systems, Seattle, WA, vol. 2, p. 435 (2009)

    Google Scholar 

  38. Tang, H., et al.: Searching efficient 3D architectures with sparse point-voxel convolution. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12373, pp. 685–702. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58604-1_41

    Chapter  Google Scholar 

  39. Thomas, H., Qi, C.R., Deschaud, J.E., Marcotegui, B., Goulette, F., Guibas, L.J.: KPConv: flexible and deformable convolution for point clouds. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6411–6420 (2019)

    Google Scholar 

  40. Tombari, F., Salti, S., Di Stefano, L.: Unique shape context for 3d data description. In: Proceedings of the ACM Workshop on 3D Object Retrieval, pp. 57–62 (2010)

    Google Scholar 

  41. Truong, G., Gilani, S.Z., Islam, S.M.S., Suter, D.: Fast point cloud registration using semantic segmentation. In: 2019 Digital Image Computing: Techniques and Applications, DICTA 2019, Perth, Australia, 2-4 December 2019, pp. 1–8 (2019). https://doi.org/10.1109/DICTA47822.2019.8945870

  42. Wang, H., Liu, Y., Dong, Z., Wang, W.: You only hypothesize once: Point cloud registration with rotation-equivariant descriptors. In: Proceedings of the 30th ACM International Conference on Multimedia, pp. 1630–1641 (2022)

    Google Scholar 

  43. Wu, X., Lao, Y., Jiang, L., Liu, X., Zhao, H.: Point transformer v2: grouped vector attention and partition-based pooling. In: NeurIPS (2022)

    Google Scholar 

  44. Bai, X., Luo, Z., Zhou, L., Fu, H., Quan, L., Tai, C.-L.: D3Feat: joint learning of dense detection and description of 3D local features. arXiv arXiv:2003.03164 [cs.CV] (2020)

  45. Bai, X., et al.: PointDSC: robust point cloud registration using deep spatial consistency. In: CVPR (2021)

    Google Scholar 

  46. Yan, X., et al.: 2DPASS: 2D priors assisted semantic segmentation on LiDAR Point Clouds. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision, ECCV 2022. LNCS, vol. 13688, pp. 677–695. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19815-1_39

  47. Yang, H., Shi, J., Carlone, L.: TEASER: fast and certifiable point cloud registration. IEEE Trans. Robot. 37, 314–333 (2020)

    Article  Google Scholar 

  48. Yang, J., Zhang, X., Fan, S., Ren, C., Zhang, Y.: Mutual voting for ranking 3D correspondences. IEEE Trans. Pattern Anal. Mach. Intel. 46, 4041–4057 (2023)

    Article  Google Scholar 

  49. Yang, Y., Feng, C., Shen, Y., Tian, D.: FoldingNet: point cloud auto-encoder via deep grid deformation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 206–215 (2018)

    Google Scholar 

  50. Yew, Z.J., Lee, G.H.: REGTR: end-to-end point cloud correspondences with transformers. In: CVPR (2022)

    Google Scholar 

  51. Yin, P., Yuan, S., Cao, H., Ji, X., Zhang, S., Xie, L.: Segregator: global point cloud registration with semantic and geometric cues. arXiv preprint arXiv:2301.07425 (2023)

  52. Yu, H., Li, F., Saleh, M., Busam, B., Ilic, S.: CoFiNet: reliable coarse-to-fine correspondences for robust pointcloud registration. In: Advances in Neural Information Processing Systems, vol. 34, pp. 23872–23884 (2021)

    Google Scholar 

  53. Yu, H., Qin, Z., Hou, J., Saleh, M., Li, D., Busam, B., Ilic, S.: Rotation-invariant transformer for point cloud matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5384–5393 (2023)

    Google Scholar 

  54. Zaganidis, A., Sun, L., Duckett, T., Cielniak, G.: Integrating deep semantic segmentation into 3-D point cloud registration. IEEE Robot. Autom. Lett. 3(4), 2942–2949 (2018)

    Article  Google Scholar 

  55. Zaharescu, A., Boyer, E., Varanasi, K., Horaud, R.: Surface feature detection and description with applications to mesh matching. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 373–380. IEEE (2009)

    Google Scholar 

  56. Zeng, A., Song, S., Nießner, M., Fisher, M., Xiao, J., Funkhouser, T.: 3DMatch: learning local geometric descriptors from RGB-D reconstructions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1802–1811 (2017)

    Google Scholar 

  57. Zhang, X., Yang, J., Zhang, S., Zhang, Y.: 3d registration with maximal cliques. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17745–17754 (2023)

    Google Scholar 

  58. Zhijian Qiao, Zehuan Yu, H.Y., Shen, S.: Pyramid semantic graph-based global point cloud registration with low overlap. In: 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2023)

    Google Scholar 

  59. Zhou, Q.Y., Park, J., Koltun, V.: Open3D: a modern library for 3D data processing. arXiv arXiv:1801.09847 (2018)

  60. Zhu, X., et al.: Cylindrical and asymmetrical 3D convolution networks for LiDAR segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 9939–9948 (2021)

    Google Scholar 

Download references

Acknowledgements

This work was supported by the Wuhan university-Huawei Geoinformatics Innovation Laboratory Open Fund under Grant TC20210901025-2023-06.

Author information

Authors and Affiliations

  1. School of Remote Sensing and Information Engineering, Wuhan University, Wuhan, China

    Shaocheng Yan & Jiayuan Li

  2. School of Computer Science, Wuhan University, Wuhan, China

    Pengcheng Shi

Authors
  1. Shaocheng Yan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Pengcheng Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiayuan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jiayuan Li .

Editor information

Editors and Affiliations

  1. University of Birmingham, Birmingham, UK

    Aleš Leonardis

  2. University of Trento, Trento, Italy

    Elisa Ricci

  3. Technical University of Darmstadt, Darmstadt, Hessen, Germany

    Stefan Roth

  4. Princeton University, Palo Alto, CA, USA

    Olga Russakovsky

  5. Czech Technical University in Prague, Prague, Czech Republic

    Torsten Sattler

  6. École des Ponts ParisTech, Marne-la-Vallée, France

    Gül Varol

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 2180 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yan, S., Shi, P., Li, J. (2025). ML-SemReg: Boosting Point Cloud Registration with Multi-level Semantic Consistency. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15102. Springer, Cham. https://doi.org/10.1007/978-3-031-72784-9_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-72784-9_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-72783-2

  • Online ISBN: 978-3-031-72784-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation