Skip to main content
Springer Nature Link
Log in

Bidirectional Point Cloud Holes Repair Obtained by SfM and Structured Light

  • Original Research
  • Published:
SN Computer Science Aims and scope Submit manuscript

Abstract

The structure from motion (SfM) algorithm is widely used for point cloud reconstruction. However, one drawback of conventional SfM based methods is that the obtained final point sets may contain holes and noise with rich color information. At the same time, the point cloud collected based on coded structured light is dense and smooth, but lack of color information. To get better point cloud information by combining structured light point cloud and SFM point cloud, this paper proposes a bidirectional point cloud complementation for three-dimensional(3D) holes repairing combining coded structured light and structure from motion (SfM). Firstly, the point feature histogram which is used to automatically extract the point cloud features to determine the initial control point set is constructed according to the surface characteristics. Then, the principal components analysis (PCA) method is used for rough registration, and then the improved iterative nearest point algorithm is used for accurate registration. Secondly, the nearest point search algorithm is used to fuse the color information distributed in the Point cloud obtained by SfM data with the point cloud coordinates of the coded structured light. Finally, obtained by the reverse engineering software Geomagic Studio, the repaired point cloud model is subdivided into triangular meshes to reconstruct the 3D face surface. Experimental results and data analysis show that this method can accurately perform two-way repair fusion on the encoded point cloud obtained by structured light and point cloud obtained by SfM, and realize the hole repair of the face model with the complete surface reconstruction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

References

  1. Mao S, Liu Z, Lian X, Xu J, Zhao W. A three-dimensional reconstruction system obtained by panoramic vision and two-dimensional ranging laser. In: International Conference on Electrical and Control Engineering, Yichang, 3796–3799, 2011.

  2. Salvi J, Pages J, Batlle J. Pattern codification strategies in structured light systems. Pattern Recogn. 2004;37(4):827–49.

    Article  MATH  Google Scholar 

  3. Fonstad MA, Dietrich JT, Courville BC, et al. Topographic structure from motion: a new development in photogrammetric measurement. Earth Surface Processes Landforms. 2013;38(4):421–30.

    Article  Google Scholar 

  4. Carr JC, Beatsin RK, Cherrie JB, et al. Reconstruction and representation of 3D objects with radial basis functions. In: Proceedings of the 28th Annual conference on Computer Graphics and Interactive Techniques, 67–76, 2001.

  5. Liang R, Lin Y, Li J, et al. Craniofacial model reconstruction from skull data obtained by feature points. In: 11th IEEE International Conference on Computer-Aided Design and Computer Graphics, Huangshan, 602–605, 2009.

  6. Liepa P. Filling holes in meshes. In: Proceedings of the Euro graphics/ACM SIGGRAPH Symposium on Geometry Processing, 200–205, 2003.

  7. Brunton A, Wuhrer S, Shu C, et al. Filling holes in triangular meshes by curve un folding. In: IEEE International Conference on Shape Modeling and Applications, 66–72, 2009.

  8. Leong KF, Chua CK, Ng YM. A study of stereo lithography file errors and repair. Part1: Generic solution. Int J Adv Manuf Technol. 1996;12:407–14.

    Article  Google Scholar 

  9. Lin H, Chen J, Zheng Y, et al. Feature preserving filling of holes on point sampled surfaces based on tensor voting. Math Problems Eng. 2018;8:1–12.

    Google Scholar 

  10. Gai S, Da FP, Zeng LL, et al. Research on a hole filling algorithm of a point cloud based on structure from motion. J Opt Soc Am. 2019;36(2):39–46.

    Article  Google Scholar 

  11. Jeong Y, Bok Y, Kim JS, et al. Complementation of cameras and lasers for accureta 6D slam: From correspondences to bundle adjustment. In: IEEE International Conference on Robotics and Automation, 3581–3588, 2011.

  12. Zhou H, Zhang T. The combination of SfM and monocular SLAM. In: The 26th Chinese Control and Decision Conference, 5282–5286, 2014.

  13. Peng F, et al. Street view cross-sourced point cloud matching and registration. In: IEEE International Conference on Image Processing, 2026–2030, 2014.

  14. Yuffa AJ, Gurton KP, Videen G. Three-dimensional facial recognition using passive long-wavelength infrared polarimentric imaging. Appl Optics. 2014;53(36):8514–21.

    Article  Google Scholar 

  15. Jo HG, Cho HM, Lee S, et al. Multi-resolution point cloud generation based on heterogeneous sensor fusion system. In: International Conference on Ubiquitous Robots & Ambient Intelligence, 886–888, 2017.

  16. Zhang YF, Yang BH, Li JP, et al. An accurate TLS and UAV image point clouds registration method for deformation detection of chaotic hillside areas. Remote Sens. 2019;11(6):647–83.

    Article  Google Scholar 

  17. Zhang L, Zhang X, Song J, et al. Viewpoint calibration method based on point features for point cloud fusion. In: IEEE International Conference on Image Processing (ICIP), 2224–2228, 2019.

  18. Li SM, Ge XM, Hu HA, et al. Laplacian fusion approach of multi-source point clouds for detail enhancement. ISPRS J Photogram Remote Sens. 2021;171:385–96.

    Article  Google Scholar 

  19. Rusu RB, Marton ZC, Blodow N, et al. Persistent point feature histograms for 3D point cloud. In: Proc 10th Int Conf. Intel Autonomous Syst, 119–128, 2008.

  20. Rusu RB, Marton ZC, Blodow N, et al. Learning informative point classes for the acquisition of object model maps. In: 10th International Conference on Control, Automation, Robotics and Vision, 643–650, 2008.

  21. Rusu RB, Blodow N, Marton ZC, et al. Aligning point cloud views using persistent feature histograms. In: International Conference on Intelligent Robots and Systems, 3384–3391, 2008.

  22. Rusu RB, Blodow N, Beetz M. Fast Point Feature Histograms (FPFH) for 3D registration. In: IEEEInternational Conference on Robotics and Automation, Kobe, 3212–3217, 2009.

  23. Chung DH, Yun ID, Sang UL. Registration of multiple-range view using the reverse-calibration technique. Pattern Recogn. 1988;31(4):457–64.

    Article  Google Scholar 

  24. Schenker PS. Method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell. 1992;14(2):239–56.

    Article  Google Scholar 

  25. Székely GHB. Selected readings in vision and graphics, Vol. 23: Mark Pauly, point primitives for interactive modeling and processing of 3D geometry, 2003.

  26. Han H, Han X, Sun F, et al. Point cloud simplification with preserved edge obtained by normal vector. Optik - Int J Light Electron Opt. 2015;126(19):2157–62.

    Article  Google Scholar 

  27. Ju Q, Wang B. Hierarchical 3D face alignment obtained by region segmentation. J Convergence Info Technol. 2012;7(10):231–8.

    Article  Google Scholar 

  28. Zeng LL, Gau SY, Huang Y. Hole filling algorithm of three-dimensional point cloud based on structure from motion. Acta Optica Sinica. 2018;36:131–7.

    Google Scholar 

  29. Yin PY, Wang D, Du SY, et al. CoBigICP: Robust and Precise Point Set Registration using Correntropy Metrics and Bidirectional Correspondence. In: International Conference on Intelligent Robots and Systems (IROS), 4692–4699, 2020.

  30. Le HM, Do T, Hoang T, Cheung N. SDRSAC: Semidefinite-Based Randomized Approach for RobustPoint Cloud Registration Without Correspondences. In: Conference on Computer Vision and Pattern Recognition (CVPR), 124-133, 2019.

Download references

Acknowledgements

The authors would like to acknowledge the anonymous reviewers for their valuable comments.

Funding

This research was supported in part by the Natural Science Foundation of Shanghai (Grant No. 20ZR1421300), in part by the Shanghai Pujiang Talents Program (Grant No. 21PJD025), and in part by the Shanghai Science and Technology Commission Program (Grant No.21DZ1207300).

Author information

Authors and Affiliations

  1. School of Automation Engineering, Shanghai University of Electric Power, Shanghai, 200094, China

    Hui Chen, Fangyong Xu & Yan Feng

  2. Department of Electrical Engineering, Mirpur University of Science and Technology, MUST, Mirpur, Pakistan

    Muhammad Ilyas Menhas

  3. College of Electrical Engineering, Qingdao University, Qingdao, 266071, China

    Zhonghua Hao

Authors
  1. Hui Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Fangyong Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yan Feng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Ilyas Menhas
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhonghua Hao
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Conceptualization, H. Chen and F. Xu; metho- dology, H. Chen; software, Y. Feng and F. Xu; validation, F. Xu and Y. Feng; formal analysis, M.I. Menhas; investigation, Z. Hao; resources, H. Chen; data curation, F. Xu and Y. Feng; writing–original draft preparation, F. Xu and Y. Feng; writing–review and editing, H. Chen and F. Xu; funding acquisition, H. Chen. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Zhonghua Hao.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, H., Xu, F., Feng, Y. et al. Bidirectional Point Cloud Holes Repair Obtained by SfM and Structured Light. SN COMPUT. SCI. 3, 457 (2022). https://doi.org/10.1007/s42979-022-01301-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-022-01301-y

Keywords