Skip to main content
SpringerLink
Log in

Registration and occlusion handling based on the FAST ICP-ORB method for augmented reality systems

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Spatial position consistency and occlusion consistency are two important problems in augmented reality systems. In this paper, we proposed a novel method that can address the registration problem and occlusion problem simultaneously by using an RGB-D camera. First, to solve the image alignment errors caused by the imaging mode of the RGB-D camera, we developed a depth map inpainting method that combines the FMM and RGB-D information. Second, we established an automatic method to judge the close-range mode based on the depth histogram to solve the registration failure problem caused by hardware limitations. In the close-range mode, the registration method combining the fast ICP and ORB was adopted to calculate the camera pose. Third, we developed an occlusion handling method based on the geometric analysis of the scene. Several experiments were performed to validate the performance of the proposed method. The experimental results indicate that our method can obtain stable and accurate registration and occlusion handling results in both the close-range and non-close-range modes. Moreover, the mutual occlusion problem can be handled effectively, and the proposed method can satisfy the real-time requirements of augmented reality systems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15

Similar content being viewed by others

References

  1. Alhaija HA, Mustikovela SK, Mescheder L et al (2018) Augmented reality meets computer vision. Efficient data generation for urban driving scenes. Int J Comput Vision 126(9):961–972

    Article  Google Scholar 

  2. Besl PJ, Mckay ND (1992) Method for registration of 3-D shapes. IEEE Trans Pattern Anal Mach Intell 14(2):239–256

    Article  Google Scholar 

  3. Du C, Chen YL, Ye M et al (2016) Edge snapping-based depth enhancement for dynamic occlusion handling in augmented reality. In: Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality, pp 54–62

  4. Feng Q, Shum H, Morishima S (2018) Resolving occlusion for 3D object manipulation with hands in mixed reality. In: Proceedings of ACM Symposium on Virtual Reality Software and Technology, pp 119–120

  5. Guan T, Wang Y, Duan LY (2015) On-device mobile landmark recognition using binarized descriptor with multifeature fusion. ACM Trans Intell Syst Technol 7(1):1–29

    Article  Google Scholar 

  6. Hebborn AK, Höhner N, Müller S (2017) Occlusion matting: realistic occlusion handling for augmented reality applications. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality, pp 62–71

  7. Jesus AL, Leopoldo AR, Jesus AG (2013) Occlusion handling in video-based augmented reality using the Kinect sensor for indoor registration. In: Proceedings of Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp 447–454

  8. Kasperi J, Edwardsson MP, Romero M (2017) Occlusion in outdoor Augmented Reality using geospatial building data. In: Proceedings of ACM Symposium on Virtual Reality Software and Technology, pp 30–39

  9. Klein G, Murray D (2007) Parallel tracking and mapping for small AR workspaces. In: Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality, pp 1–10

  10. Lieberknecht S, Huber A, Ilic S et al (2011) RGB-D camera-based parallel tracking and meshing. In: Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality, pp 147–155

  11. Markman A, Shen X, Hua H et al (2016) Augmented reality three-dimensional object visualization and recognition with axially distributed sensing. Opt Lett 41(2):297–300

    Article  Google Scholar 

  12. Newcombe RA, Izadi S, Hilliges O et al (2011) Kinect fusion real-time dense surface mapping and tracking. In: Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality, pp 127–136

  13. Ong KC, The HC, Tan TS (1998) Resolving occlusion in image sequence made easy. Vis Comput 14(4):153–165

    Article  Google Scholar 

  14. Tan W, Liu H, Dong Z et al (2013) Robust monocular SLAM in dynamic environments. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality, pp 209–218

  15. Tian Y, Guan T, Wang C (2010) An automatic occlusion handling method in augmented reality. Sensor Review 30(3):210–218

    Article  Google Scholar 

  16. Tian Y, Long Y, Xia D et al (2015) Handling occlusions in augmented reality based on 3D reconstruction method. Neurocomputing 156:96–104

    Article  Google Scholar 

  17. Tian Y, Wang XF, Yao H et al (2018) Occlusion handling using moving volume and ray casting techniques for augmented reality systems. Multimed Tools Appl 77(13):16561–16578

    Article  Google Scholar 

  18. Wei BC, Guan T, Duan LY et al (2015) Wide area localization and tracking on camera phones for mobile augmented reality systems. Multimedia Syst 21(4):381–399

    Article  Google Scholar 

  19. Xu C, Li S, Wang J et al (2008) Occlusion handling in augmented reality system for human-assisted assembly task. Intell Robot Appl 8:121–130

    Google Scholar 

  20. Yilmaz O, Karakus F (2013) Stereo and Kinect fusion for continuous 3D reconstruction and visual odometry. In: Proceedings of International Conference on Electronics, Computer and Computation, pp 115–118

  21. Yun K, Lu T, Chow E (2018) Occluded object reconstruction for first responders with augmented reality glasses using conditional generative adversarial networks. Pattern Recognit Tracking 28:10649–10654

    Google Scholar 

Download references

Acknowledgements

This work is financially supported by the Humanities and Social Sciences Foundation of the Ministry of Education (No: 19YJC880079) and self-determined research funds of CCNU from the colleges’ basic research and operation of MOE (No. CCNU20ZN002).

Author information

Authors and Affiliations

  1. School of Educational Information Technology, Central China Normal University, Wuhan, 430079, China

    Yuan Tian, Xiaolei Zhou, Xuefan Wang, Zhifeng Wang & Huang Yao

Authors
  1. Yuan Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaolei Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuefan Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhifeng Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Huang Yao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yuan Tian.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tian, Y., Zhou, X., Wang, X. et al. Registration and occlusion handling based on the FAST ICP-ORB method for augmented reality systems. Multimed Tools Appl 80, 21041–21058 (2021). https://doi.org/10.1007/s11042-020-10342-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-020-10342-5

Keywords

Search

Navigation