Skip to main content
SpringerLink
Log in

Real-Time 3D Reconstruction for Mixed Reality Telepresence Using Multiple Depth Sensors

  • Conference paper
  • First Online:
Advanced Communication and Intelligent Systems (ICACIS 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1749))

Abstract

In recent years, there has been a great development in real-time three-dimensional (3D) scene reconstruction from depth sensor data, as well as the study of such data in Virtual Reality (VR) and Augmented Reality (AR) contexts. Although it has been extensively investigated and has attracted the attention of many researchers, the challenge of real-time 3D reconstruction remains a difficult research task. The majority of current techniques, target real-time 3D reconstruction for the single-view-based system rather than multi-view. In order to provide multi-view 3D reconstruction for Mixed Reality (MR) telepresence, this chapter aims to propose a multiple depth sensor capture using a marching square approach to produce a single full 3D reconstruction surface of a moving user in real-time. The chapter explains the design stage that involves setup from multiple depth sensors, surface reconstruction and merging of 3D reconstruction data for MR Telepresence. The chapter ends with results and a conclusion.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 99.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 129.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Itani, O.S., Hollebeek, L.D.: Light at the end of the tunnel: visitors’ virtual reality (versus in-person) attraction site tour-related behavioral intentions during and post-COVID-19. Tour. Manag. 84, 104290 (2021). https://doi.org/10.1016/J.TOURMAN.2021.104290

    Article  Google Scholar 

  2. Luevano, L., Lopez de Lara, E., Quintero, H.: Professor avatar holographic telepresence model. Hologr. Mater. Appl. (2019). https://doi.org/10.5772/INTECHOPEN.85528

  3. Zahorik, P., Jenison, R.L.: Presence as being-in-the-world. Presence Teleop. Virtual Environ. 7, 78–89 (1998). https://doi.org/10.1162/105474698565541

    Article  Google Scholar 

  4. Sakashita, M., Minagawa, T., Koike, A., Suzuki, I., Kawahara, K., Ochiai, Y.: You as a puppet: evaluation of telepresence user interface for puppetry. In: UIST 2017 - Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 217–228 (2017). https://doi.org/10.1145/3126594.3126608

  5. Stotko, P., Krumpen, S., Hullin, M.B., Weinmann, M., Klein, R.: SLAMCast: large-scale, real-time 3D reconstruction and streaming for immersive multi-client live telepresence. IEEE Trans. Vis. Comput. Graph. 25, 2102–2112 (2019). https://doi.org/10.1109/TVCG.2019.2899231

    Article  Google Scholar 

  6. Pejsa, T., Kantor, J., Benko, H., Ofek, E., Wilson, A.: Room2Room: enabling life-size telepresence in a projected augmented reality environment. In: Proceedings of ACM Conference on Computer Supported Cooperative Work, CSCW, vol. 27, pp. 1716–1725 (2016). https://doi.org/10.1145/2818048.2819965

  7. Ingale, A.K.: Real-time 3D reconstruction techniques applied in dynamic scenes: a systematic literature review. Comput. Sci. Rev. 39, 100338 (2021). https://doi.org/10.1016/J.COSREV.2020.100338

  8. Cai, H., Feng, W., Feng, X., Wang, Y., Zhang, J.: Neural Surface Reconstruction of Dynamic Scenes with Monocular RGB-D Camera (2022)

    Google Scholar 

  9. Teo, T., Hayati, A.F., Lee, G.A., Billinghurst, M., Adcock, M.: A technique for mixed reality remote collaboration using 360 panoramas in 3D reconstructed scenes. In: Proceedings of ACM Symposium on Virtual Reality Software and Technology, VRST, vol. 11 (2019). https://doi.org/10.1145/3359996.3364238

  10. Joachimczak, M., Liu, J.: Real-time mixed-reality telepresence via 3D reconstruction with HoloLens and commodity depth sensors (2017). https://doi.org/10.1145/3136755.3143031

  11. John, B., Wickramasinghe, N.: A review of mixed reality in health care. In: Wickramasinghe, N., Bodendorf, F. (eds.) Delivering Superior Health and Wellness Management with IoT and Analytics. HDIA, pp. 375–382. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-17347-0_18

    Chapter  Google Scholar 

  12. Harborth, D.: A systematic literature review on augmented reality augmented reality in information systems research: a systematic literature review (2017)

    Google Scholar 

  13. Sutherland, I.E.: A head-mounted three dimensional display (1968)

    Google Scholar 

  14. Azuma, R.T.: A survey of augmented reality. Presence Teleop. Virtual Environ. 6, 355–385 (1997)

    Article  Google Scholar 

  15. Li, J., Huang, S., Cui, H., Ma, Y., Chen, X.: Automatic point cloud registration for large outdoor scenes using a priori semantic information. Remote Sens. 13, 3474–3474 (2021). https://doi.org/10.3390/RS13173474

  16. Jia, Q., et al.: Real-time 3D reconstruction method based on monocular vision. Sensors 21, 5909–5909 (2021). https://doi.org/10.3390/S21175909

  17. Nor’a, M.N.A., Fadzli, F.E., Ismail, A.W.: A review on real-time 3D reconstruction methods in dynamic scene. Int. J. Innov. Comput. 12, 91–97 (2022). https://doi.org/10.11113/IJIC.V12N1.317

  18. Clark, R.A., Mentiplay, B.F., Hough, E., Pua, Y.H.: Three-dimensional cameras and skeleton pose tracking for physical function assessment: a review of uses, validity, current developments and Kinect alternatives. Gait Posture. 68, 193–200 (2019). https://doi.org/10.1016/J.GAITPOST.2018.11.029

    Article  Google Scholar 

  19. Meruvia-Pastor, O.: Enhancing 3D capture with multiple depth camera systems: a state-of-the-art report. In: Rosin, P.L., Lai, Y.-K., Shao, L., Liu, Y. (eds.) RGB-D Image Analysis and Processing. ACVPR, pp. 145–166. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28603-3_7

    Chapter  Google Scholar 

  20. Mühlenbrock, A., Fischer, R., Schröder-Dering, C., Weller, R., Zachmann, G.: Fast, accurate and robust registration of multiple depth sensors without need for RGB and IR images. Vis. Comput. 38, 3995–4008 (2022). https://doi.org/10.1007/s00371-022-02505-2

    Article  Google Scholar 

  21. Muhlenbrock, A., Fischer, R., Weller, R., Zachmann, G.: Fast and robust registration of multiple depth-sensors and virtual worlds. In: Proceedings of 2021 International Conference on Cyberworlds, CW 2021, pp. 41–48 (2021). https://doi.org/10.1109/CW52790.2021.00014

  22. Espinosa, R., Ponce, H., Gutiérrez, S., Martínez-Villaseñor, L., Brieva, J., Moya-Albor, E.: Application of convolutional neural networks for fall detection using multiple cameras. Stud. Syst. Decis. Control 273, 97–120 (2020). https://doi.org/10.1007/978-3-030-38748-8_5/COVER

    Article  Google Scholar 

  23. Li, J., Gao, W., Wu, Y., Liu, Y., Shen, Y.: High-quality indoor scene 3D reconstruction with RGB-D cameras: a brief review. Comput. Vis. Media 83(8), 369–393 (2022)

    Article  Google Scholar 

  24. Lin, W., Zheng, C., Yong, J.-H., Xu, F.: OcclusionFusion: occlusion-aware motion estimation for real-time dynamic 3D reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1736–1745 (2022)

    Google Scholar 

  25. Sun, J., Chen, X., Wang, Q., Li, Z., Averbuch-Elor, H., Zhou, X., Snavely, N.: Neural 3D reconstruction in the wild. In: Special Interest Group on Computer Graphics and Interactive Techniques Conference Proceedings, pp. 1–9 (2022)

    Google Scholar 

  26. Fadzli, F.E., Ismail, A.W.: A robust real-time 3D reconstruction method for mixed reality telepresence. Int. J. Innov. Comput. 10, 15–20 (2020). https://doi.org/10.11113/IJIC.V10N2.265

  27. Mandal, S., et al.: Lyft 3D object detection for autonomous vehicles. In: Artificial Intelligence for Future Generation Robotics, pp. 119–136 (2021). https://doi.org/10.1016/B978-0-323-85498-6.00003-4

  28. Papaefthymiou, M., Kanakis, M.E., Geronikolakis, E., Nochos, A., Zikas, P., Papagiannakis, G.: Rapid reconstruction and simulation of real characters in mixed reality environments. In: Ioannides, M. (ed.) Digital Cultural Heritage. Lecture Notes in Computer Science, vol. 10605, pp. 267–276. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75826-8_22

    Chapter  Google Scholar 

  29. Das, S., et al.: Advance machine learning and artificial intelligence applications in service robot. In: Artificial Intelligence for Future Generation Robotics, pp. 83–91 (2021). https://doi.org/10.1016/B978-0-323-85498-6.00002-2

  30. Moon, S., Park, Y., Ko, D.W., Suh, I.H.: Multiple kinect sensor fusion for human skeleton tracking using Kalman filtering. 13 (2017). https://doi.org/10.5772/62415

  31. Mukhopadhyay, M., et al.: Facial emotion recognition based on textural pattern and convolutional neural network. In: 2021 IEEE 4th International Conference on Computing, Power and Communication Technologies (GUCON), pp. 1–6 (2021). https://doi.org/10.1109/GUCON50781.2021.9573860

  32. Mocanu, C., Mocanu, I.: Human body posture recognition using a kinect sensor (2015)

    Google Scholar 

  33. Fadzli, F.E., Ismail, A.W., Ishigaki, S.A.K., Nor’a, M.N.A., Aladin, M.Y.F.: Real-time 3D reconstruction method for holographic telepresence. Appl. Sci. 12, 4009–4009 (2022). https://doi.org/10.3390/APP12084009

  34. Khatamian, A., Arabnia, H.R.: Survey on 3D surface reconstruction (2016). https://doi.org/10.3745/JIPS.01.0010

  35. Jang, G.R., et al.: Real-time polygon generation and texture mapping for tele-operation using 3D point cloud data. J. Inst. Control Robot. Syst. 19, 928–935 (2013). https://doi.org/10.5302/J.ICROS.2013.13.8012

    Article  Google Scholar 

  36. Brandão, S., Costeira, J., Costeira, J.P., Veloso, M.: Effortless Scanning of 3D Object Models by Boundary Aligning and Stitching mobile phone dataset analysis View project Point matching View project Effortless Scanning of 3D Object Models by Boundary Aligning and Stitching (2014)

    Google Scholar 

  37. Naik, R., Singh, P., Kalra, P.: Putting jewellery and accessories on a 3D face model generated from 2D image. In: Babu, R.V., Prasanna, M., Namboodiri, V.P. (eds.) NCVPRIPG 2019. CCIS, vol. 1249, pp. 229–238. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-8697-2_21

    Chapter  Google Scholar 

Download references

Acknowledgement

We deeply appreciate the Mixed and Virtual Reality Laboratory (mivielab), and ViCubeLab at Universiti Teknologi Malaysia (UTM) for the equipment and technical assistance. This work has been funded by the Ministry of Higher Education under FRGS, Registration Proposal No: FRGS/1/2020/ICT10/UTM/02/1.

Author information

Authors and Affiliations

  1. Mixed and Virtual Reality Research Lab, ViCubeLab, School of Computing, Universiti Teknologi Malaysia, 81310, Johor, Malaysia

    Shafina Abd Karim Ishigaki & Ajune Wanis Ismail

Authors
  1. Shafina Abd Karim Ishigaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ajune Wanis Ismail
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Shafina Abd Karim Ishigaki .

Editor information

Editors and Affiliations

  1. Bharath Institute of Higher Education and Research, Chennai, India

    Rabindra Nath Shaw

  2. Systems Research Institute, Warsaw, Poland

    Marcin Paprzycki

  3. The Neotia University, Sarisha, India

    Ankush Ghosh

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ishigaki, S.A.K., Ismail, A.W. (2023). Real-Time 3D Reconstruction for Mixed Reality Telepresence Using Multiple Depth Sensors. In: Shaw, R.N., Paprzycki, M., Ghosh, A. (eds) Advanced Communication and Intelligent Systems. ICACIS 2022. Communications in Computer and Information Science, vol 1749. Springer, Cham. https://doi.org/10.1007/978-3-031-25088-0_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25088-0_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25087-3

  • Online ISBN: 978-3-031-25088-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation