Skip to main content
Springer Nature Link
Log in

Evaluation of Point Cloud Streaming and Rendering for VR-Based Telepresence in the OR

  • Conference paper
  • First Online:
Virtual Reality and Mixed Reality (EuroXR 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13484))

Included in the following conference series:

Abstract

Immersive and high-quality VR-based telepresence systems could be of great benefit in the medical field and the operating room (OR) specifically, as they allow distant experts to interact with each other and to assist local doctors as if they were physically present. Despite recent advances in VR technology, and more telepresence systems making use of it, most of the current solutions in use in health care (if any), are just video-based and don’t provide the feeling of presence or spatial awareness, which are highly important for tasks such as remote consultation, -supervision, and -teaching. Reasons still holding back VR telepresence systems are high demands regarding bandwidth and computational power, subpar visualization quality, and complicated setups. We propose an easy-to-set-up telepresence system that enables remote experts to meet in a multi-user virtual operating room, view live-streamed and 3D-visualized operations, interact with each other, and collaboratively explore medical data. Our system is based on Azure Kinect RGB-D cameras, a point cloud streaming pipeline, and fast point cloud rendering methods integrated into a state-of-the-art 3D game engine. Remote experts are visualized via personalized real-time 3D point cloud avatars. For this, we have developed a high-speed/low-latency multi-camera point cloud streaming pipeline including efficient filtering and compression. Furthermore, we have developed splatting-based and mesh-based point cloud rendering solutions and integrated them into the Unreal Engine 4. We conducted two user studies with doctors and medical students to evaluate our proposed system, compare the rendering solutions, and highlight our system’s capabilities.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.unrealengine.com/marketplace/en-US/product/lidar-point-cloud.

  2. 2.

    http://www.igroup.org/pq/ipq/download.php.

References

  1. Amamra, A., Aouf, N.: GPU-based real-time RGBD data filtering. J. Real-Time Image Process. 14(2), 323–340 (2014). https://doi.org/10.1007/s11554-014-0453-7

    Article  Google Scholar 

  2. Anton, D., Kurillo, G., Yang, A.Y., Bajcsy, R.: Augmented telemedicine platform for real-time remote medical consultation. In: Amsaleg, L., Guðmundsson, G.Þ, Gurrin, C., Jónsson, B.Þ, Satoh, S. (eds.) MMM 2017. LNCS, vol. 10132, pp. 77–89. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-51811-4_7

    Chapter  Google Scholar 

  3. Augestad, K., Lindsetmo, R.O.: Overcoming distance: video-conferencing as a clinical and educational tool among surgeons. World J. Surg. 33, 1356–1365 (2009)

    Article  Google Scholar 

  4. Baños, R., Botella, C., Alcañiz Raya, M., Liaño, V., Guerrero, B., Rey, B.: Immersion and emotion: their impact on the sense of presence. Cyberpsychology Behav. Impact Internet Mult. Virtual Reality Behav. Soc. 7, 734–41 (2005)

    Google Scholar 

  5. Böhlen, C.F.v., Brinkmann, A., Mävers, S., Hellmers, S., Hein, A.: Virtual reality integrated multi-depth-camera-system for real-time telepresence and telemanipulation in caregiving. In: 2020 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 294–297 (2020)

    Google Scholar 

  6. Cao, C., Preda, M., Zaharia, T.: 3d point cloud compression: a survey, pp. 1–9, July 2019

    Google Scholar 

  7. Cho, S., Kim, S.W., Lee, J., Ahn, J., Han, J.: Effects of volumetric capture avatars on social presence in immersive virtual environments. In: 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 26–34 (2020)

    Google Scholar 

  8. Dedeilia, A., Sotiropoulos, M., Hanrahan, J., Janga, D., Dedeilias, P., Sideris, M.: Medical and surgical education challenges and innovations in the covid-19 era: a systematic review. Vivo 34, 1603–1611 (2020)

    Article  Google Scholar 

  9. Dou, M., et al.: Fusion4d: real-time performance capture of challenging scenes. ACM Trans. Graph. 35(4), 1–13 (2016)

    Article  MathSciNet  Google Scholar 

  10. Dussault, G., Franceschini, M.: Not enough there, too many here: understanding geographical imbalances in the distribution of the health workforce. Hum. Resour. Health 4, 12 (2006)

    Article  Google Scholar 

  11. Flodgren, G., Rachas, A., Farmer, A., Inzitari, M., Shepperd, S.: Interactive telemedicine: effects on professional practice and health care outcomes. Cochrane Database Syst. Rev. 9, CD002098 (2015)

    Google Scholar 

  12. Essmaeel, K., Gallo, L., Damiani, E., De Pietro, G., Dipanda, A.: Comparative evaluation of methods for filtering Kinect depth data. Multimedia Tools Appl. 74(17), 7331–7354 (2014). https://doi.org/10.1007/s11042-014-1982-6

    Article  Google Scholar 

  13. Gamelin, G., Chellali, A., Cheikh, S., Ricca, A., Dumas, C., Otmane, S.: Point-cloud avatars to improve spatial communication in immersive collaborative virtual environments. Personal Ubiquitous Comput. 25(3), 467–484 (2020). https://doi.org/10.1007/s00779-020-01431-1

    Article  Google Scholar 

  14. Gasques, D., et al.: Artemis: a collaborative mixed-reality system for immersive surgical telementoring. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI 2021, Association for Computing Machinery, New York (2021)

    Google Scholar 

  15. Huber, P., et al.: A multiresolution 3d morphable face model and fitting framework. In: Proceedings of the 11th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications, University of Surrey (2016)

    Google Scholar 

  16. Jang-Jaccard, J., Nepal, S., Celler, B., Yan, B.: Webrtc-based video conferencing service for telehealth. Computing 98, 169–193 (2016)

    Article  MathSciNet  Google Scholar 

  17. Kamimura, K., Fujita, Y., Miura, T., Matsumoto, Y., Maeda, Y., Zempo, K.: Teleclinical support system via MR-HMD displaying doctor’s instructions and patient information, pp. 477–479, March 2021

    Google Scholar 

  18. Kolkmeier, J., Harmsen, E., Giesselink, S., Reidsma, D., Theune, M., Heylen, D.: With a little help from a holographic friend: the open impress mixed reality telepresence toolkit for remote collaboration systems, pp. 1–11, November 2018

    Google Scholar 

  19. Kvedar, J., Coye, M., Everett, W.: Connected health: a review of technologies and strategies to improve patient care with telemedicine and telehealth. Health affairs (Project Hope) 33, 194–199 (2014)

    Google Scholar 

  20. Latifi, R., et al.: Telemedicine and telepresence for trauma and emergency management. Scand. J. Surg. 96, 281–289 (2007)

    Article  Google Scholar 

  21. Lin, B.S., Su, M.J., Cheng, P.H., Tseng, P.J., Chen, S.J.: Temporal and spatial denoising of depth maps. Sensors 15, 18506–18525 (2015)

    Article  Google Scholar 

  22. Liu, Y., et al.: Hybrid lossless-lossy compression for real-time depth-sensor streams in 3D telepresence applications. In: Ho, Y.-S., Sang, J., Ro, Y.M., Kim, J., Wu, F. (eds.) PCM 2015. LNCS, vol. 9314, pp. 442–452. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24075-6_43

    Chapter  Google Scholar 

  23. Mao, A., Zhang, H., Liu, Y., Zheng, Y., Li, G., Han, G.: Easy and fast reconstruction of a 3d avatar with an RGB-d sensor. Sensors 17(5), 1113 (2017)

    Article  Google Scholar 

  24. Mekuria, R., Blom, K., César, P.: Design, implementation and evaluation of a point cloud codec for tele-immersive video. IEEE Trans. Circ. Syst. Video Technol. 27, 1 (2016)

    Google Scholar 

  25. Mühlenbrock, A., Fischer, R., Weller, R., Zachmann, G.: Fast and robust registration of multiple depth-sensors and virtual worlds. In: 2021 International Conference on Cyberworlds (CW), pp. 41–48 (2021)

    Google Scholar 

  26. Nowak, K., Biocca, F.: The effect of the agency and anthropomorphism on users’ sense of telepresence, copresence, and social presence in virtual environments. Presence Teleoperators Virtual Environ. 12, 481–494 (2003)

    Google Scholar 

  27. Orts, S., et al.: Holoportation: virtual 3d teleportation in real-time, December 2016

    Google Scholar 

  28. Pece, F., Kautz, J., Weyrich, T.: Adapting standard video codecs for depth streaming, pp. 59–66, January 2011

    Google Scholar 

  29. Ragan, E., Kopper, R., Schuchardt, P., Bowman, D.: Studying the effects of stereo, head tracking, and field of regard on a small-scale spatial judgment task. IEEE Trans. Vis. Comput. Graph. 19(5), 886–896 (2012)

    Article  Google Scholar 

  30. Rojas, E., et al.: Telementoring in leg fasciotomies via mixed-reality: clinical evaluation of the star platform. Mil. Med. 185, 513–520 (2020)

    Article  Google Scholar 

  31. Roth, D., et al.: Real-time mixed reality teleconsultation for intensive care units in pandemic situations, pp. 693–694, March 2021

    Google Scholar 

  32. Schröder, C., Sharma, M., Teuber, J., Weller, R., Zachmann, G.: Dyncam: a reactive multithreaded pipeline library for 3d telepresence in VR. In: Proceedings of the 20th ACM Virtual Reality International Conference (VRIC 2018), ACM (2018)

    Google Scholar 

  33. Schubert, T., Friedmann, F., Regenbrecht, H.: The experience of presence: factor analytic insights. Presence Teleoperators Virtual Environ. 10(3), 266–281 (2001)

    Google Scholar 

  34. Söderholm, H., Sonnenwald, D., Cairns, B., Manning, J., Welch, G., Fuchs, H.: The potential impact of 3d telepresence technology on task performance in emergency trauma care, pp. 79–88, November 2007

    Google Scholar 

  35. Teng, C.C., Jensen, N., Smith, T., Forbush, T., Fletcher, K., Hoover, M.: Interactive augmented live virtual reality streaming: a health care application, pp. 143–147, June 2018

    Google Scholar 

  36. Thoravi Kumaravel, B., Anderson, F., Fitzmaurice, G., Hartmann, B., Grossman, T.: Loki: facilitating remote instruction of physical tasks using bi-directional mixed-reality telepresence, pp. 161–174, October 2019

    Google Scholar 

  37. Tölgyessy, M., Dekan, M., Chovanec, L., Hubinský, P.: Evaluation of the azure Kinect and its comparison to Kinect v1 and Kinect v2. Sensors 21, 413 (2021)

    Article  Google Scholar 

  38. Wilson, A.: Fast lossless depth image compression, pp. 100–105, October 2017

    Google Scholar 

  39. Yu, K., Gorbachev, G., Eck, U., Pankratz, F., Navab, N., Roth, D.: Avatars for teleconsultation: effects of avatar embodiment techniques on user perception in 3d asymmetric telepresence. IEEE Trans. Vis. Comput. Graph. 27(11), 4129–4139 (2021)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially funded by the German Federal Ministry of Education and Research (BMBF) under the grant 16SV8077.

Author information

Authors and Affiliations

  1. University of Bremen, Bremen, Germany

    Roland Fischer, Andre Mühlenbrock, Farin Kulapichitr & Gabriel Zachmann

  2. University Hospital for Visceral Surgery, University of Oldenburg, PIUS-Hospital, Oldenburg, Germany

    Verena Nicole Uslar & Dirk Weyhe

Authors
  1. Roland Fischer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andre Mühlenbrock
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Farin Kulapichitr
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Verena Nicole Uslar
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dirk Weyhe
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Gabriel Zachmann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roland Fischer .

Editor information

Editors and Affiliations

  1. University of Bremen, Bremen, Germany

    Gabriel Zachmann

  2. Universitat Politècnica de València, Valencia, Spain

    Mariano Alcañiz Raya

  3. University of Paris-Saclay, Orsay Cedex, France

    Patrick Bourdot

  4. INSA, IRISA, University of Rennes, Rennes Cedex, France

    Maud Marchal

  5. University of Utah, Salt Lake City, UT, USA

    Jeanine Stefanucci

  6. Shanghai Jiao Tong University, Shanghai, China

    Xubo Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fischer, R., Mühlenbrock, A., Kulapichitr, F., Uslar, V.N., Weyhe, D., Zachmann, G. (2022). Evaluation of Point Cloud Streaming and Rendering for VR-Based Telepresence in the OR. In: Zachmann, G., Alcañiz Raya, M., Bourdot, P., Marchal, M., Stefanucci, J., Yang, X. (eds) Virtual Reality and Mixed Reality. EuroXR 2022. Lecture Notes in Computer Science, vol 13484. Springer, Cham. https://doi.org/10.1007/978-3-031-16234-3_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-16234-3_6

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-16233-6

  • Online ISBN: 978-3-031-16234-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics