Skip to main content
SpringerLink
Log in

Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

  • Conference paper
  • First Online:
Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures (CLIP 2020, ML-CDS 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12445))

Abstract

Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient’s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ahn, J., Choi, H., Hong, J., Hong, J.: Tracking accuracy of a stereo-camera-based augmented reality navigation system for orthognathic surgery. J. Oral Maxillofac. Surg. 77(5), 1070.e1–1070.e11 (2019)

    Article  Google Scholar 

  2. Chen, X., et al.: Development of a surgical navigation system based on augmented reality using an optical see-through head-mounted display. J. Biomed. Inform. 55, 124–131 (2015)

    Article  Google Scholar 

  3. Chen, Y., Medioni, G.: Object modelling by registration of multiple range images. Image Vis. Comp. 10(3), 145–155 (1992)

    Article  Google Scholar 

  4. Choi, S., Zhou, Q.Y., Koltun, V.: Robust reconstruction of indoor scenes. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

    Google Scholar 

  5. Eggers, G., Mühling, J., Marmulla, R.: Image-to-patient registration techniques in head surgery. Int. J. Oral Maxillofac. Surg. 35(12), 1081–1095 (2006)

    Article  Google Scholar 

  6. Fan, Y., Jiang, D., Wang, M., Song, Z.: A new markerless patient-to-image registration method using a portable 3D scanner. Med. Phys. 41(10), 101910 (2014)

    Article  Google Scholar 

  7. Gsaxner, C., Pepe, A., Wallner, J., Schmalstieg, D., Egger, J.: Markerless image-to-face registration for untethered augmented reality in head and neck surgery. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11768, pp. 236–244. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32254-0_27

    Chapter  Google Scholar 

  8. Gsaxner, C., Wallner, J., Chen, X., Zemann, W., Egger, J.: Facial model collection for medical augmented reality in oncologic cranio-maxillofacial surgery. Scientific Data 6(1), 310 (2019)

    Article  Google Scholar 

  9. Holz, D., Ichim, A.E., Tombari, F., Rusu, R.B., Behnke, S.: Registration with the point cloud library: a modular framework for aligning in 3-D. IEEE Robot. Autom. Mag. 22(4), 110–124 (2015)

    Article  Google Scholar 

  10. Jackson, A.S., Bulat, A., Argyriou, V., Tzimiropoulos, G.: Large pose 3D face reconstruction from a single image via direct volumetric CNN regression. In: International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  11. Jayender, J., Xavier, B., King, F., Hosny, A., Black, D., Pieper, S., Tavakkoli, A.: A novel mixed reality navigation system for laparoscopy surgery. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11073, pp. 72–80. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00937-3_9

    Chapter  Google Scholar 

  12. Jiang, T., Zhu, M., Chai, G., Li, Q.: Precision of a novel craniofacial surgical navigation system based on augmented reality using an occlusal splint as a registration strategy. Sci. Rep. 9(1), 501 (2019)

    Article  Google Scholar 

  13. Lamecker, H., et al.: Automatic segmentation of mandibles in low-dose CT-data. Int. J. Comput. Assisted Radiol. Surg. 1, 393 (2006)

    Google Scholar 

  14. Lorensen, W.E., Cline, H.E.: Marching cubes: a high resolution 3d surface construction algorithm. ACM Siggraph Comput. Graph. 21(4), 163–169 (1987)

    Article  Google Scholar 

  15. Markelj, P., Tomaževič, D., Likar, B., Pernuš, F.: A review of 3D/2D registration methods for image-guided interventions. Med. Image Anal. 16(3), 642–661 (2012)

    Article  Google Scholar 

  16. Maruyama, K., et al.: Smart glasses for neurosurgical navigation by augmented reality. Operative Neurosurgery 15(5), 551–556 (2018)

    Article  Google Scholar 

  17. McCann, M.T., Nilchian, M., Stampanoni, M., Unser, M.: Fast 3d reconstruction method for differential phase contrast x-ray CT. Optics Express 24(13), 14564–14581 (2016)

    Article  Google Scholar 

  18. Meulstee, J.W., et al.: Toward holographic-guided surgery. Surgical Innov. 26(1), 86–94 (2019)

    Google Scholar 

  19. Olabarriaga, S.D., Smeulders, A.W.: Interaction in the segmentation of medical images: a survey. Med. Image Anal. 5(2), 127–142 (2001)

    Article  Google Scholar 

  20. Orentlicher, G., Goldsmith, D., Horowitz, A.: Applications of 3-dimensional virtual computerized tomography technology in oral and maxillofacial surgery: current therapy. J. Oral Maxillofacial Surgery 68(8), 1933–1959 (2010)

    Article  Google Scholar 

  21. Pepe, A., et al.: Pattern recognition and mixed reality for computer-aided maxillofacial surgery and oncological assessment. In: Proceedings Biomedical Engineering International Conference (BMEiCON), pp. 1–5. IEEE, January 2019

    Google Scholar 

  22. Pepe, A., et al.: A marker-less registration approach for mixed reality–aided maxillofacial surgery: a pilot evaluation. J. Dig. Imag. 32(6), 1008–1018 (2019). https://doi.org/10.1007/s10278-019-00272-6

    Article  Google Scholar 

  23. Rusu, R.B., Blodow, N., Beetz, M.: Fast point feature histograms (FPFH) for 3D registration. In: ICRA (2009)

    Google Scholar 

  24. Sielhorst, T., Feuerstein, M., Navab, N.: Advanced medical displays: a literature review of augmented reality. J. Disp. Technol. 4(4), 26 (2008)

    Google Scholar 

  25. Sylos Labini, M., Gsaxner, C., Pepe, A., Wallner, J., Egger, J., Bevilacqua, V.: Depth-awareness in a system for mixed-reality aided surgical procedures. In: Huang, D.-S., Huang, Z.-K., Hussain, A. (eds.) ICIC 2019. LNCS (LNAI), vol. 11645, pp. 716–726. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-26766-7_65

    Chapter  Google Scholar 

  26. Tucker, S., et al.: Comparison of actual surgical outcomes and 3-dimensional surgical simulations. J. Oral Maxillofacial Surg. 68(10), 2412–2421 (2010)

    Article  Google Scholar 

  27. Wallner, J., et al.: Clinical evaluation of semi-automatic open-source algorithmic software segmentation of the mandibular bone: practical feasibility and assessment of a new course of action. PLoS ONE 13(5), 156–165 (2018)

    Google Scholar 

  28. Wallner, J., Schwaiger, M., Hochegger, K., Gsaxner, C., Zemann, W., Egger, J.: A review on multiplatform evaluations of semi-automatic open-source based image segmentation for cranio-maxillofacial surgery. In: Computer Methods and Programs in Biomedicine, p. 105102 (2019)

    Google Scholar 

  29. Wang, J., Shen, Yu., Yang, S.: A practical marker-less image registration method for augmented reality oral and maxillofacial surgery. Int. J. Comput. Assisted Radiol. Surg. 14(5), 763–773 (2019). https://doi.org/10.1007/s11548-019-01921-5

    Article  Google Scholar 

  30. Weber, M., Wild, D., Wallner, J., Egger, J.: A client/server based online environment for the calculation of medical segmentation scores. In: EMBC, pp. 3463–3467 (2019). https://doi.org/10.1109/EMBC.2019.8856481

  31. Wild, D., Weber, M., Wallner, J., Egger, J.: Client/server based online environment for manual segmentation of medical images. CoRR abs/1904.08610 (2019). http://arxiv.org/abs/1904.08610

Download references

Acknowledgment

This work received funding from the Austrian Science Fund (FWF) KLI 678-B31 (enFaced - Virtual and Augmented Reality Training and Navigation Module for 3D-Printed Facial Defect Reconstructions). Further, this work sees the support of CAMed - Clinical additive manufacturing for medical applications (COMET K-Project 871132), which is funded by the Austrian Federal Ministry of Transport, Innovation and Technology (BMVIT), and the Austrian Federal Ministry for Digital and Economic Affairs (BMDW), and the Styrian Business Promotion Agency (SFG), and the TU Graz Lead Project (Mechanics, Modeling and Simulation of Aortic Dissection). Moreover, the Summer Bachelor (SB) Program of the Institute of Computer Graphics and Vision (ICG) of the Graz University of Technology (TU Graz). Finally, we want to point out to our medical online framework Studierfenster (www.studierfenster.at), where an automatic single-shot 3D face reconstruction and registration module has been integrated, and a video tutorial is available on YouTube (3D Face Reconstruction and Registration with Studierfenster: https://www.youtube.com/watch?v=DbbFm9XxlGE).

Author information

Authors and Affiliations

  1. Institute of Computer Graphics and Vision, Graz University of Technology, Graz, Austria

    Florian Karner, Christina Gsaxner, Antonio Pepe, Jianning Li, Philipp Fleck, Clemens Arth & Jan Egger

  2. Computer Algorithms for Medicine Laboratory (Café-Lab), Graz, Austria

    Florian Karner, Christina Gsaxner, Antonio Pepe, Jianning Li, Jürgen Wallner & Jan Egger

  3. Department of Oral and Maxillofacial Surgery, Medical University of Graz, Graz, Austria

    Christina Gsaxner, Jürgen Wallner & Jan Egger

Authors
  1. Florian Karner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christina Gsaxner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Antonio Pepe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianning Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Philipp Fleck
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Clemens Arth
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jürgen Wallner
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jan Egger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jan Egger .

Editor information

Editors and Affiliations

  1. IBM Almaden Research Center, San Jose, CA, USA

    Tanveer Syeda-Mahmood

  2. Aachen University of Applied Sciences, Aachen, Germany

    Klaus Drechsler

  3. Tel Aviv University, Ramat Aviv, Israel

    Hayit Greenspan

  4. Case Western Reserve University, Cleveland, OH, USA

    Anant Madabhushi

  5. IBM Almaden Research Center, San Jose, CA, USA

    Alexandros Karargyris

  6. Children’s National Hospital, Washington, DC, USA

    Marius George Linguraru

  7. Fraunhofer-Institute for Computer Graphics Research (IGD), Darmstadt, Germany

    Cristina Oyarzun Laura

  8. Children’s National Hospital, Washington, DC, USA

    Raj Shekhar

  9. Fraunhofer-Institute for Computer Graphics Research (IGD), Darmstadt, Germany

    Stefan Wesarg

  10. Universitat Pompeu Fabra, Barcelona, Spain

    Miguel Ángel González Ballester

  11. Fraunhofer Singapore, Singapore, Singapore

    Marius Erdt

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Karner, F. et al. (2020). Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality. In: Syeda-Mahmood, T., et al. Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures. CLIP ML-CDS 2020 2020. Lecture Notes in Computer Science(), vol 12445. Springer, Cham. https://doi.org/10.1007/978-3-030-60946-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60946-7_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60945-0

  • Online ISBN: 978-3-030-60946-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation