Skip to main content
SpringerLink
Log in

Augmented Reality in Orthognathic Surgery: A Multi-Modality Tracking Approach to Assess the Temporomandibular Joint Motion

  • Conference paper
  • First Online:
Extended Reality (XR Salento 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14219))

Included in the following conference series:

  • 556 Accesses

Abstract

Augmented Reality (AR) is a rapidly emerging technology finding growing applications in various surgery domains. In this study, we develop and test the feasibility of a novel AR application for Microsoft HoloLens2 Head Mounted Display (HMD) to support surgeons in the clinical evaluation of temporomandibular joint (TMJ) alterations that may require surgery. The application implements a multi-modality tracking based on the combination of a marker-less and a marker-based approach to simultaneously track the fixed part of the joint and the moving mandible. The AR application was tested on a volunteer performing the TMJ task, i.e. the opening and closing of the mouth. During the task, video recordings were taken from the HoloLens cameras to derive the trajectories as well as the horizontal and vertical excursions of the jaw movements. The AR-derived TMJ movements were then compared with standard kinesiographic acquisitions. The results demonstrated the feasibility of the proposed AR application in superimposing the 3D visualization of the joint to the patient’s head, thus facilitating the diagnostic evaluation for the surgeon. The AR-derived trajectories were consistent with the kinesiography curves. Future improvements are needed to reduce the encumbrance of the tracker and to provide additional visual cues for the surgeon. The presented methodology can be easily transferred to other surgical applications which require simultaneous tracking of two anatomical parts, such as the case of bone repositioning to a pre-planned target location in maxillofacial and orthopedic surgery.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Cannizzaro, D., et al.: Augmented reality in neurosurgery, state of art and future projections: a systematic review. Front Surg. 9, 864792 (2022). https://doi.org/10.3389/fsurg.2022.864792

    Article  Google Scholar 

  2. Dadario, N.B., Quinoa, T., Khatri, D., Boockvar, J., Langer, D., D’Amico, R.S.: Examining the benefits of extended reality in neurosurgery: a systematic review. J. Clin. Neurosci. 94, 41–53 (2021). https://doi.org/10.1016/j.jocn.2021.09.037

    Article  Google Scholar 

  3. Hunter Matthews, J., Shields, J.S.: The clinical application of augmented reality in orthopaedics: where do we stand? Curr. Rev. Musculoskel. Med. 14(5), 316–319 (2021). https://doi.org/10.1007/s12178-021-09713-8

    Article  Google Scholar 

  4. Jud, Lukas, et al.: Applicability of augmented reality in orthopedic surgery – a systematic review. BMC Musculoskel. Disord. 21(1), 103 (2020). https://doi.org/10.1186/s12891-020-3110-2

    Article  Google Scholar 

  5. Badiali, G., et al.: Review on augmented reality in oral and cranio-maxillofacial surgery: toward “Surgery-Specific” head-up displays. IEEE Access 8, 59015–59028 (2020). https://doi.org/10.1109/ACCESS.2020.2973298

    Article  Google Scholar 

  6. Benmahdjoub, M., van Walsum, T., van Twisk, P., Wolvius, E.B.: Augmented reality in craniomaxillofacial surgery: added value and proposed recommendations through a systematic review of the literature. Int. J. Oral Maxillofacial Surg. 50(7), 969–978 (2021). https://doi.org/10.1016/j.ijom.2020.11.015

    Article  Google Scholar 

  7. Ceccariglia, F., Cercenelli, L., Badiali, G., Marcelli, E., Tarsitano, A.: Application of augmented reality to maxillary resections: a three-dimensional approach to maxillofacial oncologic surgery. J. Pers. Med. 12(12), 2047 (2022). https://doi.org/10.3390/jpm12122047

    Article  Google Scholar 

  8. Battaglia, S., et al.: Augmented reality-assisted periosteum pedicled flap harvesting for head and neck reconstruction: an anatomical and clinical viability study of a galeo-pericranial flap. J Clin Med 9(7), E2211 (2020). https://doi.org/10.3390/jcm9072211

    Article  Google Scholar 

  9. Battaglia, S., et al.: Combination of CAD/CAM and augmented reality in free fibula bone harvest. Plast. Reconstr. Surg. Glob. Open 7(11), e2510 (2019). https://doi.org/10.1097/GOX.0000000000002510

    Article  Google Scholar 

  10. Reis, G., et al.: Mixed reality applications in urology: requirements and future potential. Ann. Med. Surg. (Lond.) 66, 102394 (2021). https://doi.org/10.1016/j.amsu.2021.102394

  11. Schiavina, Riccardo, et al.: Real-time augmented reality three-dimensional guided robotic radical prostatectomy: preliminary experience and evaluation of the impact on surgical planning. Eur. Urol. Focus 7(6), 1260–1267 (2021). https://doi.org/10.1016/j.euf.2020.08.004

    Article  Google Scholar 

  12. Schiavina, R., et al.: Augmented reality to guide selective clamping and tumor dissection during robot-assisted partial nephrectomy: a preliminary experience. Clin. Genitourin Cancer 19(3), e149–e155 (2021). https://doi.org/10.1016/j.clgc.2020.09.005

    Article  MathSciNet  Google Scholar 

  13. Bianchi, L., et al.: The use of augmented reality to guide the intraoperative frozen section during robot-assisted radical prostatectomy. Eur. Urol. 80(4), 480–488 (2021). https://doi.org/10.1016/j.eururo.2021.06.020

    Article  Google Scholar 

  14. Li, T., et al.: Augmented reality in ophthalmology: applications and challenges. Front Med. (Lausanne) 8, 733241 (2021). https://doi.org/10.3389/fmed.2021.733241

  15. Lareyre, F., Chaudhuri, A., Adam, C., Carrier, M., Mialhe, C., Raffort, J.: Applications of head-mounted displays and smart glasses in vascular surgery. Ann. Vasc. Surg. 75, 497–512 (2021). https://doi.org/10.1016/j.avsg.2021.02.033

    Article  Google Scholar 

  16. Molina, C.A., Sciubba, D.M., Greenberg, J.K., Khan, M., Witham, T.: Clinical accuracy, technical precision, and workflow of the first in human use of an augmented-reality head-mounted display stereotactic navigation system for spine surgery. Oper. Neurosurg. 20(3), 300–309 (2021). https://doi.org/10.1093/ons/opaa398

    Article  Google Scholar 

  17. McCloskey, K., Turlip, R., Ahmad, H.S., Ghenbot, Y.G., Chauhan, D., Yoon, J.W.: Virtual and augmented reality in spine surgery: a systematic review. World Neurosurg. 173, 96–107 (2023). https://doi.org/10.1016/j.wneu.2023.02.068

    Article  Google Scholar 

  18. Doughty, M., Ghugre, N.R., Wright, G.A.: Augmenting performance: a systematic review of optical see-through head-mounted displays in surgery. J. Imaging 8(7), 203 (2022). https://doi.org/10.3390/jimaging8070203

    Article  Google Scholar 

  19. Gsaxner, C., et al.: The HoloLens in medicine: a systematic review and taxonomy. Med. Image Anal. 85, 102757 (2023). https://doi.org/10.1016/j.media.2023.102757

    Article  Google Scholar 

  20. Cercenelli, L., et al.: Augmented reality to assist skin paddle harvesting in osteomyocutaneous fibular flap reconstructive surgery: a pilot evaluation on a 3D-printed leg phantom. Front. Oncol. 11, 804748 (2022). https://doi.org/10.3389/fonc.2021.804748

    Article  Google Scholar 

  21. Puxun, T., Gao, Y., Lungu, A.J., Li, D., Wang, H., Chen, X.: Augmented reality based navigation for distal interlocking of intramedullary nails utilizing Microsoft HoloLens 2. Comput. Biol. Med. 133, 104402 (2021). https://doi.org/10.1016/j.compbiomed.2021.104402

    Article  Google Scholar 

  22. Zhou, Z., Jiang, S., Yang, Z., Bin, X., Jiang, B.: Surgical navigation system for brachytherapy based on mixed reality using a novel stereo registration method. Virt. Real. 25(4), 975–984 (2021). https://doi.org/10.1007/s10055-021-00503-8

    Article  Google Scholar 

  23. Uhl, C., Hatzl, J., Meisenbacher, K., Zimmer, L., Hartmann, N., Böckler, D.: Mixed-reality-assisted puncture of the common femoral artery in a phantom model. J. Imaging 8(2), 47 (2022). https://doi.org/10.3390/jimaging8020047

    Article  Google Scholar 

  24. Ruggiero, F., et al.: Preclinical application of augmented reality in pediatric craniofacial surgery: an accuracy study. J. Clin. Med. 12(7), 2693 (2023). https://doi.org/10.3390/jcm12072693

    Article  Google Scholar 

  25. Cercenelli, L., et al.: The wearable VOSTARS system for augmented reality-guided surgery: preclinical phantom evaluation for high-precision maxillofacial tasks. J. Clin. Med. 9(11), E3562 (2020). https://doi.org/10.3390/jcm9113562

    Article  Google Scholar 

  26. Condino, S., et al.: Wearable augmented reality platform for aiding complex 3D trajectory tracing. Sensors (Basel) 20(6), E1612 (2020). https://doi.org/10.3390/s20061612

    Article  Google Scholar 

  27. Carbone, M., et al.: Architecture of a hybrid video/optical see-through head-mounted display-based augmented reality surgical navigation platform. Information 13(2), 81 (2022). https://doi.org/10.3390/info13020081

    Article  Google Scholar 

  28. Badiali, G., et al.: The vostars project: a new wearable hybrid video and optical see-through augmented reality surgical system for maxillofacial surgery. Int. J. Oral Maxillofacial Surg. 48, 153 (2019). https://doi.org/10.1016/j.ijom.2019.03.472

    Article  Google Scholar 

  29. Venturi, G., et al.: Use of kinesiography to assess mandibular function following segmental resection and microvascular reconstruction. J Craniofac Surg 31(8), 2256–2259 (2020). https://doi.org/10.1097/SCS.0000000000006774

    Article  Google Scholar 

  30. Ma, L., Huang, T., Wang, J., Liao, H.: Visualization, registration and tracking techniques for augmented reality guided surgery: a review. Phys Med Biol 68(4), 04TR02 (2023). https://doi.org/10.1088/1361-6560/acaf23

    Article  Google Scholar 

  31. Liebmann, F., et al.: Pedicle screw navigation using surface digitization on the Microsoft HoloLens. Int. J. Cars 14(7), 1157–1165 (2019). https://doi.org/10.1007/s11548-019-01973-7

    Article  Google Scholar 

  32. Frantz, T., Jansen, B., Duerinck, J., Vandemeulebroucke, J.: Augmenting Microsoft’s HoloLens with vuforia tracking for neuronavigation. Healthc Technol. Lett. 5(5), 221–225 (2018). https://doi.org/10.1049/htl.2018.5079

    Article  Google Scholar 

  33. Luzon, J.A., Stimec, B.V., Bakka, A.O., Edwin, B., Ignjatovic, D.: Value of the surgeon’s sightline on hologram registration and targeting in mixed reality. Int. J. Cars 15(12), 2027–2039 (2020). https://doi.org/10.1007/s11548-020-02263-3

    Article  Google Scholar 

  34. Zhou, Z., Yang, Z., Jiang, S., Zhuo, J., Zhu, T., Ma, S.: Augmented reality surgical navigation system based on the spatial drift compensation method for glioma resection surgery. Med. Phys. 49(6), 3963–3979 (2022). https://doi.org/10.1002/mp.15650

    Article  Google Scholar 

  35. Dibble, C.F., Molina, C.A.: Device profile of the XVision-spine (XVS) augmented-reality surgical navigation system: overview of its safety and efficacy. Expert Rev. Med. Dev. 18(1), 1–8 (2021). https://doi.org/10.1080/17434440.2021.1865795

    Article  Google Scholar 

  36. Gu, W., Shah, K., Knopf, J., Navab, N., Unberath, M.: Feasibility of image-based augmented reality guidance of total shoulder arthroplasty using microsoft HoloLens 1. Comput. Methods Biomech. Biomed. Eng. Imaging Vis. 9(3), 261–270 (2021). https://doi.org/10.1080/21681163.2020.1835556

    Article  Google Scholar 

  37. Pepe, A., et al.: A marker-less registration approach for mixed reality-aided maxillofacial surgery: a pilot evaluation. J. Digit Imaging 32(6), 1008–1018 (2019). https://doi.org/10.1007/s10278-019-00272-6

    Article  Google Scholar 

  38. Guo, N., Wang, T., Yang, B., Hu, L., Liu, H., Wang, Y.: An online calibration method for microsoft hololens. IEEE Access 7, 101795–101803 (2019). https://doi.org/10.1109/ACCESS.2019.2930701

    Article  Google Scholar 

  39. Chan, H.H.L., et al.: An integrated augmented reality surgical navigation platform using multi-modality imaging for guidance. PLOS ONE 16(4), e0250558 (2021). https://doi.org/10.1371/journal.pone.0250558

    Article  Google Scholar 

  40. Hu, X., Baena, F.R., Cutolo, F.: Head-mounted augmented reality platform for markerless orthopaedic navigation. IEEE J. Biomed. Health Inf. 26(2), 910–921 (2022). https://doi.org/10.1109/JBHI.2021.3088442

    Article  Google Scholar 

  41. Gao, Y., Liu, K., Lin, L., Wang, X., Xie, L.: Use of augmented reality navigation to optimise the surgical management of craniofacial fibrous dysplasia. Brit. J. Oral Maxillofacial Surg. 60(2), 162–167 (2022). https://doi.org/10.1016/j.bjoms.2021.03.011

    Article  Google Scholar 

  42. El-Hariri, H., Pandey, P., Hodgson, A.J., Garbi, R.: Augmented reality visualisation for orthopaedic surgical guidance with pre- and intra-operative multimodal image data fusion. Healthcare Technol. Lett. 5(5), 189–193 (2018). https://doi.org/10.1049/htl.2018.5061

    Article  Google Scholar 

  43. Teatini, A., Kumar, R.P., Elle, O.J., Wiig, O.: Mixed reality as a novel tool for diagnostic and surgical navigation in orthopaedics. Int. J. Comput. Assist. Radiol. Surg. 16(3), 407–414 (2021). https://doi.org/10.1007/s11548-020-02302-z

    Article  Google Scholar 

  44. Palumbo, A.: Microsoft HoloLens 2 in medical and healthcare context: state of the art and future prospects. Sensors (Basel) 22(20), 7709 (2022). https://doi.org/10.3390/s22207709

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Laboratory of Bioengineering – eDIMES Lab, Department of Medical and Surgical Sciences (DIMEC), University of Bologna, Bologna, Italy

    Laura Cercenelli, Nicolas Emiliani & Emanuela Marcelli

  2. Department of Biomedical and Neuromotor Sciences (DIBINEM), University of Bologna, Bologna, Italy

    Chiara Gulotta & Giovanni Badiali

  3. Oral and Maxillofacial Surgery Unit, IRCCS Azienda Ospedaliero Universitaria di Bologna, Bologna, Italy

    Mirko Bevini & Giovanni Badiali

Authors
  1. Laura Cercenelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Nicolas Emiliani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chiara Gulotta
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mirko Bevini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Giovanni Badiali
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Emanuela Marcelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Laura Cercenelli .

Editor information

Editors and Affiliations

  1. University of Salento, Lecce, Italy

    Lucio Tommaso De Paolis

  2. University of Naples Federico II, Naples, Italy

    Pasquale Arpaia

  3. CNR-STIIMA, Lecco, Italy

    Marco Sacco

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cercenelli, L., Emiliani, N., Gulotta, C., Bevini, M., Badiali, G., Marcelli, E. (2023). Augmented Reality in Orthognathic Surgery: A Multi-Modality Tracking Approach to Assess the Temporomandibular Joint Motion. In: De Paolis, L.T., Arpaia, P., Sacco, M. (eds) Extended Reality. XR Salento 2023. Lecture Notes in Computer Science, vol 14219. Springer, Cham. https://doi.org/10.1007/978-3-031-43404-4_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43404-4_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43403-7

  • Online ISBN: 978-3-031-43404-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation