Skip to main content

Advertisement

SpringerLink
Log in

Validation of real-time inside-out tracking and depth realization technologies for augmented reality-based neuronavigation

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Concomitant with the significant advances in computing technology, the utilization of augmented reality-based navigation in clinical applications is being actively researched. In this light, we developed novel object tracking and depth realization technologies to apply augmented reality-based neuronavigation to brain surgery.

Methods

We developed real-time inside-out tracking based on visual inertial odometry and a visual inertial simultaneous localization and mapping algorithm. The cube quick response marker and depth data obtained from light detection and ranging sensors are used for continuous tracking. For depth realization, order-independent transparency, clipping, and annotation and measurement functions were developed. In this study, the augmented reality model of a brain tumor patient was applied to its life-size three-dimensional (3D) printed model.

Results

Using real-time inside-out tracking, we confirmed that the augmented reality model remained consistent with the 3D printed patient model without flutter, regardless of the movement of the visualization device. The coordination accuracy during real-time inside-out tracking was also validated. The average movement error of the X and Y axes was 0.34 ± 0.21 and 0.04 ± 0.08 mm, respectively. Further, the application of order-independent transparency with multilayer alpha blending and filtered alpha compositing improved the perception of overlapping internal brain structures. Clipping, and annotation and measurement functions were also developed to aid depth perception and worked perfectly during real-time coordination. We named this system METAMEDIP navigation.

Conclusions

The results validate the efficacy of the real-time inside-out tracking and depth realization technology. With these novel technologies developed for continuous tracking and depth perception in augmented reality environments, we are able to overcome the critical obstacles in the development of clinically applicable augmented reality neuronavigation.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Barfield W, Caudell T (2001) Boeing’s wire bundle assembly project. In: Barfield W, Caudell T (eds) Fundamentals of wearable computers and augmented reality. CRC Press, Boca Raton, pp 462–482. https://doi.org/10.1201/9780585383590-21

  2. Sutherland IE (1968) A head-mounted three dimensional display. In: Proceedings of the December 9–11, 1968 fall joint computer conference Part. I, pp 757–764. https://doi.org/10.1145/1476589.1476686

  3. Liberatore MJ, Wagner WP (2021) Virtual, mixed, and augmented reality: a systematic review for immersive systems research. Virtual Real 25:773–799. https://doi.org/10.1007/s10055-020-00492-0

    Article  Google Scholar 

  4. Venkatesan M, Mohan H, Ryan JR, Schürch CM, Nolan GP, Frakes DH, Coskun AF (2021) Virtual and augmented reality for biomedical applications. Cell Rep Med 2:100348. https://doi.org/10.1016/j.xcrm.2021.100348

    Article  PubMed  PubMed Central  Google Scholar 

  5. Galloway RL, Maciunas RJ, Edwards CA (1992) Interactive image-guided neurosurgery. IEEE Trans Biomed Eng 39:1226–1231. https://doi.org/10.1109/10.184698

    Article  PubMed  Google Scholar 

  6. Kikinis R, Gleason PL, Moriarty TM, Moore MR, Alexander E, Stieg PE, Matsumae M, Lorensen WE, Cline HE, Black PM (1996) Computer-assisted interactive three-dimensional planning neurosurgical procedures. Neurosurg 38:640–649. https://doi.org/10.1227/00006123-199604000-00003

    Article  CAS  Google Scholar 

  7. Cho J, Rahimpour S, Cutler A, Goodwin CR, Lad SP, Codd P (2020) Enhancing reality: a systematic review of augmented reality in neuronavigation and education. World Neurosurg 139:186–195. https://doi.org/10.1016/j.wneu.2020.04.043

    Article  PubMed  Google Scholar 

  8. Mikhail M, Mithani K, Ibrahim GM (2019) Presurgical and intraoperative augmented reality in neuro-oncologic surgery: clinical experiences and limitations. World Neurosurg 128:268–276. https://doi.org/10.1016/j.wneu.2019.04.256

    Article  PubMed  Google Scholar 

  9. Fick T, van Doormaal JA, Hoving EW, Willems PW, van Doormaal TP (2021) Current accuracy of augmented reality neuronavigation systems: Systematic review and meta-analysis. World Neurosurg 146:179–188. https://doi.org/10.1016/j.wneu.2020.11.029

    Article  PubMed  Google Scholar 

  10. Haemmerli J, Davidovic A, Meling TR, Chavaz L, Schaller K, Bijlenga P (2021) Evaluation of the precision of operative augmented reality compared to standard neuronavigation using a 3D-printed skull. Neurosurg Focus 50:E17. https://doi.org/10.3171/2020.10.FOCUS20789

    Article  PubMed  Google Scholar 

  11. Bernard F, Haemmerli J, Zegarek G, Kiss-Bodolay D, Schaller K, Bijlenga P (2021) Augmented reality–assisted roadmaps during periventricular brain surgery. Neurosurg Focus 51:E4. https://doi.org/10.3171/2021.5.FOCUS21220

    Article  PubMed  Google Scholar 

  12. Louis RG, Steinberg GK, Duma C, Britz G, Mehta V, Pace J, Selman W, Jean WC (2021) Early experience with virtual and synchronized augmented reality platform for preoperative planning and intraoperative navigation: a case series. Oper Neurosurg (Hagerstown) 21:189–196. https://doi.org/10.1093/ons/opab188

    Article  PubMed  Google Scholar 

  13. Dho YS, Park SJ, Choi H, Kim Y, Moon HC, Kim KM, Kang H, Lee EJ, Kim MS, Kim JW (2021) Development of an inside-out augmented reality technique for neurosurgical navigation. Neurosurg Focus 51:E21. https://doi.org/10.3171/2021.5.FOCUS21184

    Article  PubMed  Google Scholar 

  14. Bichlmeier C, Wimmer F, Heining SM, Navab N (2007) Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality. In: 6th IEEE and ACM international symposium on mixed and augmented reality, vol. 2007, 2007, pp 129–138. https://doi.org/10.1109/ISMAR.2007.4538837

  15. Sielhorst T, Bichlmeier C, Heining SM, Navab N (2006) Depth perception—a major issue in medical AR: Evaluation study by twenty surgeons. In: Larsen R, Nielsen M, Sporring J (eds) Medical Image Computing and Computer-Assisted Intervention. Lecture Notes in Computer Science, vol 4190, Springer, Berlin, Heidelberg. pp 364–372. https://doi.org/10.1007/11866565_45

  16. Dho YS, Lee D, Ha T, Ji SY, Kim KM, Kang H, Kim M-S, Kim JW, Cho W-S, Kim YH (2021) Clinical application of patient-specific 3D printing brain tumor model production system for neurosurgery. Sci Rep 11:7005. https://doi.org/10.1038/s41598-021-86546-y

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  17. Salvi M, Vaidyanathan K (2014) Multi-layer alpha blending. In: Proceedings of the 18th Meeting of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, pp 151–158. https://doi.org/10.1145/2556700.2556705

  18. Maruyama K, Watanabe E, Kin T, Saito K, Kumakiri A, Noguchi A, Nagane M, Shiokawa Y (2018) Smart glasses for neurosurgical navigation by augmented reality. Oper Neurosurg (Hagerstown) 15:551–556. https://doi.org/10.1093/ons/opx279

    Article  PubMed  Google Scholar 

  19. Choi H, Cho B, Mesamune K, Hashizume M, Hong J (2016) An effective visualization technique for depth perception in augmented reality-based surgical navigation. Int J Med Robot 12:62–72. https://doi.org/10.1002/rcs.1657

    Article  PubMed  Google Scholar 

  20. Cidota MA, Clifford RMS, Lukosch SG, Bilinghurst M (2016) Using visual effects to facilitate depth perception for spatial tasks in virtual and augmented reality. In: IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct), vol. 2016, 2016, pp 172–177. https://doi.org/10.1109/ISMAR-Adjunct.2016.0070

  21. El Jamiy F, Marsh R, (2019) Distance estimation in virtual reality and augmented reality: a survey. In: IEEE International Conference on Electro Information Technology (EIT), vol. 2019, 2019, pp 063–068. https://doi.org/10.1109/EIT.2019.8834182

  22. Reitinger B, Welrberger P, Bornik A, Beichel R, Schmalstieg D (2005) Spatial measurements for medical augmented reality. In: Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR'05). IEEE, pp 208–209. https://doi.org/10.1109/ISMAR.2005.53

  23. Soler L, Hostettler A, Collins T, Pessaux P, Mutter D, Marescaux J (2021) The visible patient: augmented reality in the operating theater. In: Atallah S (ed) Digital surgery, Springer, Cham, pp 247–260. https://doi.org/10.1007/978-3-030-49100-0_18

Download references

Funding

This work was supported by the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: 202012E08).

Author information

Author notes

  1. Yun-Sik Dho and Byeong Cheol Lee have contributed equally to this work.

Authors and Affiliations

  1. Neuro-Oncology Clinic, National Cancer Center, Goyang, Republic of Korea

    Yun-Sik Dho

  2. Research and Science Division, Research and Development Center, MEDICALIP Co. Ltd., Seoul, Republic of Korea

    Byeong Cheol Lee & Sang Joon Park

  3. Department of Neurosurgery, Chungbuk National University Hospital, Cheongju, Republic of Korea

    Hyeong Cheol Moon

  4. Department of Neurosurgery, Inha University Hospital, Inha University College of Medicine, Incheon, Korea

    Kyung Min Kim

  5. Department of Neurosurgery, Seoul National University Hospital, Seoul National University College of Medicine, 101 Daehak-Ro, Jongno-Gu, Seoul, 03080, Republic of Korea

    Ho Kang, Eun Jung Lee, Min-Sung Kim, Jin Wook Kim, Yong Hwy Kim & Chul-Kee Park

  6. Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, 101 Daehak-Ro, Jongno-Gu, Seoul, 03080, Republic of Korea

    Sang Joon Park

Authors
  1. Yun-Sik Dho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Byeong Cheol Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hyeong Cheol Moon
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kyung Min Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ho Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eun Jung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Min-Sung Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jin Wook Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Yong Hwy Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sang Joon Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Chul-Kee Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Sang Joon Park or Chul-Kee Park.

Ethics declarations

Conflict of interest

Sang Joon Park is the founder and CEO of MEDICALIP. Chul-Kee Park and Yun-Sik Dho own stock options and investments in MEDICALIP. The other authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 7786 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dho, YS., Lee, B.C., Moon, H.C. et al. Validation of real-time inside-out tracking and depth realization technologies for augmented reality-based neuronavigation. Int J CARS 19, 15–25 (2024). https://doi.org/10.1007/s11548-023-02993-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-023-02993-0

Keywords

Search

Navigation