Skip to main content
SpringerLink
Log in

Mobile augmented reality for computer-assisted percutaneous nephrolithotomy

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose   

Percutaneous nephrolithotomy (PCNL) plays an integral role in treatment of renal stones. Creating percutaneous renal access is the most important and challenging step in the procedure. To facilitate this step, we evaluated our novel mobile augmented reality (AR) system for its feasibility of use for PCNL.

Methods   

A tablet computer, such as an iPad\(^{\circledR }\), is positioned above the patient with its camera pointing toward the field of intervention. The images of the tablet camera are registered with the CT image by means of fiducial markers. Structures of interest can be superimposed semi-transparently on the video images. We present a systematic evaluation by means of a phantom study. An urological trainee and two experts conducted 53 punctures on kidney phantoms.

Results   

The trainee performed best with the proposed AR system in terms of puncturing time (mean: 99 s), whereas the experts performed best with fluoroscopy (mean: 59 s). iPad assistance lowered radiation exposure by a factor of 3 for the inexperienced physician and by a factor of 1.8 for the experts in comparison with fluoroscopy usage. We achieve a mean visualization accuracy of 2.5 mm.

Conclusions   

The proposed tablet computer-based AR system has proven helpful in assisting percutaneous interventions such as PCNL and shows benefits compared to other state-of-the-art assistance systems. A drawback of the system in its current state is the lack of depth information. Despite that, the simple integration into the clinical workflow highlights the potential impact of this approach to such interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Notes

  1. iKlip stand adapter, IK Multimedia Production, http://www.ikmultimedia.com/products/iklip/.

    Fig. 2
    figure 2

    The components of the navigation system: a the navigation server, b Apple iPad as mobile display, c tablet fixation on a flexible endoscope stand, d radio-dense navigation markers, optionally attached with color labels for a more robust localization

    Full size image

  2. Braun Melsungen AG, http://www.bbraun.de/.

  3. Rebeck patient markers, Fobeck GbR, http://fobeck.com/cms2/en.

  4. For a detailed specification, please refer to the manufacturer‘s website http://www.siemens.com.

References

  1. Romero V, Akpinar H, Assimos DG (2010) Kidney stones: a global picture of prevalence, incidence, and associated risk factors. Rev Urol 12(2–3):86–96

    Google Scholar 

  2. Skolarikos A, Alivizatos G, de la Rosette JJ (2005) Percutaneous nephrolithotomy and its legacy. Eur Urol 47(1):22–28

    Article  PubMed  CAS  Google Scholar 

  3. Michel MS, Trojan L, Rassweiler JJ (2007) Complications in percutaneous nephrolithotomy. Eur Urol 51(4):899–906

    Google Scholar 

  4. Andonian S, Scoffone C, Louie MK, Gross AJ, Grabe M, Daels F, Shah HN, De La Rosette J (2012) Does imaging modality used for percutaneous renal access make a difference? a matched case analysis. J Endourol 27(1):24–28

    Google Scholar 

  5. Nicolau S, Soler L, Mutter D, Marescaux J (2011) Augmented reality in laparoscopic surgical oncology. Surg Oncol 20(3): 189–201

    Google Scholar 

  6. Maier-Hein L, Tekbas A, Seitel A, Pianka F, Müller SA, Satzl S, Schawo S, Radeleff B, Tetzlaff R, Franz AM, Müller-Stich BP, Wolf I, Kauczor H-U, Schmied BM, Meinzer H-P (2008) In vivo accuracy assessment of a needle-based navigation system for CT-guided radiofrequency ablation of the liver. Med Phys 35(12):5385–5396

    Article  PubMed  Google Scholar 

  7. Fichtinger G, Deguet A, Fischer G, Iordachita I, Balogh E, Masamune K, Taylor RH, Fayad LM, de Oliveira M, Zinreich SJ (2005) Image overlay for CT-guided needle insertions. Comput Aided Surg 10(4):241–255

    PubMed  Google Scholar 

  8. Song DY, Burdette EC, Fiene J, Armour E, Kronreif G, Deguet A, Zhang Z, Iordachita I, Fichtinger G, Kazanzides P (2011) Robotic needle guide for prostate brachytherapy: clinical testing of feasibility and performance. Brachytherapy 10(1):57–63

    Article  PubMed  Google Scholar 

  9. Wein W, Brunke S, Khamene A, Callstrom MR, Navab N (2008) Automatic CT-ultrasound registration for diagnostic imaging and image-guided intervention. Med Image Anal 12(5):577–585

    Article  PubMed  Google Scholar 

  10. Wood BJ, Kruecker J, Abi-Jaoudeh N, Locklin JK, Levy E, Xu S, Solbiati L, Kapoor A, Amalou H, Venkatesan AM (2010) Navigation systems for ablation. J Vasc Interv Radiol 21(8 Suppl):257–263

    Article  Google Scholar 

  11. Lazarus J, Williams J (2011) The locator: novel percutaneous nephrolithotomy apparatus to aid collecting system puncture-a preliminary report. J Endourol 25(5):747–750

    Article  PubMed  Google Scholar 

  12. Huber J, Wegner I, Meinzer HP, Hallscheidt P, Hadaschik B, Pahernik S, Hohenfellner M (2011) Navigated renal access using electromagnetic tracking: an initial experience. Surg Endosc 25(4):1307–1312

    Article  PubMed  Google Scholar 

  13. Ritter M, Rassweiler MC, Hacker A, Michel MS (2012) Laser-guided percutaneous kidney access with the Uro Dyna-CT: first experience of three-dimensional puncture planning with an ex vivo model. World J Urol. 30:1–5

    Google Scholar 

  14. Baumhauer M, Simpfendörfer T, Stich BM, Teber D, Gutt C, Rassweiler J, Meinzer H-P, Wolf I (2008) Soft tissue navigation for laparoscopic partial nephrectomy. Int J Comput Assist Radiol Surg 3:307–314

    Article  Google Scholar 

  15. Maier-Hein L, Franz AM, Fangerau M, Schmidt M, Seitel A, Mersmann S, Kilgus T, Groch A, Yung K, dos Santos TR, Meinzer H-P (2011) Towards mobile augmented reality for on-patient visualization of medical images. In: Handels H, Ehrhardt J, Deserno TM, Meinzer H-P, Tolxdorff T (eds) Bildverarbeitung für die Medizin. Springer, Berlin, pp 389–393

  16. Rassweiler JJ, Müller M, Fangerau M, Klein J, Goezen AS, Pereira P, Meinzer HP, Teber D (2012) iPad-assisted percutaneous access to the kidney using marker-based navigation: initial clinical experience. Eur Urol 61(3):628–631

    Article  PubMed  Google Scholar 

  17. Wolf I, Vetter M, Wegner I, Böttger T, Nolden M, Schöbinger M, Hastenteufel M, Kunert T, Meinzer H-P (2005) The medical imaging interaction toolkit. Med Image Anal 9(6):594–604

    Google Scholar 

  18. Fangerau M (2011) Medical imaging interaction toolkit for mobile devices. Master’s thesis, Hochschule Mannheim, University of Applied Sciences

  19. Müller M, Groch A, Baumhauer M, Maier-Hein L, Teber D, Rassweiler J, Meinzer H-P, Wegner I (2012) Robust and efficient fiducial tracking for augmented reality in HD-laparoscopic video streams. In: DRH III, Wong KH, (eds) SPIE medical imaging 2012: visualization, image-guided procedures, and modeling, vol 8316, No 1. SPIE, p 83161M

  20. Zhang Z (2000) A flexible new technique for camera calibration. IEEE T Pattern Anal 22:1330–1334

    Article  Google Scholar 

  21. Lowe DG (1991) Fitting parameterized three-dimensional models to images. IEEE Trans Pattern Anal Mach Intell 13:441–450

    Article  Google Scholar 

  22. DeMenthon DF, Davis LS (1995) Model-based object pose in 25 lines of code. Int J Comput Vis 15:123–141

    Article  Google Scholar 

  23. Lu C-P, Hager GD, Mjolsness E (2000) Fast and globally convergent pose estimation from video images. IEEE Trans Pattern Anal Mach Intell 22:610–622

    Article  Google Scholar 

  24. Lepetit V, Moreno-Noguer F, Fua P (2008) EP\(n\)P: an accurate o(\(n\)) solution to the P\(n\)P problem. Int J Comput Vis 81(2):155–166

    Article  Google Scholar 

  25. Sarkis M, Diepold K (2012) Camera-pose estimation via projective Newton optimization on the manifold. IEEE Trans Image Process 21(4):1729–1741

    Article  PubMed  Google Scholar 

  26. Li S, Xu C, Xie M (2012) A robust o(n) solution to the perspective-n-point problem. IEEE Trans Pattern Anal Mach Intell. 34(7):1444–1450

    Article  Google Scholar 

  27. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24(6):381–395

    Article  Google Scholar 

  28. Nocedal J, Wright SJ (2006) Numerical optimization, 2nd edn. ser. Springer series in operations research and financial engineering. Springer, New York

  29. Holloway RL (1997) Registration error analysis for augmented reality. Presence Teleoper Virtual Environ 6(4):413–432

    Google Scholar 

  30. Baumhauer M (2008) Real-time compensation of organ motion for augmented reality in laparoscopic surgery. Ph.D. dissertation, Ruprecht-Karls University Heidelberg

  31. Simpfendörfer T, Baumhauer M, Müller M, Gutt CN, Meinzer HP, Rassweiler JJ, Guven S, Teber D (2011) Augmented reality visualization during laparoscopic radical prostatectomy. J Endourol 25:1841–1845

    Google Scholar 

  32. Laganière R (2011) OpenCV 2 computer vision application programming cookbook. Packt Publishing, Birmingham

  33. Miller NL, Lingeman JE (2007) Management of kidney stones. BMJ 334(7591):468–472

    Article  PubMed  Google Scholar 

  34. Ritter M, Siegel F, Krombach P, Martinschek A, Weiss C, Hacker A, Pelzer AE (2012) Influence of surgeon’s experience on fluoroscopy time during endourological interventions. World J Urol 31(1): 183–187

    Google Scholar 

  35. Seitel A, Maier-Hein L, Schawo S, Radeleff B, Müller SA, Pianka F, Schmied BM, Wolf I, Meinzer H-P (2007) In-vitro evaluation of different visualization approaches for computer assisted targeting in soft tissue. In: Lemke H, Inamura K, Doi K, Vannier M, Farman A (eds) International journal of Computer Assisted Radiology and Surgery. Berlin (Germany), pp 188–190, June 2007

  36. Karami H, Rezaei A, Mohammadhosseini M, Javanmard B, Mazloomfard M, Lotfi B (2010) Ultrasonography-guided percutaneous nephrolithotomy in the flank position versus fluoroscopy-guided percutaneous nephrolithotomy in the prone position: a comparative study. J Endourol 24(8):1357–1361

    Google Scholar 

  37. Agarwal M, Agrawal MS, Jaiswal A, Kumar D, Yadav H, Lavania P (2011) Safety and efficacy of ultrasonography as an adjunct to fluoroscopy for renal access in percutaneous nephrolithotomy (PCNL). BJU Int 108(8):1346–1349

    Article  PubMed  Google Scholar 

  38. Maier-Hein L, Walsh CJ, Seitel A, Hanumara NC, Shepard J-A, Franz AM, Pianka F, Müller SA, Schmied B, Slocum AH, Gupta R, Meinzer H-P (2009) Human vs. robot operator error in a needle-based navigation system for percutaneous liver interventions. In: SPIE medical imaging 2009: visualization, image-guided procedures, and modelling, vol 7261, p 72610Y (12 p)

  39. Franz A, März K, Hummel J, Birkfellner W, Bendl R, Delorme S, Schlemmer H-P, Meinzer H-P, Maier-Hein L (2012) Electromagnetic tracking for us-guided interventions: standardized assessment of a new compact field generator. Int J Comp Assist Radiol Surg 7:1–6

    Google Scholar 

  40. Yaniva Z, Wilson E, Lindisch D, Cleary K (2009) Electromagnetic tracking in the clinical environment. Med Phys 36(3):876–892

    Article  Google Scholar 

  41. Lee T, Hollerer T (2009) Multithreaded hybrid feature tracking for markerless augmented reality. IEEE Trans Vis Comput Graph 15(3):355–368

    Article  PubMed  Google Scholar 

  42. Grimm R, Bauer S, Sukkau J, Hornegger J, Greiner G (2012) Markerless estimation of patient orientation, posture and pose using range and pressure imaging. Int J Comput Assist Radiol Surg 1:1–9

    Google Scholar 

  43. Newcombe RA, Davison AJ, Izadi S, Kohli P, Hilliges O, Shotton J, Molyneaux D, Hodges S, Kim D, Fitzgibbon A (2011) Kinectfusion: real-time dense surface mapping and tracking. In: Mixed and augmented reality (ISMAR), 2011 10th IEEE international symposium on, pp. 127–136, Oct 2011

  44. Mersmann S, Gergel I, Seitel A, Gaa J, Wegner I, Meinzer H-P, Maier-Hein L (2011) Microsoft kinect controller as intra-operative imaging modality. Int J Comput Assist Radiol Surg 6(Suppl 1): 251–252

    Google Scholar 

  45. Seitel A, Engel M, Sommer CM, Radeleff BA, Essert-Villard C, Baegert C, Fangerau M, Fritzsche KH, Yung K, Meinzer H-P, Maier-Hein L (2011) Computer-assisted trajectory planning for percutaneous needle insertions. Med Phys 38(6):3246–3259

    Article  PubMed  Google Scholar 

Download references

Acknowledgments

The presented work was conducted within the setting of the “Research group 1126: Intelligent Surgery-Development of new computer-based methods for the future workplace in surgery” funded by the German Research Foundation (DFG). Furthermore, we want to thank the staff in the urological departments of our partner hospitals for the support during our experiments. The presented software was developed as part of the Medical Imaging Interaction Toolkit (MITK, http://www.mitk.org).

Conflict of Interest

None.

Author information

Authors and Affiliations

  1. Department of Medical and Biological Informatics, German Cancer Research Center (DKFZ), Im Neuenheimer Feld 280, 69120, Heidelberg, Germany

    Michael Müller, Alexander Seitel, Matthias Baumhauer, Hans-Peter Meinzer & Lena Maier-Hein

  2. Department of Urology, Mannheim University Hospital, Theodor-Kutzer-Ufer 1-3, 68167, Mannheim, Germany

    Marie-Claire Rassweiler

  3. Department of Urology, SLK-Kliniken Heilbronn, Am Gesundbrunnen 20, 74078, Heilbronn, Germany

    Jan Klein & Jens J. Rassweiler

  4. Institute of Medical Biometry and Informatics, University of Heidelberg, Im Neuenheimer Feld 305, 69120, Heidelberg, Germany

    Matthias Gondan

  5. Department of Urology, Heidelberg University Hospital, Im Neuenheimer Feld 400, 69120, Heidelberg, Germany

    Dogu Teber

Authors
  1. Michael Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marie-Claire Rassweiler
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jan Klein
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Alexander Seitel
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Matthias Gondan
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Matthias Baumhauer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Dogu Teber
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Jens J. Rassweiler
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Hans-Peter Meinzer
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lena Maier-Hein
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Müller.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mpeg 18092 KB)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Müller, M., Rassweiler, MC., Klein, J. et al. Mobile augmented reality for computer-assisted percutaneous nephrolithotomy. Int J CARS 8, 663–675 (2013). https://doi.org/10.1007/s11548-013-0828-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-013-0828-4

Keywords

Search

Navigation