Skip to main content

Advertisement

SpringerLink
Log in

Medical imaging and virtual reality: a personal perspective

  • Original Article
  • Published:
Virtual Reality Aims and scope Submit manuscript

Abstract

The evolution of medical imaging, and concomitantly virtual reality (VR) technology, especially over the past 2–3 decades, has significantly accelerated the use of multi-modality images and VR instrumentation in guiding medical procedures, including surgery. The imaging capabilities have not only increased in variety of modalities (CT, MRI, PET, ultrasound, etc.), but also in dimensions and resolution. It is becoming more common to talk about 3D, 4D and even 5D images produced by modern imaging modalities. However, a relatively unexploited potential and capability of this increase in multimodality, multidimensional image data is the synergistic fusion of these datasets into a unified form that describes more accurately and extensively the complex nature of human anatomy, physiology, biology and pathology. The assist in achieving this potential, through realistic simulation, training, rehearsal and delivery of surgery and other interventional procedures by use of VR technology, has been increasingly evident, particularly in education. This paper attempts an overview of this potential, describing the evolution of medical imaging systems and VR that has lead to development of powerful computational techniques to fuse, visualize, analyze and use these images for advanced use in medical practice. This overview is based primarily on the author’s experience, opinion and perspective, explaining the preponderance of citations to his own work. A brief history of medical imaging and VR, a description of current imaging systems, and a summary of important image processing methods used in image-guided interventions will be given. Examples of use of these methods on several types of multidimensional image datasets will be illustrated, and several examples of real clinical applications described using 3D, 4D and 5D fused image datasets and VR technology for image-guided interventions, image-guided surgery, and image-guided therapy. Finally, the paper will discuss some barriers to progress and provide some prognostic views on the promising future of image-guided medical procedures and surgical interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30
Fig. 31
Fig. 32
Fig. 33
Fig. 34
Fig. 35
Fig. 36
Fig. 37
Fig. 38

Similar content being viewed by others

References

  • Akay MA, Marsh A (2001) Information technologies in medicine, vol 1. Wiley, New York

  • Augustine KE, Huddleston PM, Holmes III DR, Shridharanni SM, Robb RA (2008) Optimization of spine surgery planning with 3D image templating tools. Proc. SPIE–Medical Imaging 2008, San Diego, CA, 16–21 February

  • Brinkmann BJ, Robb RA, O’Brien TJ, Sharbough FW (1997) Localization, correlation, and visualization of electroencephalographic surface electrodes and brain anatomy in epilepsy studies. Proc. SPIE–The International Society for Optical Engineering 3033:159–169

  • Brinkmann BH, Robb RA, O’Brien TJ, O’Connor MK, Mullan BP (1998) Multimodality imaging for epilepsy diagnosis and surgical focus localization: three-dimensional image correlation and dual isotope SPECT. Proc. of medical image computing and computer-assisted intervention–MICCAI 1998, Cambridge, MA, 1496:1087–1098

  • Brinkmann BH, O’Brien TJ, Aharon S, O’Connor MK, Mullan BP, Hanson DP, Robb RA (1999) Quantitative and clinical analysis of SPECT image registration for epilepsy studies. J Nucl Med 40(7):1098–1105

    Google Scholar 

  • Burdea G, Coiffet P (1994) Virtual reality technology. Wiley, New York

    Google Scholar 

  • Cameron BM, Robb RA (2004) Patient-specific dynamic geometric models from sequential volumetric time series image data. Proceedings of medicine meets virtual reality 12. In: Westwood JD, Haluck RS, Hoffman HM, Mogel GT, Phillips R, Robb RA (eds) IOS Press, Amsterdam, Netherlands, 98:40–45

  • Cameron BM, Robb RA (2006) Virtual-reality-assisted interventional procedures. Clin Orthop 442:63–73

    Google Scholar 

  • Cameron BM, Holmes III, DR, Rettmann ME, Robb RA (2008) Patient specific physical anatomy models. Proc. medicine meets virtual reality 16—parallel, combinatorial, convergent: nextmed by design, In: Westwood JD, Haluck RS, Hoffman HM, Mogel GT, Phillips R, Robb RA, Vosburgh KG (eds) IOS Press, Amsterdam, Netherlands, 132:68–73

  • Camp J, Robb RA (1999) Realistic visualization for surgery simulation using dynamic volume texture mapping and model deformation. Proc. SPIE–The International Society for Optical Engineering, 3661:24–31

  • IEEE computer graphics and applications: virtual reality. November/December 2001

  • Haider CR, Bartholmai BJ, Holmes III DR, Camp JJ, Robb RA (2005) Quantitative characterization of lung disease. Comput Med Imaging Graph 29(7):555–563

    Article  Google Scholar 

  • Harris SS, Robb RA (2003) Piecewise registration for point-to-surface mapping of cardiac data. Proc. SPIE–Medical Imaging 2003, physiology and function: methods, systems and applications, 5031:146–153.

  • Heffernan PB, Robb RA (1985) A new method for shaded surface display of biological and medical images. IEEE Transactions on Medical Imaging MI-4:26–38

    Google Scholar 

  • Holmes III DR, Rettmann M, Cameron B, Camp J, Robb RA (2008) Developing patient-specific anatomic models for validation of cardiac ablation guidance procedures. Proc. SPIE–Medical Imaging 2008, San Diego, California, February 16–21

  • Holmes III DR, Robb RA (2000) Trans-urethral ultrasound (TUUS) imaging for visualization and analysis of the prostate and associated tissues. Proc. SPIE–medical imaging 2000: image display and visualization 3976:22–27

  • Khurana VG, Cameron BM, Bates LM, Robb RA (1999a) Virtual frontiers, part 1: fundamental concepts and recent advances in virtual reality technology. In: Fisher III WS (ed) Perspectives in neurological surgery. Thieme Publishing, New York 10(2):113–127

  • Khurana VG, Bates LM, Meyer FB, Robb RA (1999b) Virtual frontiers, part 2: role of virtual reality technology in neurosurgery. In: Fisher III WS (ed) Perspectives in neurological surgery. Thieme Publishing, New York 10(2):113–127

  • Lepard KO, Robb RA (1996) Shape-based segmentation and characterization of biomedical images. Proc. SPIE–the international society for optical engineering, vol 2710

  • Lin W, Robb RA (2000) A 5-D model for accurate representation and visualization of dynamic cardiac structures. Proc. SPIE–Biomedical Diagnostic, Guidance, and Surgical Assist Systems II, 3911:322–329

  • Melton LJIII, Riggs BL, Keaveny TM, Achenbach SJ, Hoffman PF, Camp JJ, Rouleau PA, Bouxsein ML, Amin S, Atkinson EJ, Robb RA, Khosla S (2007) Structural determinants of vertebral fracture risk. J Bone Miner Res 22(12):1885–1892

    Article  Google Scholar 

  • Packer DL, Asirvatham S, Seward JB, Breen JF, Robb RA (2004) Imaging of the cardiac and thoracic veins. Chap. 8. In: Chen, Haissaguerre, Zipes (eds) Thoracic vein arrhythmias: mechanisms and treatments. Prepress Projects, Ltd., Scotland

  • Rajagopalan S, Robb RA (2006) Fourier-domain based datacentric performance ranking of competing medical image processing algorithms. Proc. SPIE–medical imaging 2006, San Diego, CA, 11–16 February

  • Rettmann ME, Holmes III DR, Su Y, Cameron BM, Packer DL, Robb RA (2006) An integrated system for real-time image guided cardiac catheter ablation. Proc. medicine meets virtual reality 14. In:Westwood JD, Haluck RS, Hoffmann HM, Mogel GT, Phillips R, Robb RA, Vosburgh KG (eds) IOS Press, Amsterdam, Netherlands, 119:455–460

  • Rettmann ME, Holmes III DR, Camp JJ, Packer DL, Robb RA (2008) Validation of semi-automatic segmentation of the left atrium. Proc. SPIE–medical imaging 2008, San Diego, CA, 16–21 February

  • Ritman EL, Robb RA, Johnson SA, Chevalier PA, Gilbert BK, Greenleaf JF, Sturm RE, Wood EH (1978) Quantitative imaging of the structure and function of the heart, lungs, and circulation. Mayo Clin Proc 53:3–11

    Google Scholar 

  • Ritman EL, Kinsey JH, Robb RA, Gilbert BK, Harris LD, Wood EH (1980) Three-dimensional imaging of heart, lungs, and circulation. Science 210:273–280

    Article  Google Scholar 

  • Robb RA (1971) Computer-aided contour determination and dynamic display of individual cardiac chambers from digitized serial angiocardiographic film. In: Heintzen PH (ed) Roentgen-, cine-, and videodensitometry. Fundamentals and application for blood flow and heart volume determination. Georg Thieme Verlag, Stuttgart, pp 170–178

  • Robb RA (1982a) X-ray computed tomography: from basic principles to applications. Annu Rev Biophys Bioeng 11:177–201

    Article  Google Scholar 

  • Robb RA (1982b) The dynamic spatial reconstructor: an X-ray video-fluoroscopic CT scanner for dynamic volume imaging of moving organs. IEEE Transactions on Medical Imaging, MI-1(1):22–23

    Google Scholar 

  • Robb RA (1995) Three-dimensional biomedical imaging—principles and practice. VCH Publishers, New York

    Google Scholar 

  • Robb RA (1999a) Biomedical imaging, visualization and analysis. Wiley, New York

    Google Scholar 

  • Robb RA (1999b) Visualization in biomedical computing. Parallel Comput 25:2067–2110

    Article  Google Scholar 

  • Robb RA (2000) Virtual endoscopy: development and evaluation using the visible human datasets. Comput Med Imaging Graph 24:133–151

    Article  Google Scholar 

  • Robb RA (2001a) The biomedical imaging resource at mayo clinic. IEEE TMI 20(9):854–867

    Google Scholar 

  • Robb RA (2001b) Virtual reality in medicine and biology. In: Akay M, Marsh A (eds) Information technologies in medicine: medical simulation and education. Wiley, New York, vol 1, Chap. 1, pp 3–31

  • Robb RA (2002a) The virtualization of medicine: a decade of pitfalls and progress. Medicine meets virtual reality 02/10. In: Westwood JD, Hoffman HM, Robb RA, Stredney D (eds) IOS Press Amsterdam, Netherlands, vol 85, pp 31–37

  • Robb RA (2002b) Virtual reality in medicine: a personal perspective. J Visualization 5(4):317–326

    Article  Google Scholar 

  • Robb RA (2005) Image processing for anatomy and physiology: fusion of form and function. Proceedings of sixth annual national forum on biomedical imaging and oncology, NCI/NEMA/DCTD, Bethesda, MD, 7–8 April

  • Robb RA, Barillot C (1989) Interactive display and analysis of 3-D medical images. IEEE Transactions on Medical Imaging 8(3):217–226

    Article  Google Scholar 

  • Robb RA, Hanson DP (1995) The ANALYZETM software system for visualization and analysis in surgery simulation. In: Lavallée S, Taylor R, Burdea G, Mösges R (eds) Computer integrated surgery. MIT Press, Cambridge, pp 175–190

  • Robb RA, Ritman EL (1979) High-speed synchronous volume computed tomography of the heart. Radiology 133:655–661

    Google Scholar 

  • Robb RA, Greenleaf JF, Ritman EL, Johnson SA, Sjostrand JD, Herman GT, Wood EH (1974) Three-dimensional visualization of the intact thorax and contents: a technique for cross-sectional reconstruction from multiplanar X-ray views. Comput Biomed Res 7:395–419

    Article  Google Scholar 

  • Robb RA, Ritman EL, Gilbert BK, Kinsey JH, Harris LD, Wood EH (1979) The DSR: a high-speed three-dimensional X-ray computed tomography system for dynamic spatial reconstruction of the heart and circulation. IEEE Transactions on Nuclear Science NS-26(2):2713–2717

    Article  Google Scholar 

  • Robb RA, Hanson DP, Camp JJ (1996) Computer-aided surgery planning and rehearsal at Mayo Clinic. Computer 29(1):39–47

    Article  Google Scholar 

  • Robb RA, Cameron BM, Aharon S (1997) Efficient shape-based algorithms for modeling patient specific anatomy from 3D medical images: applications in virtual endoscopy and surgery. Proceedings of shape modeling and applications, pp 97–108, Aizu-Wakamatsu, Japan, 3–6 March

  • Satava R (1998) Cybersurgery: advanced technologies for surgical practice. Wiley, New York

    Google Scholar 

  • Satava RM, Robb RA (1997) Virtual endoscopy: applications of 3-D visualization to medical diagnosis. PRESENCE 6(2):179–197

    Google Scholar 

  • Su Y, Davis BJ, Furutani KM, Herman MG, Robb RA (2007) Seed localization and TRUS-fluoroscopy fusion for intraoperative prostate brachytherapy dosimetry. Comput Aided Surg 12(1):25–34

    Article  Google Scholar 

  • Wu QR, Bourland JD, Robb RA (1996) Morphology guided radiotherapy treatment planning and optimization. Proc. SPIE–The International Society for Optical Engineering, 2707:180–189

  • Zavaletta VA, Bartholmai BJ, Robb RA (2007) High resolution multi-detector CT aided tissue analysis and quantification of lung fibrosis. Acad Radiol 14:772–787

    Article  Google Scholar 

Download references

Acknowledgments

The author is grateful for the many talents and contributions of his colleagues in the Mayo Biomedical Imaging Resource, with whom he has been privileged to work for many years, and for the expertise and interests of many physicians, surgeons and healthcare practitioners at Mayo Clinic and other institutions with whom he has been fortunate to collaborate during his entire professional career. Portions of the reported work have been funded by NIH grants: NIBIB EB0283, NIDDK DK68055, NHLBI HR46158, NCI CA107933, NIAMS AR27065, and NIAMS AR 056212.

Author information

Authors and Affiliations

  1. Mayo Clinic College of Medicine, Rochester, MN, USA

    Richard A. Robb

Authors
  1. Richard A. Robb
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Richard A. Robb.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Robb, R.A. Medical imaging and virtual reality: a personal perspective. Virtual Reality 12, 235–257 (2008). https://doi.org/10.1007/s10055-008-0104-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10055-008-0104-z

Keywords

Search

Navigation