Skip to main content
SpringerLink
Log in

Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking, and System Evaluation

  • Published:
International Journal of Computer Vision Aims and scope Submit manuscript

Abstract

Augmented Reality is an emerging technology that seeks to enhance a user’s view by overlaying graphical information. We developed a prototype AR system geared for medical applications. It is built around a stereoscopic head-mounted display of the video-see-through variety. The newest generation of this prototype system exhibits high performance on a standard PC platform. Stereoscopic video images are augmented with medical graphics in real-time at 30 frames per second and with XGA (1024× 768) resolution. The system provides a compelling AR perception: the graphics appears firmly anchored in the scene—there is no time lag between video and graphics or any apparent jitter of the graphics. With the head-mounted display, the user has a natural and direct access to understanding the 3D structure of the scene, based on both stereo and kinetic depth cues. In the present paper, we describe in detail the architecture and several features of the AR prototype system. Head tracking is accomplished with a single-camera system, with the dedicated tracker camera placed on the head-mounted display. This configuration is the foundation of achieving a high-accuracy graphics overlay. We are now exploring the use of the prototype system for a variety of medical applications. This paper gives an overview over the pre-clinical tests that we have performed for interventional guidance. Overall, the feedback has been very positive and encouraging, and we are continuing to work towards realizing the clinical potential of the technology.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Bajura, M., Fuchs, H., and Ohbuchi, R. 1992. Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. ACM SIGGRAPH Computer Graphics, 26(2):203–210.

    Article  Google Scholar 

  • State, A., Livingston, M.A., Hirota, G., Garrett, W.F., Whitton, M.C., Fuchs, H., and Pisano, E.D. 1996. Technologies for augmented-reality systems: realizing ultrasound-guided needle biopsies. in SIGGRAPH 96 Computer Graphics Proceedings, Annual Conference Series, pp. 439–446.

  • Rosenthal, M., State, A., Lee, J., Hirota, G., Ackerman, J., Keller, K., Pisano, E.D., Jiroutek, M., Muller, K., and Fuchs, H. 2002. Augmented reality guidance for needle biopsies: An initial randomized, controlled trial in phantoms. Medical Image Analysis, 6(2):313–320.

    Article  Google Scholar 

  • Fuchs, H., Livingston, M.A., Raskar, R., Colucci, D., Keller, K., State, A., Crawford, J.R., Rademacher, P., Drake, S.H., and Meyer, A.A. 1998. Augmented reality visualization for laparoscopic surgery. in Proceedings of the 1st International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 934–943.

  • Grimson, W.E.L., Kikinis, R., Jolesz, F.A., and Black, P.M. 1999. Image-guided surgery. Scientific American, 280(6):62–69.

    Article  Google Scholar 

  • Edwards, P.J., Hawkes, D.J., Hill, D.L.G., Jewell, D., Spink, R., Strong, A.J., and Gleeson, M.J., 1995. Augmentation of reality in the stereo operating microscope for otolaryngology and neurosurgical guidance. Computer Assisted Surgery, 1(3):172–178.

    Google Scholar 

  • King, A.P., Edwards, P., Maurer Jr., C.R., de Cunha, D.A., Gaston, R.P., Clarkson, M., Hill, D.L.G., Hawkes, D.J., Fenlon, M.R., Strong, A.J., Cox, T.C.S., and Gleeson, M.J. 2000. Stereo augmented reality in the surgical microscope. Presence: Teleoperators and Virtual Environments, 9(4):360–368.

    Article  Google Scholar 

  • Edwards, P.J., King, A.P., Maurer, Jr., C.R., de Cunha, D.A., Hawkes, D.J., Hill, D.L.G., Gaston, R.P., Fenlon, M.R., Jusczyzck, A., Strong, A.J., Chandler, C.L., and Gleeson, M.J. 2000. Design and evaluation of a system for microscope-assisted guided interventions (MAGI). IEEE Transactions on Medical Imaging, 19(11):1082–1093.

    Article  Google Scholar 

  • Birkfellner, W., Huber, K., Watzinger, F., Figl, M., Wanschitz, F., Hanel, R., Rafolt, D., Ewers, R., and Bergmann, H. 2000. Development of the Varioscope AR, a see-through HMD for computer-aided surgery. in Proceedings of the IEEE and ACM International Symposium on Augmented Reality (ISAR), pp. 54–59.

  • Rolland, J. and Fuchs, H. 2000. Optical versus video see-through head-mounted displays in medical visualization. Presence: Teleoperators and Virtual Environments, 9(3):287–309.

    Article  Google Scholar 

  • Azuma, R.T. 1997. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):355–385.

    Google Scholar 

  • Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., and MacIntyre, B. 2001. Recent advances in augmented reality. IEEE Computer Graphics and Applications, 21(6):34–47.

    Article  Google Scholar 

  • Sauer, F., Wenzel, F., Vogt, S., Tao, Y., Genc, Y., and Bani-Hashemi, A. 2000. Augmented workspace: Designing an AR testbed. in Proceedings of the IEEE and ACM International Symposium on Augmented Reality (ISAR), pp. 47–53.

  • Vogt, S., Khamene, A., Sauer, F., and Niemann, H. 2002. Single camera tracking of marker clusters: Multiparameter cluster optimization and experimental verification. in Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), pp. 127–136.

  • Sauer, F., Khamene, A., and Vogt, S. 2002. An augmented reality navigation system with a single-camera tracker: System design and needle biopsy phantom trial. in Proceedings of the 5th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 116–124.

  • Wang, J., Chi, V., and Fuchs, H. 1990. A real-time optical 3D tracker for head-mounted display systems. ACM SIGGRAPH Computer Graphics, 24(2):205–215.

    Article  Google Scholar 

  • Hoff, W.A. 1998. Fusion of data from head-mounted and fixed sensors. in Proceedings of the 1st International Workshop on Augmented Reality (IWAR), pp. 167–182.

  • Haralick, R. and Shapiro, L. 1993. Computer and Robot Vision. Reading, MA: Adison-Wesley

    Google Scholar 

  • Tsai, R.Y. 1987. A versatile camera calibration technique for high accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE Journal of Robotics and Automation, 3(4):323–344.

    Article  Google Scholar 

  • Haralick, R.M., Lee, C., Ottenberg, K., and Nolle, M. 1991. Analysis and solutions of the three point perspective pose estimation problem. in Proceedings of the IEEE Computer Vision and Pattern Recognition Conference (CVPR), pp. 592–598.

  • Quan, L. and Lan, Z. 1999. Linear n-point camera pose determination. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8):774–780.

    Article  Google Scholar 

  • Sauer, F., Schoepf, U.J., Khamene, A., Vogt, S., Das, M., and Silverman, S.G. 2003. Augmented reality system for CT-guided interventions: System description and initial phantom trials. in Proceedings of SPIE’s Conference of Medical Imaging 2003: Visualization, Image-Guided Procedures, and Display, vol. 5029, pp. 384–394.

  • Vogt, S., Wacker, F., Khamene, A., Elgort, D.R., Sielhorst, T., Niemann, H., Duerk, J., Lewin, J., and Sauer, F. 2004. Augmented reality system for MR-guided interventions: Phantom studies and first animal test. in Proceedings of SPIE’s Conference of Medical Imaging 2004: Visualization, Image-Guided Procedures, and Display, vol. 5367, pp. 100–109.

  • Khamene, A., Vogt, S., Azar, F., Sielhorst, T., Sauer, F., and Niemann, H. 2003. Local 3D reconstruction and augmented reality visualization of free-hand ultrasound for needle biopsy procedures. in Proceedings of the 6th International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), ser. Lecture Notes in Computer Science, Springer-Verlag, vol. 2879, pp. 344–355.

  • Vogt, S., Khamene, A., Niemann, H., and Sauer, F. 2004. An AR system with intuitive user interface for manipulation and visualization of 3D medical data. in Proceedings of the 12th Annual Medicine Meets Virtual Reality Conference (MMVR), pp. 397–403.

  • Rubino, G.J., Farahani, K., McGill, D., Van de Wiele, B., Villablanca, J.P., and Wang-Mathieson, A. 2000. Magnetic resonance imaging-guided neurosurgery in the magnetic fringe fields: the next step in neuronavigation. Neurosurgery, 46(3):643–653.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Imaging & Visualization Department, Siemens Corporate Research, 755 College Road East, Princeton, NJ, 08540, USA

    Sebastian Vogt, Ali Khamene & Frank Sauer

Authors
  1. Sebastian Vogt
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ali Khamene
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frank Sauer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sebastian Vogt.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Vogt, S., Khamene, A. & Sauer, F. Reality Augmentation for Medical Procedures: System Architecture, Single Camera Marker Tracking, and System Evaluation. Int J Comput Vision 70, 179–190 (2006). https://doi.org/10.1007/s11263-006-7938-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11263-006-7938-1

Keywords

Search

Navigation