Skip to main content
SpringerLink
Log in

Mixed reality ultrasound guidance system: a case study in system development and a cautionary tale

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Real-time ultrasound has become a crucial aspect of several image-guided interventions. One of the main constraints of such an approach is the difficulty in interpretability of the limited field of view of the image, a problem that has recently been addressed using mixed reality, such as augmented reality and augmented virtuality. The growing popularity and maturity of mixed reality has led to a series of informal guidelines to direct development of new systems and to facilitate regulatory approval. However, the goals of mixed reality image guidance systems and the guidelines for their development have not been thoroughly discussed. The purpose of this paper is to identify and critically examine development guidelines in the context of a mixed reality ultrasound guidance system through a case study.

Methods

A mixed reality ultrasound guidance system tailored to central line insertions was developed in close collaboration with an expert user. This system outperformed ultrasound-only guidance in a novice user study and has obtained clearance for clinical use in humans. A phantom study with 25 experienced physicians was carried out to compare the performance of the mixed reality ultrasound system against conventional ultrasound-only guidance. Despite the previous promising results, there was no statistically significant difference between the two systems.

Results

Guidelines for developing mixed reality image guidance systems cannot be applied indiscriminately. Each design decision, no matter how well justified, should be the subject of scientific and technical investigation. Iterative and small-scale evaluation can readily unearth issues and previously unknown or implicit system requirements.

Conclusions

We recommend a wary eye in development of mixed reality ultrasound image guidance systems emphasizing small-scale iterative evaluation alongside system development. Ultimately, we recommend that the image-guided intervention community furthers and deepens this discussion into best practices in developing image-guided interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Abhari K, Baxter JS, Chen ECS, Khan AR, Peters TM, de Ribaupierre S, Eagleson R (2015) Training for planning tumour resection: augmented reality and human factors. IEEE Trans Biomed Eng 62(6):1466–1477

    Article  PubMed  Google Scholar 

  2. Akoglu H, Piskinpasa S, Yenigun EC, Ozturk R, Dede F, Odabas AR (2012) Real-time ultrasound guided placement of temporary internal jugular vein catheters: assessment of technical success and complication rates in nephrology practice. Nephrology 17(7):603–606

    Article  PubMed  Google Scholar 

  3. Ameri G, Baxter JS, McLeod AJ, Peters TM, Chen ECS (2015) Augmented reality ultrasound guidance for central line procedures: preliminary results. In: Workshop on augmented environments for computer-assisted interventions, Springer, pp 11–20

  4. Bannon MP, Heller SF, Rivera M (2011) Anatomic considerations for central venous cannulation. Risk Manag Healthc Policy 4:27

    Article  PubMed  PubMed Central  Google Scholar 

  5. Baumann M, Mozer P, Daanen V, Troccaz J (2009) Prostate biopsy assistance system with gland deformation estimation for enhanced precision. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 67–74

  6. Bax J, Cool D, Gardi L, Knight K, Smith D, Montreuil J, Sherebrin S, Romagnoli C, Fenster A (2008) Mechanically assisted 3D ultrasound guided prostate biopsy system. Med Phys 35(12):5397–5410

    Article  PubMed  Google Scholar 

  7. Bichlmeier C, Wimmer F, Heining SM, Navab N (2007) Contextual anatomic mimesis hybrid in-situ visualization method for improving multi-sensory depth perception in medical augmented reality. In: 6th IEEE and ACM international symposium on mixed and augmented reality, ISMAR 2007, IEEE, pp 129–138

  8. Blaivas M, Adhikari S (2009) An unseen danger: frequency of posterior vessel wall penetration by needles during attempts to place internal jugular vein central catheters using ultrasound guidance. Crit Care Med 37(8):2345–2349

    Article  PubMed  Google Scholar 

  9. Bowdle A (2014) Vascular complications of central venous catheter placement: evidence-based methods for prevention and treatment. J Cardiothorac Vasc Anesth 28(2):358–368

    Article  PubMed  Google Scholar 

  10. Cavanna L, Civardi G, Mordenti P, Vallisa D, Bert R, Di Nunzio C (2013) Central venous catheter care for the patients with cancer: ultrasound-guided insertion should be strongly recommended for internal jugular vein catheterization. Ann Oncol 24(11):2928–2929

    Article  CAS  PubMed  Google Scholar 

  11. Chandler WF, Knake JE, McGillicuddy JE, Lillehei KO, Silver TM (1982) Intraoperative use of real-time ultrasonography in neurosurgery. J Neurosurg 57(2):157–163

    Article  CAS  PubMed  Google Scholar 

  12. Chen ECS, Peters TM, Ma B (2016) Guided ultrasound calibration: where, how, and how many calibration fiducials. Int J Comput Assist Radiol Surg 11(6):889–898

    Article  CAS  PubMed  Google Scholar 

  13. Commission IE (2006) Medical device software–software life cycle processes. IEC Standard 62304

  14. Currie M, McLeod J, Patel R, Peters T, Kiaii B (2015) Augmented reality system for ultrasound guidance of transcatheter aortic valve replacement. Can J Cardiol 31(10):S182

    Article  Google Scholar 

  15. Denys BG, Uretsky BF (1991) Anatomical variations of internal jugular vein location: impact on central venous access. Crit Care Med 19(12):1516–1519

    Article  CAS  PubMed  Google Scholar 

  16. Ender J, Končar-Zeh J, Mukherjee C, Jacobs S, Borger MA, Viola C, Gessat M, Fassl J, Mohr FW, Falk V (2008) Value of augmented reality-enhanced transesophageal echocardiography (TEE) for determining optimal annuloplasty ring size during mitral valve repair. Ann Thorac Surg 86(5):1473–1478

    Article  PubMed  Google Scholar 

  17. Horn BK (1987) Closed-form solution of absolute orientation using unit quaternions. J Opt Soc Am 4:629–642

    Article  Google Scholar 

  18. Hung J, Lang R, Flachskampf F, Shernan SK, McCulloch ML, Adams DB, Thomas J, Vannan M, Ryan T (2007) 3d echocardiography: a review of the current status and future directions. J Am Soc Echocardiogr 20(3):213–233

    Article  PubMed  Google Scholar 

  19. Jakola AS, Reinertsen I, Selbekk T, Solheim O, Lindseth F, Gulati S, Unsgård G (2014) Three-dimensional ultrasound-guided placement of ventricular catheters. World Neurosurg 82(3):536-e5

    Article  Google Scholar 

  20. Jenssen C, Brkljacic B, Hocke M, Ignee A, Piscaglia F, Radzina M, Sidhu PS, Dietrich CF (2015) EFSUMB guidelines on interventional ultrasound (INVUS), part vi. Ultraschall Med Eur J Ultrasound 36:E1–E14

    Google Scholar 

  21. Khaykin Y, Klemm O, Verma A (2007) First human experience with real-time integration of intracardiac echocardiography and 3D electroanatomical imaging to guide right free wall accessory pathway ablation. Europace 10(1):116–117

    Article  PubMed  Google Scholar 

  22. Linte CA, Davenport KP, Cleary K, Peters C, Vosburgh KG, Navab N, Jannin P, Peters TM, Holmes DR, Robb RA (2013) On mixed reality environments for minimally invasive therapy guidance: systems architecture, successes and challenges in their implementation from laboratory to clinic. Comput Med Imaging Gr 37(2):83–97

    Article  Google Scholar 

  23. Linte CA, Moore J, Wiles AD, Wedlake C, Peters TM (2008) Virtual reality-enhanced ultrasound guidance: a novel technique for intracardiac interventions. Comput Aided Surg 13(2):82–94

    Article  PubMed  Google Scholar 

  24. Martin C, Eon B, Auffray JP, Saux P, Gouin F (1990) Axillary or internal jugular central venous catheterization. Crit Care Med 18(4):400–402

    Article  CAS  PubMed  Google Scholar 

  25. McGee DC, Gould MK (2003) Preventing complications of central venous catheterization. N Engl J Med 348(12):1123–1133

    Article  PubMed  Google Scholar 

  26. McHugh M, McCaffery F, Casey V (2012) Barriers to adopting agile practices when developing medical device software. In: International conference on software process improvement and capability determination, Springer, pp 141–147

  27. McHugh M, McCaffery F, Fitzgerald B, Stol KJ, Casey V, Coady G (2013) Balancing agility and discipline in a medical device software organisation. In: International conference on software process improvement and capability determination, Springer, pp 199–210

  28. Milgram P, Takemura H, Utsumi A, Kishino F (1994) Augmented reality: a class of displays on the reality-virtuality continuum. Telemanipulator Telepresence Technol 2351:282–292

    Article  Google Scholar 

  29. Moore CL, Copel JA (2011) Point-of-care ultrasonography. N Engl J Med 364(8):749–757

    Article  CAS  PubMed  Google Scholar 

  30. Moore JT, Chu MWA, Kiaii B, Bainbridge D, Guiraudon G, Wedlake C, Currie M, Rajchl M, Patel RV, Peters TM (2013) A navigation platform for guidance of beating heart transapical mitral valve repair. IEEE Trans Biomed Eng 60(4):1034–1040. doi:10.1109/TBME.2012.2222405

    Article  PubMed  Google Scholar 

  31. Otto CM (2012) The practice of clinical echocardiography. Elsevier Health Sciences, Amsterdam

    Google Scholar 

  32. Paul RG, Price S (2014) Central venous cannulation. Medicine 42:473–474

    Article  Google Scholar 

  33. Paulk MC (2011) On empirical research into scrum adoption. Carnegie Mellon University, Pittsburgh

    Google Scholar 

  34. Peters TM, Linte CA (2016) Image-guided interventions and computer-integrated therapy: Quo vadis? Med Image Anal 33:56–63

    Article  PubMed  Google Scholar 

  35. Pratola C, Baldo E, Artale P, Marcantoni L, Toselli T, Percoco G, Sassone B, Ferrari R (2011) Different image integration modalities to guide af ablation: impact on procedural and fluoroscopy times. Pacing Clin Electrophysiol 34(4):422–430

    Article  PubMed  Google Scholar 

  36. Rottier PA, Rodrigues V (2008) Agile development in a medical device company. In: Conference on AGILE’08, IEEE, pp 218–223

  37. Rust P, Flood D, McCaffery F (2016) Creation of an IEC 62304 compliant software development plan. J Softw Evolut Process 28(11):1005–1010

    Article  Google Scholar 

  38. Rust P, Flood D, McCaffery F (2016) Software process improvement roadmaps–using design patterns to aid SMEs developing medical device software in the implementation of IEC 62304. In: International conference on software process improvement and capability determination, Springer, pp 43–56

  39. Sauer F, Vogt S, Khamene A (2008) Augmented reality. In: Peters T, Cleary K (eds) Image-guided interventions. Springer, Boston, pp 81–119

  40. Schmidt GA, Maizel J, Slama M (2015) Ultrasound-guided central venous access: What’s new? Intensive Care Med 41(4):705–707

    Article  PubMed  Google Scholar 

  41. Selbekk T, Jakola AS, Solheim O, Johansen TF, Lindseth F, Reinertsen I, Unsgård G (2013) Ultrasound imaging in neurosurgery: approaches to minimize surgically induced image artefacts for improved resection control. Acta Neurochir 155(6):973–980

    Article  PubMed  PubMed Central  Google Scholar 

  42. Surry KJM, Austin HJB, Fenster A, Peters TM (2004) Poly(vinyl alcohol) cryogel phantoms for use in ultrasound and MR imaging. Phys Med Biol 49(24):5529–5546

    Article  CAS  PubMed  Google Scholar 

  43. Troianos CA, Kuwik RJ, Pasqual JR, Lim AJ, Odasso DP (1996) Internal jugular vein and carotid artery anatomic relation as determined by ultrasonography. Anesthesiology 85(1):43–48

  44. Unsgaard G, Rygh O, Selbekk T, Müller T, Kolstad F, Lindseth F, Hernes TN (2006) Intra-operative 3D ultrasound in neurosurgery. Acta Neurochir 148(3):235–253

    Article  CAS  PubMed  Google Scholar 

  45. Van Krevelen D, Poelman R (2010) A survey of augmented reality technologies, applications and limitations. Int J Virtual Real 9(2):1

    Google Scholar 

  46. Wei Z, Wan G, Gardi L, Mills G, Downey D, Fenster A (2004) Robot-assisted 3D-trus guided prostate brachytherapy: system integration and validation. Med Phys 31(3):539–548

    Article  PubMed  Google Scholar 

  47. Weiner MM, Geldard P, Mittnacht AJ (2013) Ultrasound-guided vascular access: a comprehensive review. J Cardiothorac Vasc Anesth 27(2):345–360

    Article  PubMed  Google Scholar 

  48. Yaniv Z, Linte CA (2016) Applications of augmented reality in the operating room. In: Barfield W (ed) Fundamentals of wearable computers and augmented reality. CRC Press, USA, pp 485–518

Download references

Acknowledgements

The authors would like to acknowledge Jonathan McLeod, John Moore, and Adam Rankin for their invaluable discussion. This research was funded by The Canadian Foundation for Innovation, # 20994, The Natural Sciences and Engineering Council of Canada, # RPGIN 2014-04504, and The Canadian Institutes for Health Research, # FDN 143232. John S. H. Baxter is supported by the Natural Sciences and Engineering Research Council of Canada through the Canadian Graduate Scholarship program.

Author information

Author notes

  1. Golafsoun Ameri and John S. H. Baxter have contributed equally to this work.

Authors and Affiliations

  1. Robarts Research Institute, Western University, London, ON, Canada

    Golafsoun Ameri, John S. H. Baxter, Terry M. Peters & Elvis C. S. Chen

  2. London Health Sciences Centre, London, ON, Canada

    Daniel Bainbridge

Authors
  1. Golafsoun Ameri
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. John S. H. Baxter
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Daniel Bainbridge
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Terry M. Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Elvis C. S. Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Golafsoun Ameri.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ameri, G., Baxter, J.S.H., Bainbridge, D. et al. Mixed reality ultrasound guidance system: a case study in system development and a cautionary tale. Int J CARS 13, 495–505 (2018). https://doi.org/10.1007/s11548-017-1665-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-017-1665-7

Keywords

Search

Navigation