Skip to main content

Advertisement

SpringerLink
Log in

A hybrid deformable registration method to generate motion-compensated 3D virtual MRI for fusion with interventional real-time 3D ultrasound

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Ultrasound is often the preferred modality for image-guided therapy or treatment in organs such as liver due to real-time imaging capabilities. However, the reduced conspicuity of tumors in ultrasound images adversely impacts the precision and accuracy of treatment delivery. This problem is compounded by deformable motion due to breathing and other physiological activity. This creates the need for a fusion method to align interventional US with pre-interventional modalities that provide superior soft-tissue contrast (e.g., MRI) to accurately target a structure-of-interest and compensate for liver motion.

Method

In this work, we developed a hybrid deformable fusion method to align 3D pre-interventional MRI and 3D interventional US volumes to target the structures-of-interest in liver accurately in real-time. The deformable multimodal fusion method involved an offline alignment of a pre-intervention MRI with a pre-intervention US volume using a traditional registration method, followed by real-time prediction of deformation using a trained deep-learning model between interventional US volumes across different respiratory states. This framework enables motion-compensated MRI-US image fusion in real-time for image-guided treatment.

Results

The proposed hybrid deformable registration method was evaluated on three healthy volunteers across the pre-intervention MRI and 20 US volume pairs in the free-breathing respiratory cycle. The mean Euclidean landmark distance of three homologous targets in all three volunteers was less than 3 mm for percutaneous liver procedures.

Conclusions

Preliminary results show that clinically acceptable registration accuracies for near real-time, deformable MRI-US fusion can be achieved by our proposed hybrid approach. The proposed combination of traditional and deep-learning deformable registration techniques is thus a promising approach for motion-compensated MRI-US fusion to improve targeting in image-guided liver interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Lee M (2014) Fusion imaging of real-time ultrasonography with CT or MRI for hepatic intervention. Ultrasonography 33(4):227–239. https://doi.org/10.14366/usg.14021

    Article  PubMed  PubMed Central  Google Scholar 

  2. Mauri G, Cova L, De Beni S, Ierace T, Tandolo T, Cerri A, Goldberg SN, Solbiati L (2015) Real-time US-CT/MRI image fusion for guidance of thermal ablation of liver tumors undetectable with US: results in 295 cases. Cardiovasc Intervent Radiol 38:143–151. https://doi.org/10.1007/s00270-014-0897-y

    Article  PubMed  Google Scholar 

  3. Wang S (2017) Real-time fusion imaging of liver ultrasound. J Med Ultrasound 25(1):9–11. https://doi.org/10.1016/j.jmu.2017.03.003

    Article  PubMed  PubMed Central  Google Scholar 

  4. European Society of Radiology (ESR) (2019) Abdominal applications of ultrasound fusion imaging technique: liver, kidney, and pancreas. Insights Imaging. https://doi.org/10.1186/s13244-019-0692-z

    Article  Google Scholar 

  5. World Health Organization (2020) Liver factsheet. Globocan. https://gco.iarc.fr/today/data/factsheets/cancers/11-Liver-fact-sheet.pdf

  6. Solbiati M, Muglia R, Goldberg S, Ierace T, Rotilio A, Passera KM, Marre I, Solbiati L (2019) A novel software platform for volumetric assessment of ablation completeness. Int J Hyperthermia 36:337–343

    Article  PubMed  Google Scholar 

  7. Biro P, Spahn DR, Pfammatter T (2009) High-frequency jet ventilation for minimizing breathing-related liver motion during percutaneous radiofrequency ablation of multiple hepatic tumours. Br J Anaesth 102:650–653

    Article  CAS  PubMed  Google Scholar 

  8. Holland AE, Goldfarb JW, Edelman RR (1998) Diaphragmatic and cardiac motion during suspended breathing: preliminary experience and implications for breath-hold mr imaging. Radiology 209:483–489

    Article  CAS  PubMed  Google Scholar 

  9. Machado I, Toews M, George E, Unadkat P, Essayed W, Luo J, Teodoro P, Carvalho H, Martins J, Golland P, Pieper S, Frisken S, Golby A, Wells W III, Ou Y (2019) Deformable MRI-Ultrasound registration using correlation-based attribute matching for brain shift correction: accuracy and generality in multi-site data. Neuroimage 202(116):094

    Google Scholar 

  10. Xiao Y, Rivaz H, Chabanas M, Fortin M, Machado I, Ou Y, Heinrich MP, Schnabel JA, Zhong X, Maier A, Wein W, Shams R, Kadoury S, Drobny D, Modat M, Reinertsen I (2020) Evaluation of mri to ultrasound registration methods for brain shift correction: the curious2018 challenge. IEEE Trans Med Imaging 39(3):777–786. https://doi.org/10.1109/TMI.2019.2935060

    Article  PubMed  Google Scholar 

  11. Hering A, Hansen L, Mok T, Chung A, Siebert H, Häger S, Lange A, Kuckertz S, Heldmann S, Shao E, Vesal S, Rusu M, Sonn G, Estienne T, Vakalopoulou M, Han L, Huang Y, Yap PT, Brudfors M, Balbastre Y et al (2020) Learn2Reg: comprehensive multi-task medical image registration challenge, dataset and evaluation in the era of deep learning. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2022.3213983

    Article  Google Scholar 

  12. Modat M, Ridgway GR, Taylor ZA, Lehmann M, Barnes J, Hawkes DJ, Fox NC, Ourselin S (2010) Fast free-form deformation using graphics processing units. Comput Methods Programs Biomed 98(3):278–284. https://doi.org/10.1016/j.cmpb.2009.09.002

    Article  PubMed  Google Scholar 

  13. Heinrich M, Handels H, Simpson I (2015) Estimating large lung motion in COPD patients by symmetric regularised correspondence fields. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp. 338–345

  14. Heinrich M (2018) Intra-operative Ultrasound to MRI fusion with a public multimodal discrete registration tool. In: International Workshops, POCUS 2018, BIVPCS 2018, CuRIOUS 2018, and CPM 2018, MICCAI. Springer, pp. 159–164

  15. Wein W (2018) Brain-shift correction with image-based registration and landmark accuracy evaluation. In: International Workshops, POCUS 2018, BIVPCS 2018, CuRIOUS 2018, and CPM 2018, MICCAI. Springer, pp. 146–151

  16. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV (2019) VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging 38(8):1788–1800. https://doi.org/10.1109/TMI.2019.2897538

    Article  Google Scholar 

  17. Heinrich MP (2019) Closing the gap between deep and conventional image registration using probabilistic dense displacement networks. In: MICCAI, pp. 50–58

  18. Mok T, Chung A (2020) Large deformation diffeomorphic image registration with laplacian pyramid networks. In: International conference on medical image computing and computer-assisted intervention. Springer (MICCAI), pp. 211–221

  19. Mok T, Chung A (2021) Conditional deformable image registration with convolutional neural network. In: International conference on medical image computing and computer-assisted intervention (MICCAI). Springer, pp. 35–45

  20. Bednarz B, Jupitz S, Lee W, Mills D, Chan H, Fiorillo T, Sabitini J, Shoudy D, Patel A, Mitra J, Sarcar S, Wang B, Shepard A, Matrosic C, Holmes J, Culberson W, Bassetti M, Hill P, McMillan A, Zagzebski J, Smith L, Foo T (2021) First-in-human imaging using a MR-compatible e4D ultrasound probe for motion management of radiotherapy. Phys Med 88:104–110. https://doi.org/10.1016/j.ejmp.2021.06.017

    Article  PubMed  PubMed Central  Google Scholar 

  21. Dixon WT (1984) Simple proton spectroscopic imaging. Radiology 153(1):189–194. https://doi.org/10.1148/radiology.153.1.6089263

    Article  CAS  PubMed  Google Scholar 

  22. Lee W, Chan H, Chan P, Fiorillo T, Fiveland E, Foo T, Mills D, Patel A, Sabatini J, Shoudy D, Smith S, Bednarz B (2017) A magnetic resonance compatible E4D ultrasound probe for motion management of radiation therapy. In: IEEE network 2017. https://doi.org/10.1109/ULTSYM.2017.8092223

  23. Mitra J, MacDonald M, Mills D, Ghose S, Smith LS, Sarcar S, Yeo DTB, Tempany C, Bednarz B, Jupitz S, Foo TK (2020) Patient-specific deep deformation models (PsDDM) to register planning and interventional ultrasound volumes in image fusion-guided interventions. In: Fei B, Linte CA (eds) Medical imaging 2020: image-guided procedures, robotic interventions, and modeling, International society for optics and photonics, vol 11315. SPIE, Bellingham, pp 239–244

    Google Scholar 

  24. Jadenberg M, Simonyan K, Zisserman A, Kavukcuoglu K (2015) Spatial transformer networks. In: 28th Intl Conf. on neural information processing systems, pp. 2017–2025

  25. Ronneberger O, Fischer P, Brox T (2015) U-Net: convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention (MICCAI), vol 9351. Springer, Berlin, pp 23–241

    Google Scholar 

  26. Ourselin S, Roche A, Subsol G, Pennec X, Ayache N (2001) Reconstructing a 3D structure from serial histological sections. In: Image and Vision Comput., 19(1–2):25–31

  27. Thirion JP (1998) Image matching as a diffusion process: an analogy with Maxwell’s demons. Med Imag Anal 2(3):243–260

    Article  CAS  Google Scholar 

  28. Monfardini L, Orsi F, Caserta R, Sallemi C, Vigna PD, Bonomo G, Varano G, Solbiati L, Mauri G (2018) Ultrasound and cone beam CT fusion for liver ablation: technical note. Int J Hyperth 35(1):500–504. https://doi.org/10.1080/02656736.2018.1509237

    Article  Google Scholar 

Download references

Funding

This project was funded in parts by NIH R01CA190298 and 1R01CA266879.

Author information

Author notes

  1. Jhimli Mitra, Chitresh Bhushan, Soumya Ghose authors contributed equally to this work.

Authors and Affiliations

  1. General Electric Research, Niskayuna, NY, USA

    Jhimli Mitra, Chitresh Bhushan, Soumya Ghose, David Mills, Aqsa Patel, Heather Chan, Matthew Tarasek, Thomas Foo & Desmond Yeo

  2. Department of Radiology, University of Wisconsin-Madison, Madison, WI, USA

    Shane Wells & Chris Brace

  3. Department of Medical Physics, University of Wisconsin-Madison, Madison, WI, USA

    Sydney Jupitz & Bryan Bednarz

  4. Department of Biomedical Engineering, University of Wisconsin-Madison, Madison, WI, USA

    Chris Brace

  5. Department of Radiology, University of Iowa, Iowa City, IA, USA

    James Holmes

Authors
  1. Jhimli Mitra
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Chitresh Bhushan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Soumya Ghose
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. David Mills
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Aqsa Patel
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Heather Chan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Matthew Tarasek
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Thomas Foo
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Shane Wells
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Sydney Jupitz
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Bryan Bednarz
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Chris Brace
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. James Holmes
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Desmond Yeo
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

JM, CB, and SG were involved in technical concept development in registration, data processing, code implementation, performing experiments, and writing. DM, AP and HC were involved in the development of 3D ultrasound probe and acquisition implementations and DM was involved in writing and review. TF and BB were involved in concept development of the simultaneous MR and US acquisition system. SJ, CB and MT were involved in systems development and managed imaging experiments; JH and DY were in involved concept development for liver microwave ablation, writing and review; SW is an expert radiologist in microwave ablation and was involved in clinical concept development, writing and review.

Corresponding authors

Correspondence to Jhimli Mitra or Chitresh Bhushan.

Ethics declarations

Conflict of interest

Aqsa Patel is a former employee of GE Research. All other employees of GE Research are indicated on the title page.

Ethics approval

Institutional Review Board E &I IRB No. IRB00007807, Study no 08160-13, Exp. 3 Dec 2021.

Consent to participate

All volunteers provided written consent to participate in this study.

Consent for publication

All authors provided consent to submit this work and publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mitra, J., Bhushan, C., Ghose, S. et al. A hybrid deformable registration method to generate motion-compensated 3D virtual MRI for fusion with interventional real-time 3D ultrasound. Int J CARS 18, 1501–1509 (2023). https://doi.org/10.1007/s11548-023-02833-1

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-023-02833-1

Keywords

Search

Navigation