Skip to main content
SpringerLink
Log in

Memory-efficient 2.5D convolutional transformer networks for multi-modal deformable registration with weak label supervision applied to whole-heart CT and MRI scans

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose 

Despite its potential for improvements through supervision, deep learning-based registration approaches are difficult to train for large deformations in 3D scans due to excessive memory requirements.

Methods 

We propose a new 2.5D convolutional transformer architecture that enables us to learn a memory-efficient weakly supervised deep learning model for multi-modal image registration. Furthermore, we firstly integrate a volume change control term into the loss function of a deep learning-based registration method to penalize occurring foldings inside the deformation field.

Results 

Our approach succeeds at learning large deformations across multi-modal images. We evaluate our approach on 100 pair-wise registrations of CT and MRI whole-heart scans and demonstrate considerably higher Dice Scores (of 0.74) compared to a state-of-the-art unsupervised discrete registration framework (deeds with Dice of 0.71).

Conclusion 

Our proposed memory-efficient registration method performs better than state-of-the-art conventional registration methods. By using a volume change control term in the loss function, the number of occurring foldings can be considerably reduced on new registration cases.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV (2019) VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans Med Imaging. https://doi.org/10.1109/TMI.2019.2897538

    Article  Google Scholar 

  2. de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. Deep learning in medical image analysis and multi-modal learning for clinical decision support. Springer, Cham, pp 204–212

    Chapter  Google Scholar 

  3. de Vos BD, Berendse FF, Viergever MA, Sokooti H, Staring M, Išgum I (2019) A deep learning framework for unsupervised affine and deformable image registration. Med Image Anal 52:128–143

    Article  Google Scholar 

  4. Fischer B, Modersitzki J (2003) Curvature based image registration. J Math Imaging Vis 18(1):81–85

    Article  Google Scholar 

  5. Haber E, Modersitzki J (2006) Intensity gradient based registration and fusion of multi-modal images. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 726–733

    Google Scholar 

  6. Heinrich MP, Jenkinson M, Brady M, Schnabel JA (2013) MRF-based deformable registration and ventilation estimation of lung CT. IEEE Trans Med imaging 32(7):1239–1248

    Article  Google Scholar 

  7. Heinrich MP, Maier O, Handels H (2015) Multi-modal multi-atlas segmentation using discrete optimisation and self-similarities. In VISCERAL Challenge@ ISBI, pp. 27–30

  8. Heinrich MP, Okay O, Bouteldja, N (2018) OBELISK-one kernel to solve nearly everything: unified 3D binary convolutions for image analysis. Med Imaging Deep Learn. https://openreview.net/pdf?id=BkZu9wooz

  9. Heinrich MP (2018) Intra-operative ultrasound to MRI fusion with a public multi-modal discrete registration tool. MICCAI workshop on simulation, image processing, and ultrasound systems for assisted diagnosis and navigation. Springer, Cham, pp 159–164

    Chapter  Google Scholar 

  10. Hering A, Kuckertz S, Heldmann S, Heinrich MP (2019) Enhancing label-driven deep deformable image registration with local distance metrics for state-of-the-art cardiac motion tracking. Bildverarbeitung für die Medizin 2019. Springer, Berlin, Heidelberg

    Google Scholar 

  11. Hu Y, Modat M, Gibson E, Li W, Ghavami N, Bonmati E, Ourselin S (2018) Weakly-supervised convolutional neural networks for multi-modal image registration. Med Image Anal 49:1–13

    Article  CAS  Google Scholar 

  12. Jaderberg M, Simonyan K, Zisserman A (2015) Spatial transformer networks. In: Advances in neural information processing systems, pp. 2017–2025

  13. Krebs J, Mansi T, Mailhé B, Ayache N, Delingette H (2018) Unsupervised probabilistic deformation modeling for robust diffeomorphic registration. Deep learning in medical image analysis and multi-modal learning for clinical decision support. Springer, Cham, pp 101–109

    Chapter  Google Scholar 

  14. Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI (2017) A survey on deep learning in medical image analysis. Med Image Anal 42:60–88

    Article  Google Scholar 

  15. Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-Net: learning deformable image registration using shape matching. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 266–274

    Google Scholar 

  16. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. International Conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 234–241

    Google Scholar 

  17. Roth HR, Lu L, Farag A, Shin HC, Liu J, Turkbey EB, Summers RM (2015) Deeporgan: multi-level deep convolutional networks for automated pancreas segmentation. International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 556–564

    Google Scholar 

  18. Rühaak J, Heldmann S, Kipshagen T, Fischer B (2013) Highly accurate fast lung CT registration. In Medical imaging 2013: image processing, vol. 8669, p. 86690Y. International Society for Optics and Photonics

  19. Rühaak J, Derksen A, Heldmann S, Hallmann M, Meine H (2015) Accurate CT-MR image registration for deep brain stimulation: a multi-observer evaluation study. In Medical imaging 2015: image processing, vol. 9413, p. 941337. International Society for Optics and Photonics

  20. Prasoon A, Petersen K, Igel C, Lauze F, Dam E, Nielsen M (2013) Deep feature learning for knee cartilage segmentation using a triplanar convolutional neural network. International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 246–253

    Google Scholar 

  21. Sotiras A, Davatzikos C, Paragios N (2013) Deformable medical image registration: a survey. IEEE Trans Med Imaging 32(7):1153–1190

    Article  Google Scholar 

  22. Xia Y, Xie L, Liu F, Zhu Z, Fishman EK, Yuille AL (2018) Bridging the gap between 2D and 3D organ segmentation. arXivpreprint arXiv:1804.00392

  23. Xu Z, Lee CP, Heinrich MP, Modat M, Rueckert D, Ourselin S, Landman B (2016) Evaluation of six registration methods for the human abdomen on clinically acquired CT. IEEE Trans Biomed Eng 63(8):1563–1572

    Article  Google Scholar 

  24. Zhuang X, Shen J (2016) Multi-scale patch and multi-modality atlases for whole heart segmentation of MRI. Med Image Anal 31:77–87

    Article  Google Scholar 

Download references

Funding

This work was funded in part by the German Research Foundation (DFG) under grant number 320997906.

Author information

Author notes

  1. Alessa Hering and Sven Kuckertz contributed equally to this work.

Authors and Affiliations

  1. Fraunhofer Institute for Digital Medicine MEVIS, Maria-Goeppert-Str. 3, 23562, Lübeck, Germany

    Alessa Hering, Sven Kuckertz & Stefan Heldmann

  2. Diagnostic Image Analysis Group, Radboudumc, Geert Grooteplein 10, 6525 GA, Nijmegen, Netherlands

    Alessa Hering

  3. Institute of Medical Informatics, University of Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Germany

    Mattias P. Heinrich

Authors
  1. Alessa Hering
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sven Kuckertz
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Stefan Heldmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mattias P. Heinrich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alessa Hering.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

This article does not contain patient data.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hering, A., Kuckertz, S., Heldmann, S. et al. Memory-efficient 2.5D convolutional transformer networks for multi-modal deformable registration with weak label supervision applied to whole-heart CT and MRI scans. Int J CARS 14, 1901–1912 (2019). https://doi.org/10.1007/s11548-019-02068-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-019-02068-z

Keywords

Search

Navigation