Skip to main content
SpringerLink
Log in

SEN-FCB: an unsupervised twinning neural network for image registration

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Medical image registration is a fundamental and vital task in medical image analysis. Deformable medical image registration generates a dense nonlinear transformation from the moving image to the fixed image. Current learning-based image registration methods utilize U-shaped networks, concatenate moving and fixed images as one input, and then impose a global regularization to ensure smooth deformation fields. However, existing deformable image registration approaches concatenate image pairs as one input to their model and may ignore independent anatomical relevance of the images. Moreover, the global regularization causes over/underconstraining, affecting their model registration accuracy and over/under enforcing the deformation field’s smoothness. To address these two problems, we propose a twinning network, consisting of two subnetworks. The first subnetwork is the proposed separate encoding neural network (SEN) for predicting high-accuracy deformation fields, and the second subnetwork is a folding correction block (FCB) to correct the deformation fields to achieve folding reduction. Comparing our experimental results to the state-of-the-art displacement and diffeomorphic methods, the proposed method provides superior registration accuracy and reduces the folding numbers. Moreover, we utilize the FCB to correct the baselines’ output deformation fields, proving that the FCB outperforms global regularization.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Code Availability

Our code is available at https://github.com/MingR-Ma/SEN-FCB.

References

  1. Veiga C, Janssens G, Teng C-L, Baudier T, Hotoiu L, McClelland JR, Royle G, Lin L, Yin L, Metz J, Solberg TD, Tochner Z, Simone CB, McDonough J, Kevin Teo B-K (2016) First clinical investigation of cone beam computed tomography and deformable registration for adaptive proton therapy for lung cancer. Int J Rad Oncol Biol Phys 95(1):549–559

    Article  Google Scholar 

  2. Nakao M, Kobayashi K, Tokuno J, Chen-Yoshikawa T, Date H, Matsuda T (2021) Deformation analysis of surface and bronchial structures in intraoperative pneumothorax using deformable mesh registration. Med Image Anal 102181:73

    Google Scholar 

  3. Alvarez P, Rouzé S, Miga MI, Payan Y, Dillenseger J-L, Chabanas M (2021) A hybrid, image-based and biomechanics-based registration approach to markerless intraoperative nodule localization during video-assisted thoracoscopic surgery. Med Image Anal 101983:69

    Google Scholar 

  4. Balakrishnan G, Zhao A, Sabuncu MR, Guttag J, Dalca AV (2018) An unsupervised learning model for deformable medical image registration. In: 2018 IEEE/CVF conference on computer vision and pattern recognition

  5. Kim B, Dong HK, Park SH, Kim J, Ye JC (2021) Cyclemorph: Cycle consistent unsupervised deformable image registration. Med Image Anal 71(1):102036

    Article  Google Scholar 

  6. Dalca AV, Balakrishnan G, Guttag J, Sabuncu MR (2019) Unsupervised learning of probabilistic diffeomorphic registration for images and surfaces. Med Image Anal 57:226–236

    Article  Google Scholar 

  7. Zhao S, Dong Y, Chang EI-C, Xu Y (2019) Recursive cascaded networks for unsupervised medical image registration. In: Proceedings of the IEEE/CVF international conference on computer vision (ICCV)

  8. Wang J, Zhang M (2020) Deepflash: an efficient network for learning-based medical image registration. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR)

  9. Jaderberg M, Simonyan K, Zisserman A, Kavukcuoglu K (2015) Spatial transformer networks Advances in neural information processing systems 28: Annual conference on neural information processing systems 2015, december 7-12, 2015, montreal, quebec, canada, pp 2017–2025

  10. Avants BB, Epstein CL, Grossman M, Gee JC (2008) Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal 12(1):26–41

    Article  Google Scholar 

  11. Chen J, He Y, Frey EC, Li Y, Du Y (2021) ViT-V-Net vision transformer for unsupervised volumetric medical image registration

  12. Mok T, Chung A (2020) Fast symmetric diffeomorphic image registration with convolutional neural networks. In: 2020 IEEE/CVF Conference on computer vision and pattern recognition (CVPR)

  13. Kang M, Hu X, Huang W, Scott MR, Reyes M (2022) Dual-stream pyramid registration network. Med Image Anal 78:102379

    Article  Google Scholar 

  14. Jud C, Möri N, Bitterli B, Cattin PC (2016) Bilateral regularization in reproducing kernel hilbert spaces for discontinuity preserving image registration. In: International workshop on machine learning in medical imaging, Springer, pp 10–17

  15. Shen D, Davatzikos C (2002) Hammer: hierarchical attribute matching mechanism for elastic registration. IEEE Trans Med Imaging 21(11):1421–1439

    Article  Google Scholar 

  16. Ardekani BA, Guckemus S, Bachman A, Hoptman MJ, Wojtaszek M, Nierenberg J (2005) Quantitative comparison of algorithms for inter-subject registration of 3d volumetric brain mri scans. J Neurosci Methods 142(1):67–76

    Article  Google Scholar 

  17. Woods RP, Grafton ST, Holmes CJ, Cherry SR, Mazziotta JC (1998) Automated image registration: i. general methods and intrasubject, intramodality validation. J Comput Assist Tomogr 22(1):139

    Article  Google Scholar 

  18. Hellier P, Ashburner J, Corouge I, Barillot C, Friston K (2002) Inter-subject registration of functional and anatomical data using spm. Springer, Berlin, pp 590–597

    MATH  Google Scholar 

  19. Maes F, Collignon A, Vandermeulen D, Marchal G, Suetens P (1997) Multimodality image registration by maximization of mutual information. IEEE Trans Med Imaging 16(2):187–198

    Article  Google Scholar 

  20. Davatzikos C (1997) Spatial transformation and registration of brain images using elastically deformable models. Comp Vision Image Underst (CVIU) 66(2):207

    Article  Google Scholar 

  21. Glocker B, Komodakis N, Tziritas G, Navab N, Paragios N (2008) Dense image registration through mrfs and efficient linear programming. Med Image Anal 12(6):731–741

    Article  Google Scholar 

  22. Dalca AV, Bobu A, Rost NS, Golland P (2016) Patch-based discrete registration of clinical brain images. In: International workshop on patch-based techniques in medical imaging

  23. Ou Y, Sotiras A, Paragios N, Davatzikos C (2011) Dramms: deformable registration via attribute matching and mutual-saliency weighting. Med Image Anal 15(4):622–639

    Article  Google Scholar 

  24. Beg MF, Miller MI, Trouvé A, Younes L (2005) Computing large deformation metric mappings via geodesic flows of diffeomorphisms. Int J Comput Vis 61(2):139–157

    Article  MATH  Google Scholar 

  25. Vercauteren T, Pennec X, Perchant A, Ayache N (2009) Diffeomorphic demons: efficient non-parametric image registration. Neuroimage 45(1):61–72

    Article  Google Scholar 

  26. Zhang M, Liao R, Dalca AV, Turk EA, Golland P (2017) Frequency diffeomorphisms for efficient image registration. In: International conference on information processing in medical imaging

  27. Ashburner J (2007) A fast diffeomorphic image registration algorithm. Neuroimage 38(1):95–113

    Article  Google Scholar 

  28. Cao X, Yang J, Zhang J, Nie D, Kim M, Wang Q, Shen D (2017) Deformable image registration based on similarity-steered cnn regression. In: International conference on medical image computing and computer-assisted intervention, Springer, pp 300–308

  29. Cao X, Yang J, Zhang J, Wang Q, Yap P-T, Shen D (2018) Deformable image registration using a cue-aware deep regression network. IEEE Trans Biomed Eng 65(9):1900–1911

    Article  Google Scholar 

  30. Liao R, Miao S, de Tournemire P, Grbic S, Kamen A, Mansi T, Comaniciu D (2017) An artificial agent for robust image registration. In: Proceedings of the thirty-first AAAI conference on artificial intelligence, pp 4168–4175

  31. Miao S, Piat S, Fischer PW, Tuysuzoglu A, Mewes PW, Mansi T, Liao R (2018) Dilated FCN for multi-agent 2d/3d medical image registration. In: Proceedings of the thirty-second AAAI conference on artificial intelligence, pp 4694–4701

  32. Al Safadi E, Song X (2021) Learning-based image registration with meta-regularization. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 10928–10937

  33. Christensen GE, Johnson HJ (2001) Consistent image registration. IEEE Trans Med Imaging 20(7):568–582

    Article  Google Scholar 

  34. Ronneberger O, Fischer P, Brox T (2015) U-net: convolutional networks for biomedical image segmentation. In: Navab N, Hornegger J, Wells WM, Frangi AF (eds) Medical image computing and computer-assisted intervention – MICCAI 2015, Springer, pp 234–241

  35. Ouyang D, He B, Ghorbani A, Yuan N, Ebinger J, Langlotz CP, Heidenreich PA, Harrington RA, Liang DH, Ashley EA, et al. (2020) Video-based ai for beat-to-beat assessment of cardiac function. Nature 580(7802):252–256

    Article  Google Scholar 

  36. Marcus DS, Wang TH, Parker J, Csernansky JG, Morris JC, Buckner RL (2007) Open access series of imaging studies (OASIS): cross-sectional MRI data in young, middle aged, nondemented, and demented older adults. J Cogn Neurosci 19(9):1498–1507

    Article  Google Scholar 

  37. Fischl B (2012) Freesurfer. Neuroimage 62(2):774–781. 20 YEARS OF fMRI

    Article  Google Scholar 

  38. Kingma DP, Ba J (2017) Adam: a method for stochastic optimization

Download references

Acknowledgments

This work is supported by The National Nature Science Foundation of China (Grant Nos. 61772226 and 61862056), The Natural Science Foundation of Jilin Province (Grant number No. 20210204133YY), Key Laboratory for Symbol Computation and Knowledge Engineering of the National Education Ministry of China, Jilin University.

Funding

This work was supported by the National Nature Science Foundation of China [grant number 61772226, 61862056]; Science and Technology Development Program of Jilin Province [grant number 20210204133YY]; The Natural Science Foundation of Jilin Province (Grant number No. 20200201159JC); Key Laboratory for Symbol Computation and Knowledge Engineering of the National Education Ministry of China,Jilin University.

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Jilin University, Changchun, 130012, Jilin, People’s Republic of China

    Mingrui Ma, Guixia Liu, Lei Song & Yuanbo Xu

  2. Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, 130012, Jilin, People’s Republic of China

    Mingrui Ma, Guixia Liu, Lei Song & Yuanbo Xu

Authors
  1. Mingrui Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guixia Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lei Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yuanbo Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Lei Song, Mingrui Ma and Guixia Liu. The first draft of the manuscript was written by Lei Song and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Guixia Liu.

Ethics declarations

Conflict of Interests

The authors have no conflicts of interest to declare that are relevant to the content of this article.

Additional information

Availability of data and materials

The data is a public data set and it can be obtained https://www.oasis-brains.org/. We preprocessed the data.

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Guixia Liu, Lei Song and Yuanbo Xu are contributed equally to this work.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ma, M., Liu, G., Song, L. et al. SEN-FCB: an unsupervised twinning neural network for image registration. Appl Intell 53, 12198–12209 (2023). https://doi.org/10.1007/s10489-022-04109-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04109-8

Keywords

Search

Navigation