Skip to main content

Advertisement

SpringerLink
Log in

Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients

  • Original Article
  • Published:
International Journal of Computer Assisted Radiology and Surgery Aims and scope Submit manuscript

Abstract

Purpose

Deep convolutional neural networks in their various forms are currently achieving or outperforming state-of-the-art results on several medical imaging tasks. We aim to make these developments available to the so far unsolved task of accurate correspondence finding—especially with regard to image registration.

Methods

We propose a two-step hybrid approach to make deep learned features accessible to a discrete optimization-based registration method. In a first step, in order to extract expressive binary local descriptors, we train a deep network architecture on a patch-based landmark retrieval problem as auxiliary task. As second step at runtime within a MRF-regularised dense displacement sampling, their binary nature enables highly efficient similarity computations, thus making them an ideal candidate to replace the so far used handcrafted local feature descriptors during the registration process.

Results

We evaluate our approach on finding correspondences between highly non-rigidly deformed lung CT scans from different breathing states. Although the CNN-based descriptors excell at an auxiliary learning task for finding keypoint correspondences, self-similarity-based descriptors yield more accurate registration results. However, a combination of both approaches turns out to generate the most robust features for registration.

Conclusion

We present a three-dimensional framework for large lung motion estimation based on the combination of CNN-based and handcrafted descriptors efficiently employed in a discrete registration method. Achieving best results by combining learned and handcrafted features encourages further research in this direction.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Avants BB, Epstein CL, Grossman M, Gee JC (2008) Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal 12(1):26–41

    Article  CAS  PubMed  Google Scholar 

  2. Blendowski M, Heinrich MP (2018) 3D-CNNs for deep binary descriptor learning in medical volume data. In: Maier A, Deserno T, Handels H, Maier-Hein K, Palm C, Tolxdorff T (eds) Bildverarbeitung für die Medizin. Springer, Berlin, Heidelberg, pp 23–28. https://doi.org/10.1007/978-3-662-56537-7_19

  3. Brox T, Bregler C, Malik J (2009) Largedisplacement optical flow. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp. 41–48

  4. Calonder M, Lepetit V, Strecha C, Fua P (2010) Brief: binary robust independent elementary features. In: European conference on computer vision. Springer, pp 778–792

  5. Castillo R, Castillo E, Guerra R, Johnson VE, McPhail T, Garg AK, Guerrero T (2009) A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets. Phys Med Biol 54(7):1849

    Article  PubMed  Google Scholar 

  6. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d U-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 424–432

  7. Conjeti S, Roy AG, Katouzian A, Navab N (2017) Hashing with residual networks for image retrieval. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 541–549

  8. de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. In: Cardoso M et al (eds) Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp 204–212. https://doi.org/10.1007/978-3-319-67558-9_24

  9. Dosovitskiy A, Fischer P, IlgE, Hausser P, Hazirbas C, Golkov V, van der Smagt P,Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766

  10. Dou Q, Chen H, Yu L, Qin J, Heng PA (2017) Multilevel contextual 3-D CNNS for false positive reduction in pulmonary nodule detection. IEEE Trans Biomed Eng 64(7):1558–1567

    Article  PubMed  Google Scholar 

  11. Eilertsen G, Forssén PE, Unger J (2017) Briefmatch: dense binary feature matching for real-time optical flow estimation. In: Scandinavian conference on image analysis. Springer, pp 221–233

  12. Felzenszwalb PF, Huttenlocher DP (2005) Pictorial structures for object recognition. Int J Comput Vis 61(1):55–79

    Article  Google Scholar 

  13. Glocker B, Komodakis N, Tziritas G, Navab N, Paragios N (2008) Dense image registration through MRFS and efficient linear programming. Med Image Anal 12(6):731–741

    Article  PubMed  Google Scholar 

  14. He K, Zhang X, Ren S, Sun J (2016) Deepresidual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778

  15. Heinrich MP, Blendowski M,Oktay O (2018) Ternarynet: faster deep model inference without GPUS formedical 3D segmentation using sparse and binary convolutions. arXivpreprint https://arxiv.org/abs/801.09449

  16. Heinrich MP, Handels H, Simpson IJ (2015) Estimating large lung motion in COPD patients by symmetric regularised correspondence fields. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 338–345

  17. Heinrich MP, Jenkinson M, Brady M, Schnabel JA (2013) Mrf-based deformable registration and ventilation estimation of lung CT. IEEE Trans Med Imaging 32(7):1239–1248

    Article  PubMed  Google Scholar 

  18. Heinrich MP, Jenkinson M, Papież BW, Brady M, Schnabel JA (2013) Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 187–194

  19. Heinrich MP, Oktay O (2017) Briefnet: deep pancreas segmentation using binary sparse convolutions. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 329–337

  20. Hu Y, Modat M, Gibson E, Li W, Ghavami N, Bonmati E, Wang G, Bandula S, Moore CM, Emberton M et al (2018) Weakly-supervised convolutional neural networks for multimodal image registration. Med Image Anal 49:1–13

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  21. Huang G, Liu Z, Weinberger KQ, van der Maaten L (2017) Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Patternrecognition 1:3

    Google Scholar 

  22. Liu H, Wang R, Shan S, Chen X (2016) Deep supervised hashing for fast image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2064–2072

  23. Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth international conference on 3D vision (3DV). IEEE, pp 565–571

  24. Modat M, Ridgway GR, Taylor ZA, Lehmann M, Barnes J, Hawkes DJ, Fox NC, Ourselin S (2010) Fast free-form deformation using graphics processing units. Comput Methods Prog Biomed 98(3):278–284

    Article  Google Scholar 

  25. Muenzing SE, van Ginneken B, Viergever MA, Pluim JP (2014) Dirboost-an algorithm for boosting deformable image registration: application to lung CT intra-subject registration. Med Image Anal 18(3):449–459

    Article  PubMed  Google Scholar 

  26. Muła W, Kurz N, Lemire D (2017) Faster population counts using avx2 instructions. Comput J 61(1):111–120

    Article  Google Scholar 

  27. Reinhardt JM, Ding K, Cao K, Christensen GE, Hoffman EA, Bodas SV (2008) Registration-based estimates of local lung tissue expansion compared to xenon CT measures of specific ventilation. Med Image Anal 12(6):752–763

    Article  PubMed  PubMed Central  Google Scholar 

  28. Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-net: learning deformable image registration using shape matching. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 266–274

  29. Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241

  30. Rühaak J, Polzin T, Heldmann S, Simpson IJ, Handels H, Modersitzki J, Heinrich MP (2017) Estimation of large motion in lung CT by integrating regularized keypoint correspondences into dense deformable registration. IEEE Trans Med Imaging 36(8):1746–1757

    Article  PubMed  Google Scholar 

  31. Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) Deepflow: large displacement optical flow with deep matching. In: 2013 IEEE International conference on computer vision (ICCV). IEEE, pp 1385–1392

  32. Zhang Y, Ozay M, Li S, OkataniT (2017) Truncating wide networks using binary tree architectures. arXivpreprint https://arxiv.org/abs/1704.00509

Download references

Author information

Authors and Affiliations

  1. Institute of Medical Informatics, University of Lübeck, Ratzeburger Allee 160, 23562, Lübeck, Germany

    Max Blendowski & Mattias P. Heinrich

Authors
  1. Max Blendowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mattias P. Heinrich
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Max Blendowski.

Ethics declarations

Conflict of interest

The authors declare that they have no relevant conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed consent

Statement of informed consent was not applicable since the manuscript does not contain any participants’ data.

Additional information

This work funded by the German Research Foundation (DFG) under Grant No. 320997906.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Blendowski, M., Heinrich, M.P. Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients. Int J CARS 14, 43–52 (2019). https://doi.org/10.1007/s11548-018-1888-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11548-018-1888-2

Keywords

Search

Navigation