Abstract
Purpose
Deep convolutional neural networks in their various forms are currently achieving or outperforming state-of-the-art results on several medical imaging tasks. We aim to make these developments available to the so far unsolved task of accurate correspondence finding—especially with regard to image registration.
Methods
We propose a two-step hybrid approach to make deep learned features accessible to a discrete optimization-based registration method. In a first step, in order to extract expressive binary local descriptors, we train a deep network architecture on a patch-based landmark retrieval problem as auxiliary task. As second step at runtime within a MRF-regularised dense displacement sampling, their binary nature enables highly efficient similarity computations, thus making them an ideal candidate to replace the so far used handcrafted local feature descriptors during the registration process.
Results
We evaluate our approach on finding correspondences between highly non-rigidly deformed lung CT scans from different breathing states. Although the CNN-based descriptors excell at an auxiliary learning task for finding keypoint correspondences, self-similarity-based descriptors yield more accurate registration results. However, a combination of both approaches turns out to generate the most robust features for registration.
Conclusion
We present a three-dimensional framework for large lung motion estimation based on the combination of CNN-based and handcrafted descriptors efficiently employed in a discrete registration method. Achieving best results by combining learned and handcrafted features encourages further research in this direction.




Similar content being viewed by others
References
Avants BB, Epstein CL, Grossman M, Gee JC (2008) Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med Image Anal 12(1):26–41
Blendowski M, Heinrich MP (2018) 3D-CNNs for deep binary descriptor learning in medical volume data. In: Maier A, Deserno T, Handels H, Maier-Hein K, Palm C, Tolxdorff T (eds) Bildverarbeitung für die Medizin. Springer, Berlin, Heidelberg, pp 23–28. https://doi.org/10.1007/978-3-662-56537-7_19
Brox T, Bregler C, Malik J (2009) Largedisplacement optical flow. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, pp. 41–48
Calonder M, Lepetit V, Strecha C, Fua P (2010) Brief: binary robust independent elementary features. In: European conference on computer vision. Springer, pp 778–792
Castillo R, Castillo E, Guerra R, Johnson VE, McPhail T, Garg AK, Guerrero T (2009) A framework for evaluation of deformable image registration spatial accuracy using large landmark point sets. Phys Med Biol 54(7):1849
Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3d U-net: learning dense volumetric segmentation from sparse annotation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 424–432
Conjeti S, Roy AG, Katouzian A, Navab N (2017) Hashing with residual networks for image retrieval. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 541–549
de Vos BD, Berendsen FF, Viergever MA, Staring M, Išgum I (2017) End-to-end unsupervised deformable image registration with a convolutional neural network. In: Cardoso M et al (eds) Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer, Cham, pp 204–212. https://doi.org/10.1007/978-3-319-67558-9_24
Dosovitskiy A, Fischer P, IlgE, Hausser P, Hazirbas C, Golkov V, van der Smagt P,Cremers D, Brox T (2015) Flownet: learning optical flow with convolutional networks. In: Proceedings of the IEEE international conference on computer vision, pp 2758–2766
Dou Q, Chen H, Yu L, Qin J, Heng PA (2017) Multilevel contextual 3-D CNNS for false positive reduction in pulmonary nodule detection. IEEE Trans Biomed Eng 64(7):1558–1567
Eilertsen G, Forssén PE, Unger J (2017) Briefmatch: dense binary feature matching for real-time optical flow estimation. In: Scandinavian conference on image analysis. Springer, pp 221–233
Felzenszwalb PF, Huttenlocher DP (2005) Pictorial structures for object recognition. Int J Comput Vis 61(1):55–79
Glocker B, Komodakis N, Tziritas G, Navab N, Paragios N (2008) Dense image registration through MRFS and efficient linear programming. Med Image Anal 12(6):731–741
He K, Zhang X, Ren S, Sun J (2016) Deepresidual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
Heinrich MP, Blendowski M,Oktay O (2018) Ternarynet: faster deep model inference without GPUS formedical 3D segmentation using sparse and binary convolutions. arXivpreprint https://arxiv.org/abs/801.09449
Heinrich MP, Handels H, Simpson IJ (2015) Estimating large lung motion in COPD patients by symmetric regularised correspondence fields. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 338–345
Heinrich MP, Jenkinson M, Brady M, Schnabel JA (2013) Mrf-based deformable registration and ventilation estimation of lung CT. IEEE Trans Med Imaging 32(7):1239–1248
Heinrich MP, Jenkinson M, Papież BW, Brady M, Schnabel JA (2013) Towards realtime multimodal fusion for image-guided interventions using self-similarities. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 187–194
Heinrich MP, Oktay O (2017) Briefnet: deep pancreas segmentation using binary sparse convolutions. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 329–337
Hu Y, Modat M, Gibson E, Li W, Ghavami N, Bonmati E, Wang G, Bandula S, Moore CM, Emberton M et al (2018) Weakly-supervised convolutional neural networks for multimodal image registration. Med Image Anal 49:1–13
Huang G, Liu Z, Weinberger KQ, van der Maaten L (2017) Densely connected convolutional networks. Proceedings of the IEEE Conference on Computer Vision and Patternrecognition 1:3
Liu H, Wang R, Shan S, Chen X (2016) Deep supervised hashing for fast image retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 2064–2072
Milletari F, Navab N, Ahmadi SA (2016) V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth international conference on 3D vision (3DV). IEEE, pp 565–571
Modat M, Ridgway GR, Taylor ZA, Lehmann M, Barnes J, Hawkes DJ, Fox NC, Ourselin S (2010) Fast free-form deformation using graphics processing units. Comput Methods Prog Biomed 98(3):278–284
Muenzing SE, van Ginneken B, Viergever MA, Pluim JP (2014) Dirboost-an algorithm for boosting deformable image registration: application to lung CT intra-subject registration. Med Image Anal 18(3):449–459
Muła W, Kurz N, Lemire D (2017) Faster population counts using avx2 instructions. Comput J 61(1):111–120
Reinhardt JM, Ding K, Cao K, Christensen GE, Hoffman EA, Bodas SV (2008) Registration-based estimates of local lung tissue expansion compared to xenon CT measures of specific ventilation. Med Image Anal 12(6):752–763
Rohé MM, Datar M, Heimann T, Sermesant M, Pennec X (2017) SVF-net: learning deformable image registration using shape matching. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 266–274
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer, pp 234–241
Rühaak J, Polzin T, Heldmann S, Simpson IJ, Handels H, Modersitzki J, Heinrich MP (2017) Estimation of large motion in lung CT by integrating regularized keypoint correspondences into dense deformable registration. IEEE Trans Med Imaging 36(8):1746–1757
Weinzaepfel P, Revaud J, Harchaoui Z, Schmid C (2013) Deepflow: large displacement optical flow with deep matching. In: 2013 IEEE International conference on computer vision (ICCV). IEEE, pp 1385–1392
Zhang Y, Ozay M, Li S, OkataniT (2017) Truncating wide networks using binary tree architectures. arXivpreprint https://arxiv.org/abs/1704.00509
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no relevant conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Informed consent
Statement of informed consent was not applicable since the manuscript does not contain any participants’ data.
Additional information
This work funded by the German Research Foundation (DFG) under Grant No. 320997906.
Rights and permissions
About this article
Cite this article
Blendowski, M., Heinrich, M.P. Combining MRF-based deformable registration and deep binary 3D-CNN descriptors for large lung motion estimation in COPD patients. Int J CARS 14, 43–52 (2019). https://doi.org/10.1007/s11548-018-1888-2
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11548-018-1888-2