Abstract
For interventional procedures, a real-time mapping between treatment guidance images and planning data is challenging yet essential for successful therapy implementation. Because of time and machine constraints, it involves imaging of different modalities, resolutions and dimensions, along with severe out-of-plane deformations to handle. In this paper, we introduce MSV-RegSyn-Net, a novel, scalable, deep learning-based framework for concurrent slice-to-volume registration and high-resolution modality transfer synthesis. It consists of an end-to-end pipeline made up of (i) a cycle generative adversarial network for multimodal image translation combined with (ii) a multi-slice-to-volume deformable registration network. The concurrent nature of our approach creates mutual benefit for both tasks: image translation is naturally eased by explicit handling of out-of-plane deformations while registration benefits from bringing multimodal signals into the same domain. Our model is fully unsupervised and does not require any ground-truth deformation or segmentation mask. It obtains superior qualitative and quantitative performance for multi-slice MR to 3D CT pelvic imaging compared to state-of-the-art traditional and learning-based methods on both tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Avants, B.B., Epstein, C.L., Grossman, M., Gee, J.C.: Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Med. Image Anal. 12(1), 26–41 (2008). https://doi.org/10.1016/j.media.2007.06.004
Balakrishnan, G., Zhao, A., Sabuncu, M.R., Guttag, J., Dalca, A.V.: VoxelMorph: a learning framework for deformable medical image registration. IEEE Trans. Med. Imaging 38(8), 1788–1800 (2019). https://doi.org/10.1109/TMI.2019.2897538. http://arxiv.org/abs/1809.05231, arXiv:1809.05231
Cao, X., Yang, J., Wang, L., Xue, Z., Wang, Q., Shen, D.: Deep Learning based Inter-Modality Image Registration Supervised by Intra-Modality Similarity. arXiv:1804.10735, April 2018. http://arxiv.org/abs/1804.10735
Cao, X., et al.: Deformable image registration based on similarity-steered CNN regression. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 300–308. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_35
Esteban, J., Grimm, M., Unberath, M., Zahnd, G., Navab, N.: Towards fully automatic X-Ray to CT registration. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 631–639. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_70
Estienne, T., et al.: Deep learning-based concurrent brain registration and tumor segmentation. Front. Comput. Neurosci. 14 (2020). https://www.frontiersin.org/article/10.3389/fncom.2020.00017
Ferrante, E., Paragios, N.: Non-rigid 2D-3D medical image registration using Markov random fields. In: Mori, K., Sakuma, I., Sato, Y., Barillot, C., Navab, N. (eds.) MICCAI 2013. LNCS, vol. 8151, pp. 163–170. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40760-4_21
Ferrante, E., Paragios, N.: Slice-to-volume medical image registration: a survey. Med. Image Anal. 39, 101–123 (2017). https://doi.org/10.1016/j.media.2017.04.010. https://www.sciencedirect.com/science/article/pii/S1361841517300701
Glocker, B., Komodakis, N., Tziritas, G., Navab, N., Paragios, N.: Dense image registration through MRFs and efficient linear programming. Med. Image Anal. 12(6), 731–741 (2008). https://doi.org/10.1016/j.media.2008.03.006. https://www.sciencedirect.com/science/article/pii/S1361841508000297
Glocker, B., Sotiras, A., Komodakis, N., Paragios, N.: Deformable medical image registration: setting the state of the art with discrete methods. Annu. Rev. Biomed. Eng. 13, 219–244 (2011). https://doi.org/10.1146/annurev-bioeng-071910-124649
Goodfellow, I.J., et al.: Generative Adversarial Networks. arXiv:1406.2661, June 2014. http://arxiv.org/abs/1406.2661
Guo, H., Xu, X., Xu, S., Wood, B.J., Yan, P.: End-to-end ultrasound frame to volume registration. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12904, pp. 56–65. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87202-1_6
Heinrich, M.P., et al.: MIND: modality independent neighbourhood descriptor for multi-modal deformable registration. Med. Image Anal. 16(7), 1423–1435 (2012). https://doi.org/10.1016/j.media.2012.05.008. https://www.sciencedirect.com/science/article/pii/S1361841512000643
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-Image Translation with Conditional Adversarial Networks. arXiv:1611.07004, November 2018. http://arxiv.org/abs/1611.07004
Jaganathan, S., Wang, J., Borsdorf, A., Shetty, K., Maier, A.: Deep Iterative 2D/3D Registration. arXiv:2107.10004, vol. 12904, pp. 383–392 (2021). https://doi.org/10.1007/978-3-030-87202-1_37. http://arxiv.org/abs/2107.10004
Krebs, J., et al.: Robust non-rigid registration through agent-based action learning. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 344–352. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_40
MacDonald, M.E., Dolati, P., Mitha, A.P., Eesa, M., Wong, J.H., Frayne, R.: Hemodynamic alterations measured with phase-contrast MRI in a giant cerebral aneurysm treated with a flow-diverting stent. Radiol. Case Rep. 10(2), 1109 (2015). https://doi.org/10.2484/rcr.v10i2.1109. https://www.sciencedirect.com/science/article/pii/S1930043316300334
McWilliams, J.P., Lee, E.W., Yamamoto, S., Loh, C.T., Kee, S.T.: Image-guided tumor ablation: emerging technologies and future directions. Semin. Interv. Radiol. 27(3), 302–313 (2010). https://doi.org/10.1055/s-0030-1261789. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3324186/
Miao, S., Wang, Z.J., Liao, R.: Real-time 2D/3D Registration via CNN Regression. arXiv:1507.07505, April 2016. http://arxiv.org/abs/1507.07505
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32. Curran Associates, Inc. (2019). https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
Pei, Y., et al.: Non-rigid craniofacial 2D-3D registration using CNN-based regression. In: Cardoso, M.J., et al. (eds.) DLMIA/ML-CDS -2017. LNCS, vol. 10553, pp. 117–125. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67558-9_14
Sokooti, H., de Vos, B., Berendsen, F., Lelieveldt, B.P.F., Išgum, I., Staring, M.: Nonrigid image registration using multi-scale 3D convolutional neural networks. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10433, pp. 232–239. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66182-7_27
Thirion, J.P.: Non-rigid matching using demons. In: Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 245–251, June 1996. https://doi.org/10.1109/CVPR.1996.517081. ISSN 1063-6919
Thirion, J.P.: Image matching as a diffusion process: an analogy with Maxwell’s demons. Med. Image Anal. 2(3), 243–260 (1998). https://doi.org/10.1016/S1361-8415(98)80022-4. https://www.sciencedirect.com/science/article/pii/S1361841598800224
de Vos, B.D., Berendsen, F.F., Viergever, M.A., Staring, M., Išgum, I.: End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network. arXiv:1704.06065, vol. 10553, pp. 204–212 (2017). https://doi.org/10.1007/978-3-319-67558-9_24. http://arxiv.org/abs/1704.06065
Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861
Xu, Z., et al.: Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration. arXiv:2007.02790, September 2020. http://arxiv.org/abs/2007.02790
Yan, P., Xu, S., Rastinehad, A.R., Wood, B.J.: Adversarial Image Registration with Application for MR and TRUS Image Fusion. arXiv:1804.11024, October 2018. http://arxiv.org/abs/1804.11024
Yang, H., et al.: Unpaired Brain MR-to-CT Synthesis using a Structure-Constrained CycleGAN. arXiv:1809.04536, September 2018. http://arxiv.org/abs/1809.04536
Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. arXiv:1703.10593, August 2020. http://arxiv.org/abs/1703.10593
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Leroy, A. et al. (2022). End-to-End Multi-Slice-to-Volume Concurrent Registration and Multimodal Generation. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13436. Springer, Cham. https://doi.org/10.1007/978-3-031-16446-0_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-16446-0_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16445-3
Online ISBN: 978-3-031-16446-0
eBook Packages: Computer ScienceComputer Science (R0)
