Abstract
In medical image acquisition, hardware limitations and scanning time constraints result in degraded images. Super-resolution (SR) is a post-processing approach aiming to reconstruct a high-resolution image from its low-resolution counterpart. Recent advances in medical image SR include the application of deep neural networks, which can improve image quality at a low computational cost. When dealing with medical data, accuracy is important for discovery and diagnosis, therefore, interpretable neural network models are of significant interest as they enable a theoretical study and increase trustworthiness needed in clinical practice. While several interpretable deep learning designs have been proposed to treat unimodal images, to the best of our knowledge, there is no multimodal SR approach applied for medical images. In this paper, we present an interpretable neural network model that exploits information from multiple modalities to super-resolve an image of a target modality. Experiments with simulated and real MRI data show the performance of the proposed approach in terms of numerical and visual results.
This work has been co-funded by the European Union and Greek national funds through the Operational Program Competitiveness, Entrepreneurship and Innovation, under the call RESEARCH-CREATE-INNOVATE (project code: T1EDK03895).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Notation: Lower case letters are used for scalars, boldface lower case letters for vectors, boldface upper case letters for matrices and boldface upper case letters in math calligraphy for tensors.
- 2.
References
Adler, J., Öktem, O.: Learned primal-dual reconstruction. IEEE Trans. Med. Imag. 37(6), 1322–1332 (2018)
Aubert-Broche, B., Griffin, M., Pike, G.B., Evans, A.C., Collins, D.L.: Twenty new digital brain phantoms for creation of validation image data bases. IEEE Trans. Med. Imag. 25(11), 1410–1416 (2006)
Borgerding, M., Schniter, P., Rangan, S.: AMP-inspired deep networks for sparse linear inverse problems. IEEE Trans. Sig. Process. 65(16), 4293–4308 (2017)
Daubechies, I., Defrise, M., Mol, C.D.: An iterative thresholding algorithm for linear inverse problems with a sparsity constrain. Commun. Pure Appl. Math. 57, 1413 (2004)
Deng, X., Dragotti, P.L.: Deep coupled ISTA network for multi-modal image super-resolution. IEEE Trans. Image Process. 29, 1683–1698 (2020)
Greenspan, H.: Super-resolution in medical imaging. Comput. J. 52(1), 43–63 (2009)
Gregor, K., LeCun, Y.: Learning fast approximations of sparse coding. In: Proceedings of the 27th International Conference on Machine Learning, pp. 399–406. ICML 2010, Omnipress, USA (2010)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Lesjak, Ž, et al.: A novel public MR image dataset of multiple sclerosis patients with lesion segmentations based on multi-rater consensus. Neuroinformatics 16(1), 51–63 (2018)
Liu, D., Wang, Z., Wen, B., Yang, J., Han, W., Huang, T.S.: Robust single image super-resolution via deep networks with sparse prior. IEEE Trans. Image Process. 25(7), 3194–3207 (2016)
Lucas, A., Iliadis, M., Molina, R., Katsaggelos, A.K.: Using deep neural networks for inverse problems in imaging: beyond analytical methods. IEEE Sig. Process. Mag. 35(1), 20–36 (2018)
Mansoor, A., Vongkovit, T., Linguraru, M.G.: Adversarial approach to diagnostic quality volumetric image enhancement. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 353–356. IEEE (2018)
Marivani, I., Tsiligianni, E., Cornelis, B., Deligiannis, N.: Multimodal image super-resolution via Deep Unfolding with Side Information. In: European Signal Processing Conference (EUSIPCO) (2019)
Marivani, I., Tsiligianni, E., Cornelis, B., Deligiannis, N.: Multimodal deep unfolding for guided image super-resolution. IEEE Trans. Image Process. 29, 8443–8456 (2020)
Monga, V., Li, Y., Eldar, Y.C.: Algorithm unrolling: Interpretable, efficient deep learning for signal and image processing. arXiv preprint arXiv:1912.10557 (2019)
Mota, J.F.C., Deligiannis, N., Rodrigues, M.R.D.: Compressed sensing with prior information: strategies, geometry, and bounds. IEEE Trans. Inf. Theory 63(7), 4472–4496 (2017)
Nehme, E., Weiss, L.E., Michaeli, T., Shechtman, Y.: Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica 5(4), 458–464 (2018)
Papyan, V., Romano, Y., Elad, M.: Convolutional neural networks analyzed via convolutional sparse coding. J. Mach. Learn. Res. 18(1), 2887–2938 (2017)
Qin, C., Schlemper, J., Caballero, J., Price, A.N., Hajnal, J.V., Rueckert, D.: Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imag. 38(1), 280–290 (2018)
Ribes, A., Schmitt, F.: Linear inverse problems in imaging. IEEE Sig. Process. Mag. 25(4), 84–99 (2008)
Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D.: A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imag. 37(2), 491–503 (2017)
Sreter, H., Giryes, R.: Learned convolutional sparse coding. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2191–2195. IEEE (2018)
Tsiligianni, E., Deligiannis, N.: Deep coupled-representation learning for sparse linear inverse problems with side information. IEEE Sig. Process. Lett. 26, 1768 (2019)
Xiang, L., et al.: Deep-learning-based multi-modal fusion for fast MR reconstruction. IEEE Trans. Biomed. Eng. 66(7), 2105–2114 (2018)
Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19, 2861–2873 (2010)
Yang, Y., Sun, J., Li, H., Xu, Z.: Deep ADMM-Net for compressive sensing MRI. In: Advances in Neural Information Processing Systems (NIPS), pp. 10–18 (2016)
Yedder, H.B., Cardoen, B., Hamarneh, G.: Deep learning for biomedical image reconstruction: a survey. Artif. Intell. Rev. 54, 1–37 (2020)
Zeiler, M.D., Krishnan, D., Taylor, G.W., Fergus, R.: Deconvolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2010)
Zeng, K., Zheng, H., Cai, C., Yang, Y., Zhang, K., Chen, Z.: Simultaneous single-and multi-contrast super-resolution for brain MRI images based on a convolutional neural network. Comput. Biol. Med. 99, 133–141 (2018)
Zhou, B., Zhou, S.K.: DuDoRNet: learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4273–4282 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Tsiligianni, E., Zerva, M., Marivani, I., Deligiannis, N., Kondi, L. (2021). Interpretable Deep Learning for Multimodal Super-Resolution of Medical Images. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12906. Springer, Cham. https://doi.org/10.1007/978-3-030-87231-1_41
Download citation
DOI: https://doi.org/10.1007/978-3-030-87231-1_41
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87230-4
Online ISBN: 978-3-030-87231-1
eBook Packages: Computer ScienceComputer Science (R0)
