Abstract
Spatial resolution plays a crucial role in quantitative assessment of various structures in brain MRI. Super resolution (SR) as a post-processing tool holds promise for restoring the high frequency details lost in a low resolution (LR) acquisition with no additional scan time. Prior multicontrast deep learning SR approaches are mostly in 2D and operate in a pre-upsampling or progressive setting. Here we propose an efficient shallow 3D projection based post-upsampling network for anisotropic SR of brain MRI. The network is optimized using losses in the spatial and frequency domains and a complementary high resolution (HR) input to inform SR of the low resolution (LR) input with tighter integration of features. We investigated the benefit of different feature aggregation strategies such as concatenation and multiplicative attention and gradient guidance from the HR target or the additional HR input. The models were trained and evaluated on diverse datasets and performed comparably with MINet, another recently developed multimodal SR model, with approximately half the number of model parameters. The model generalized well to an external test set; performed satisfactorily on acquired LR MRI volumes despite the LR input being simulated from HR volumes during training and resulted in lower high frequency error norm. From the ablation studies, we note that a multimodal network noticeably improves SR compared to a unimodal network and feature aggregation using concatenation and multiplicative attention performed equally well. We also highlight the leakage of information from the complementary HR input to the SR output volume and the limited value of PSNR and SSIM as evaluation metrics in such cases.
R. R. Upendra and A. Pramanik–Work done as Roche Advanced Analytics Network intern.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, Y., Xie, Y., Zhou, Z., Shi, F., Christodoulou, AG., Li, D.: Brain MRI super resolution using 3D deep densely connected neural networks. In: IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 739–742 (2018)
Pham, CH., et al.: Multiscale brain MRI super-resolution using deep 3D convolutional networks. Comput. Med. Imaging Graph. 77, Article 101647 (2019)
Shi, J., et al.: MR image super-resolution via wide residual networks with fixed skip connection. IEEE J. Biomed. Heal. Inform. 3(23), 1129–1140 (2019)
Du, J., et al.: Super-resolution reconstruction of single anisotropic 3D MR images using residual convolutional neural network. Neurocomputing 392, 209–220 (2020)
Zhao, C., Dewey, B.E., Pham, D.L., Calabresi, P.A., Reich, D.A., Prince, J.L.: SMORE: a self-supervised anti-aliasing and super-resolution algorithm for MRI using deep learning. IEEE Trans. Med. Imaging 340, 805–817 (2021)
Feng, C.M., Wang, K., Lu, S., Xu, Y., Li, X.: Brain MRI super-resolution using coupled-projection residual network. Neurocomputing 456, 190–199 (2021)
Sui, Y., Afacan, O., Gholipour, A., Warfield, S.K.: Isotropic MRI super-resolution reconstruction with multi-scale gradient field prior. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11766, pp. 3–11. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32248-9_1
Sui, Y., Afacan, O., Gholipour, A., Warfield, S.K.: Learning a gradient guidance for spatially isotropic MRI super-resolution reconstruction. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 136–146. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_14
Ma, C., Rao, Y., Cheng, Y., Chen, C., Lu, J., Zhou, J.: Structure-preserving super resolution with gradient guidance. In: Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 7766–7775 (2020)
Guo, Y., et al.: Dual Reconstruction Nets for Image Super-Resolution with Gradient Sensitive Loss. arXiv:1809.07099, (2018)
Feng, CM., Fu, H., Yuan, S., Xu, Y.: Multi-contrast MRI super-resolution via a multi-stage integration network. In: International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2021), pp. 140–149 (2021)
Zeng, K., Zheng, H., Cai, C., Yang, Y., Zhang, K., Chen, Z.: Simultaneous single- and multi-contrast super-resolution for brain MRI images based on a convolutional neural network. Comput. Biol. Med. 99, 133–141 (2018)
Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1664–1673 (2018)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error measurement to structural similarity. IEEE Trans. Image Process. 13(1), 600–612 (2004)
Auricchio, G., Codegoni, A., Gualandi, S., Zambon, L.: The Fourier Loss Function. arXiv:2102.02979 (2021)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32, pp. 8024–8035. Curran Associates, Inc. (2019)
Ravishankar, S., Bresler, Y.: MR image reconstruction from highly undersampled k-space data by dictionary learning. IEEE Trans. Med. Imaging 305, 1028–1041 (2011)
Kim, T.H., Haldar, J.P.: The fourier radial error spectrum plot: a more nuanced quantitative evaluation of image reconstruction quality. In: IEEE 15th International Symposium on Biomedical Imaging, pp. 61–64 (2018)
Shi, W., et al.: Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 1874–1883 (2016)
Miller, K.L., et al.: Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat. Neurosci. 19(11), 1523–1536 (2016)
Commowick, O., et al. Objective evaluation of multiple sclerosis lesion segmentation using a data management and processing infrastructure. Sci. Rep. 8, 13650 (2018)
Avants, B.B., Tustison, N.J., Stauffer, M., Song, G., Wu, B., Gee, J.C.: The insight ToolKit image registration framework. Front. Neuroinform. 8, 44 (2014)
Acknowledgement
Aniket Pramanik and Roshan Reddy Upendra were supported by Roche Advanced Analytics Network internship program. Data used in the preparation of this article were also obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu) and the UK Biobank.
Author information
Authors and Affiliations
Consortia
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Krishnan, A.P., Upendra, R.R., Pramanik, A., Song, Z., Carano, R.A.D., the Alzheimer’s Disease Neuroimaging Initiative. (2022). Multimodal Super Resolution with Dual Domain Loss and Gradient Guidance. In: Zhao, C., Svoboda, D., Wolterink, J.M., Escobar, M. (eds) Simulation and Synthesis in Medical Imaging. SASHIMI 2022. Lecture Notes in Computer Science, vol 13570. Springer, Cham. https://doi.org/10.1007/978-3-031-16980-9_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-16980-9_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16979-3
Online ISBN: 978-3-031-16980-9
eBook Packages: Computer ScienceComputer Science (R0)
