Abstract
In accelerated magnetic resonance imaging (MRI) reconstruction, the anatomy of a patient is recovered from a set of under-sampled measurements. Currently, unrolled hybrid architectures, incorporating both the beneficial bias of convolutions with the power of Transformers have been proven to be successful in solving this ill-posed inverse problem. The multi-scale strategy of the intra-cascades and that of the inter-cascades are used to decrease the high compute cost of Transformers and to rectify the spectral bias of Transformers, respectively. In this work, we proposed a dynamic Hybrid Unrolled Multi-Scale Network (dHUMUS-Net) by incorporating the two multi-scale strategies. A novel Optimal Scale Estimation Network is presented to dynamically create or choose the multi-scale Transformer-based modules in all cascades of dHUMUS-Net. Our dHUMUS-Net achieves significant improvements over the state-of-the-art methods on the publicly available fastMRI dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Here, “Hybrid” means the a Transformer-convolutional hybrid operations. Since the work of Xiao et al. [27], Transformers have been bound with convolutions for vision tasks.
References
Chen, E.Z., Wang, P., Chen, X., Chen, T., Sun, S.: Pyramid convolutional RNN for MRI image reconstruction. IEEE Trans. Med. Imaging 41(8), 2033–2047 (2022)
Chen, Y., et al.: AI-based reconstruction for fast MRI-a systematic review and meta-analysis. Proc. IEEE 110(2), 224–245 (2022)
Dolz, J., Desrosiers, C., Wang, L., Yuan, J., Shen, D., Ayed, I.B.: Deep CNN ensembles and suggestive annotations for infant brain MRI segmentation. Comput. Med. Imaging Graph. 79, 101660 (2020)
Fabian, Z., Tinaz, B., Soltanolkotabi, M.: HUMUS-Net: hybrid unrolled multi-scale network architecture for accelerated MRI reconstruction. Adv. Neural. Inf. Process. Syst. 35, 25306–25319 (2022)
Fan, J., Cao, X., Xue, Z., Yap, P.-T., Shen, D.: Adversarial similarity network for evaluating image alignment in deep learning based registration. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11070, pp. 739–746. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00928-1_83
Guo, P., Mei, Y., Zhou, J., Jiang, S., Patel, V.M.: ReconFormer: accelerated MRI econstruction using recurrent Transformer. IEEE Trans. Med. Imaging 43(1), 582–593 (2024)
Hammernik, K., et al.: Learning a variational network for reconstruction of accelerated MRI data. Magn. Reson. Med. 79(6), 3055–3071 (2018)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)
Hu, D., Zhang, Y., Zhu, J., Liu, Q., Chen, Y.: TRANS-Net: transformer-enhanced residual-error alternative suppression network for MRI reconstruction. IEEE Trans. Instrum. Meas. 71, 1–13 (2022)
Huang, J., et al.: Swin transformer for fast MRI. Neurocomputing 493, 281–304 (2022)
Hyun, C.M., Kim, H.P., Lee, S.M., Lee, S., Seo, J.K.: Deep learning for undersampled MRI reconstruction. Phys. Med. Biol. 63(13), 135007 (2018)
Kim, J., Kwon Lee, J., Mu Lee, K.: Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1646–1654 (2016)
Li, X.-X., Chen, Z., Lou, X.-J., Yang, J., Chen, Y., Shen, D.: Multimodal MRI acceleration via deep cascading networks with peer-layer-wise dense connections. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12906, pp. 329–339. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87231-1_32
Liang, D., Cheng, J., Ke, Z., Ying, L.: Deep magnetic resonance image reconstruction: inverse problems meet neural networks. IEEE Signal Process. Mag. 37(1), 141–151 (2020)
Liang, J., Cao, J., Sun, G., Zhang, K., Van Gool, L., Timofte, R.: SwinIR: image restoration using Swin transformer. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1833–1844 (2021)
Liu, Z., et al.: Swin transformer: hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Lustig, M., Donoho, D., Pauly, J.M.: Sparse MRI: the application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58(6), 1182–1195 (2007)
Qin, C., Schlemper, J., Caballero, J., Price, A.N., Hajnal, J.V., Rueckert, D.: Convolutional recurrent neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging 38(1), 280–290 (2019)
Rahaman, N., et al.: On the spectral bias of neural networks. In: Proceedings of the International Conference on Machine Learning, pp. 5301–5310 (2019)
Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Schlemper, J., Caballero, J., Hajnal, J.V., Price, A.N., Rueckert, D.: A deep cascade of convolutional neural networks for dynamic MR image reconstruction. IEEE Trans. Med. Imaging 37(2), 491–503 (2018)
Sriram, A., et al.: End-to-end variational networks for accelerated MRI reconstruction. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 64–73. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_7
Wang, J., Chen, Y., Chakraborty, R., Yu, S.X.: Orthogonal convolutional neural networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11502–11512 (2020)
Wang, Q., Guo, G.: DSA-face: diverse and sparse attentions for face recognition robust to pose variation and occlusion. IEEE Trans. Inf. Forensics Secur. 16, 4534–4543 (2021)
Wang, S., et al.: DeepcomplexMRI: exploiting deep residual network for fast parallel MR imaging with complex convolution. Magn. Reson. Imaging 68, 136–147 (2020)
Wang, X., Xie, L., Dong, C., Shan, Y.: Real-ESRGAN: training real-world blind super-resolution with pure synthetic data. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1905–1914 (2021)
Xiao, T., Singh, M., Mintun, E., Darrell, T., Dollár, P., Girshick, R.: Early convolutions help transformers see better. Adv. Neural. Inf. Process. Syst. 34, 30392–30400 (2021)
Yang, Y., Sun, J., Li, H., Xu, Z.: ADMM-CSNet: a deep learning approach for image compressive sensing. IEEE Trans. Pattern Anal. Mach. Intell. 42(3), 521–538 (2020)
Yiasemis, G., Sonke, J.J., Sśnchez, C., Teuwen, J.: Recurrent variational network: a deep learning inverse problem solver applied to the task of accelerated MRI reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 722–731 (2022)
You, D., Xie, J., Zhang, J.: ISTA-Net++: flexible deep unfolding network for compressive sensing. In: Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 1–6 (2021)
Zbontar, J., et al.: fastMRI: an open dataset and benchmarks for accelerated MRI. arXiv preprint arXiv:1811.08839 (2018)
Zhang, J., Ghanem, B.: ISTA-net: interpretable optimization-inspired deep network for image compressive sensing. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1828–1837 (2018)
Zhou, B., et al.: DSFormer: a dual-domain self-supervised transformer for accelerated multi-contrast MRI reconstruction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4966–4975 (2023)
Acknowledgments
This study was funded by Zhejiang Provincial Natural Science Foundation of China (Grant No. LGF22F020027) and by National Natural Science Foundation of China (Grant No. 62271448 and 62373324).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, XX., Zhu, FZ., Yang, J., Chen, Y., Shen, D. (2024). Dynamic Hybrid Unrolled Multi-scale Network for Accelerated MRI Reconstruction. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15007. Springer, Cham. https://doi.org/10.1007/978-3-031-72104-5_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-72104-5_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72103-8
Online ISBN: 978-3-031-72104-5
eBook Packages: Computer ScienceComputer Science (R0)