Abstract
Magnetic Resonance Spectroscopic Imaging (MRSI) is an essential tool for quantifying metabolites in the body, but the low spatial resolution limits its clinical applications. Deep learning-based super-resolution methods provided promising results for improving the spatial resolution of MRSI, but the super-resolved images are often blurry compared to the experimentally-acquired high-resolution images. Attempts have been made with the generative adversarial networks to improve the image visual quality. In this work, we consider another type of generative model, the flow-based model, of which the training is more stable and interpretable compared to the adversarial networks. Specifically, we propose a flow-based enhancer network to improve the visual quality of super-resolution MRSI. Different from previous flow-based models, our enhancer network incorporates anatomical information from additional image modalities (MRI) and uses a learnable base distribution. In addition, we impose a guide loss and a data-consistency loss to encourage the network to generate images with high visual quality while maintaining high fidelity. Experiments on a 1H-MRSI dataset acquired from 25 high-grade glioma patients indicate that our enhancer network outperforms the adversarial networks and the baseline flow-based methods. Our method also allows visual quality adjustment and uncertainty estimation. Our code is available at https://github.com/dsy199610/Flow-Enhancer-SR-MRSI.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ardizzone, L., et al.: Analyzing inverse problems with invertible neural networks. arXiv preprint arXiv:1808.04730 (2018)
Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6228–6237 (2018)
Cengiz, S., Valdes-Hernandez, M.C., Ozturk-Isik, E.: Super resolution convolutional neural networks for increasing spatial resolution of \(^{1}\)H magnetic resonance spectroscopic imaging. In: Valdés Hernández, M., González-Castro, V. (eds.) MIUA 2017. CCIS, vol. 723, pp. 641–650. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60964-5_56
Chen, S., Sun, S., Huang, X., Shen, D., Wang, Q., Liao, S.: Data-consistency in latent space and online update strategy to guide GAN for fast MRI reconstruction. In: Deeba, F., Johnson, P., Würfl, T., Ye, J.C. (eds.) MLMIR 2020. LNCS, vol. 12450, pp. 82–90. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-61598-7_8
Coman, D., et al.: Extracellular pH mapping of liver cancer on a clinical 3T MRI scanner. Magn. Reson. Med. 83(5), 1553–1564 (2020)
De Graaf, R.A.: In Vivo NMR Spectroscopy: Principles and Techniques. Wiley, Hoboken (2019)
Denker, A., Schmidt, M., Leuschner, J., Maass, P.: Conditional invertible neural networks for medical imaging. J. Imaging 7(11), 243 (2021)
Dinh, L., Sohl-Dickstein, J., Bengio, S.: Density estimation using real NVP. arXiv preprint arXiv:1605.08803 (2016)
Dolz, J., Ben Ayed, I., Desrosiers, C.: Dense multi-path u-net for ischemic stroke lesion segmentation in multiple image modalities. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11383, pp. 271–282. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-11723-8_27
Dong, S., Chen, E.Z., Zhao, L., Chen, X., Liu, Y., Chen, T., Sun, S.: Invertible sharpening network for MRI reconstruction enhancement. arXiv preprint arXiv:2206.02838 (2022)
Dong, S., De Feyter, H.M., Thomas, M.A., de Graaf, R.A., Duncan, J.S.: A deep learning method for sensitivity enhancement in deuterium metabolic imaging (DMI). In: Proceedings of the 28th Annual Meeting of ISMRM. No. 0391 (2020)
Dong, S., et al.: High-resolution magnetic resonance spectroscopic imaging using a multi-encoder attention u-net with structural and adversarial loss. In: 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 2891–2895. IEEE (2021)
Dong, S., et al.: Multi-scale super-resolution magnetic resonance spectroscopic imaging with adjustable sharpness. arXiv preprint arXiv:2206.08984 (2022)
Hangel, G., et al.: High-resolution metabolic imaging of high-grade gliomas using 7T-CRT-FID-MRSI. NeuroImage: Clin. 28, 102433 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Iqbal, Z., Nguyen, D., Hangel, G., Motyka, S., Bogner, W., Jiang, S.: Super-resolution 1h magnetic resonance spectroscopic imaging utilizing deep learning. Front. Oncol. 9, 1010 (2019)
Kelkar, V.A., Bhadra, S., Anastasio, M.A.: Compressible latent-space invertible networks for generative model-constrained image reconstruction. IEEE Trans. Comput. Imaging 7, 209–223 (2021)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kingma, D.P., Dhariwal, P.: Glow: generative flow with invertible 1x1 convolutions. Advances in Neural Information Processing Systems, vol. 31 (2018)
Li, W., et al.: Best-buddy GANs for highly detailed image super-resolution. arXiv preprint arXiv:2103.15295 (2021)
Li, X., et al.: Deep learning super-resolution MR spectroscopic imaging of brain metabolism and mutant IDH glioma. Neuro-Oncol. Adv. 4(1) (2022)
Liang, J., Lugmayr, A., Zhang, K., Danelljan, M., Van Gool, L., Timofte, R.: Hierarchical conditional flow: a unified framework for image super-resolution and image rescaling. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4076–4085 (2021)
Lugmayr, A., Danelljan, M., Van Gool, L., Timofte, R.: SRFlow: learning the super-resolution space with normalizing flow. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12350, pp. 715–732. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58558-7_42
Menon, S., Damian, A., Hu, S., Ravi, N., Rudin, C.: PULSE: self-supervised photo upsampling via latent space exploration of generative models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2437–2445 (2020)
Padmanabha, G.A., Zabaras, N.: Solving inverse problems using conditional invertible neural networks. J. Comput. Phys. 433, 110194 (2021)
Provencher, S.W.: Lcmodel & lcmgui user’s manual. LCModel Version 6(3) (2014)
Rezende, D., Mohamed, S.: Variational inference with normalizing flows. In: International Conference on Machine Learning, pp. 1530–1538. PMLR (2015)
Smith, S.M., et al.: Advances in functional and structural MR image analysis and implementation as FSL. Neuroimage 23, S208–S219 (2004)
Wang, Z., Chen, J., Hoi, S.C.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2020)
Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: 2003 the Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, vol. 2, pp. 1398–1402. IEEE (2003)
Winkler, C., Worrall, D., Hoogeboom, E., Welling, M.: Learning likelihoods with conditional normalizing flows. arXiv preprint arXiv:1912.00042 (2019)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Zhao, H., Gallo, O., Frosio, I., Kautz, J.: Loss functions for image restoration with neural networks. IEEE Trans. Comput. Imaging 3(1), 47–57 (2016)
Acknowledgements
This work was supported by the NIH grant R01EB025840, R01CA206180 and R01NS035193. The data acquisition was supported by the Austrian Science Fund (FWF) grants KLI 646, P 30701 and P 34198.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Dong, S. et al. (2022). Flow-Based Visual Quality Enhancer for Super-Resolution Magnetic Resonance Spectroscopic Imaging. In: Mukhopadhyay, A., Oksuz, I., Engelhardt, S., Zhu, D., Yuan, Y. (eds) Deep Generative Models. DGM4MICCAI 2022. Lecture Notes in Computer Science, vol 13609. Springer, Cham. https://doi.org/10.1007/978-3-031-18576-2_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-18576-2_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18575-5
Online ISBN: 978-3-031-18576-2
eBook Packages: Computer ScienceComputer Science (R0)
