Skip to main content
SpringerLink
Log in

Simultaneous Super-Resolution and Cross-Modality Synthesis in Magnetic Resonance Imaging

  • Chapter
  • First Online:
Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics

Part of the book series: Advances in Computer Vision and Pattern Recognition ((ACVPR))

Abstract

Multi-modality magnetic resonance imaging (MRI)  has enabled significant progress to both clinical diagnosis and medical research. Applications range from differential diagnosis to novel insights into disease mechanisms and phenotypes. However, there exist many practical scenarios where acquiring high-quality multi-modality MRI is restricted, for instance, owing to limited scanning time. This imposes constraints on multi-modality MRI processing tools, e.g., segmentation and registration. Such limitations are not only recurrent in prospective data acquisition but also when dealing with existing databases with either missing or low-quality imaging data. In this work, we explore the problem of synthesizing high-resolution images corresponding to one MRI modality from a low-resolution image of another MRI modality of the same subject. This is achieved by introducing the cross-modality dictionary learning scheme and a patch-based globally redundant model based on sparse representations. We use high-frequency multi-modality image features to train dictionary pairs, which are robust, compact, and correlated in this multimodal feature space. A feature clustering step is integrated into the reconstruction framework speeding up the search involved in the reconstruction process. Images are partitioned into a set of overlapping patches to maintain the consistency between neighboring pixels and increase speed further. Extensive experimental validations on two multi-modality databases of real brain MR images show that the proposed method outperforms state-of-the-art algorithms in two challenging tasks: image super-resolution and simultaneous SR and cross-modality synthesis. Our method was assessed on both healthy subjects and patients suffering from schizophrenia with excellent results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://brain-development.org/ixi-dataset/.

  2. 2.

    http://hdl.handle.net/1926/1687.

  3. 3.

    Following [28, 46], all the experiments data were skull stripped, linear registered and/or inhomogeneity corrected.

References

  1. Wang, Z. Lemmon, M (2015) Stability analysis of weak rural electrification microgrids with droop-controlled rotational and electronic distributed generators. In: Power and energy society general meeting. IEEE, pp 1-5

    Google Scholar 

  2. Lemann TM, Gonner C, Spitzer K (1999) Survey: interpolation methods in medical image processing [J]. IEEE Trans on Med Imaging 18(11)

    Google Scholar 

  3. Grevera GJ, Udupa JK (1998) An objective comparison of 3-D image interpolation methods. IEEE Trans Med Imaging 17(4):642–652

    Article  Google Scholar 

  4. Herman GT, Rowland SW, Yau MM (1979) A comparative study of the use of linear and modified cubic spline interpolation for image reconstruction. IEEE Trans Nuclear Sci 26(2):2879–2894

    Article  Google Scholar 

  5. Stytz MR, Parrott RW (1993) Using kriging for 3D medical imaging. Comput Med Imaging Graph 17(6):421–442

    Article  Google Scholar 

  6. Van Ouwerkerk JD (2006) Image super-resolution survey. Image Vis Comput 24(10):1039–1052

    Article  Google Scholar 

  7. Ongie G, Jacob M (2015) Recovery of discontinuous signals using group sparse higher degree total variation. IEEE Signal Process Lett 22(9):1414–1418

    Article  Google Scholar 

  8. Greenspan H, Oz G, Kiryati N, Peled S (2002) MRI inter-slice reconstruction using super-resolution. Magn Reson Imaging 20(5):437–446

    Article  MATH  Google Scholar 

  9. Shi F, Cheng J, Wang L, Yap PT, Shen D (2015) LRTV: MR image super-resolution with low-rank and total variation regularizations. IEEE Trans Med Imaging 34(12):2459–2466

    Article  Google Scholar 

  10. Yang J, Wright J, Huang TS, Ma Y (2010) Image super-resolution via sparse representation. IEEE Trans Image Process 19(11):2861–2873

    Article  MathSciNet  MATH  Google Scholar 

  11. Rousseau F (2008) Brain hallucination. In: European conference on computer vision. Springer, Berlin, pp 497–508

    Google Scholar 

  12. Chang H, Yeung DY, Xiong Y (2004) Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition. CVPR 2004, vol 1. IEEE, pp I–I

    Google Scholar 

  13. Rueda A, Malpica N, Romero E (2013) Single-image super-resolution of brain MR images using overcomplete dictionaries. Med Image Anal 17(1):113–132

    Article  Google Scholar 

  14. Freeman WT, Pasztor EC, Carmichael OT (2000) Learning low-level vision. Int J Comput Vis 40(1):25–47

    Article  MATH  Google Scholar 

  15. Sun J, Zheng NN, Tao H, Shum HY (2003). Image hallucination with primal sketch priors. In: Proceedings 2003 IEEE Computer Society Conference on computer vision and pattern recognition, vol 2. IEEE, pp II–729

    Google Scholar 

  16. Manjn JV, Carbonell-Caballero J, Lull JJ, Garca-Mart G, Mart-Bonmat L, Robles M (2008) MRI denoising using non-local means. Med Image Anal 12(4):514–523

    Article  Google Scholar 

  17. Manjn JV, Coup P, Buades A, Fonov V, Collins DL, Robles M (2010) Non-local MRI upsampling. Med Image Anal 14(6):784–792

    Article  Google Scholar 

  18. Huang Y, Shao L, Frangi AF (2017) Simultaneous super-resolution and cross-modality synthesis of 3D medical images using weakly-supervised joint convolutional sparse coding. arXiv:1705.02596

  19. Huang Y, Beltrachini L, Shao L, Frangi AF (2016) Geometry regularized joint dictionary learning for cross-modality image synthesis in magnetic resonance imaging. In: International workshop on simulation and synthesis in medical imaging. Springer, Cham, pp 118–126

    Chapter  Google Scholar 

  20. Huang Y, Shao L, Frangi AF (2018) Cross-modality image synthesis via weakly coupled and geometry co-regularized joint dictionary learning. IEEE Trans Med Imaging 37(3):815–827

    Article  Google Scholar 

  21. Bogunovic H, Pozo JM, Villa-Uriol MC, Majoie CB, van den Berg R, Gratama van Andel HA, Macho JM, Blasco J, San Romn L, Frangi AF (2011) Automated segmentation of cerebral vasculature with aneurysms in 3DRA and TOF?MRA using geodesic active regions: an evaluation study. Med Phys 38(1):210–222

    Article  Google Scholar 

  22. Nyl LG, Udupa JK, Zhang X (2000) New variants of a method of MRI scale standardization. IEEE Trans Med Imaging 19(2):143–150

    Google Scholar 

  23. Christensen JD (2003) Normalization of brain magnetic resonance images using histogram even-order derivative analysis. Magn Reson Imaging 21(7):817–820

    Article  Google Scholar 

  24. Robitaille N, Mouiha A, Crpeault B, Valdivia F, Duchesne S (2012) Tissue-based MRI intensity standardization: application to multicentric datasets. J Biomed Imaging 2012:4

    Google Scholar 

  25. Shinohara RT, Sweeney EM, Goldsmith J., Shiee, N., Mateen, FJ, Calabresi, PA, Jarso S, Pham DL, Reich DS, Crainiceanu CM, Alzheimer’s Disease Neuroimaging Initiative (2014) Statistical normalization techniques for magnetic resonance imaging. NeuroImage: Clinical 6:9–19

    Google Scholar 

  26. Roy S, Carass A, Prince JL (2013) Magnetic resonance image example-based contrast synthesis. IEEE Trans Med Imaging 32(12):2348–2363

    Article  Google Scholar 

  27. Thaipanich T, Oh BT, Wu PH, Xu D, Kuo CCJ (2010) Improved image denoising with adaptive nonlocal means (ANL-means) algorithm. IEEE Trans Consum Electron 56(4)

    Article  Google Scholar 

  28. Ye DH, Zikic D, Glocker B, Criminisi A, Konukoglu E (2013) Modality propagation: coherent synthesis of subject-specific scans with data-driven regularization. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 606–613

    Chapter  Google Scholar 

  29. Burgos N, Cardoso MJ, Thielemans K, Modat M, Pedemonte S, Dickson J, Barnes A, Ahmed R, Mahoney CJ, Schott JM, Duncan JS (2014) Attenuation correction synthesis for hybrid PET-MR scanners: application to brain studies. IEEE Trans Med Imaging 33(12):2332–2341

    Article  Google Scholar 

  30. Van Nguyen H, Zhou K, Vemulapalli R (2015) Cross-domain synthesis of medical images using efficient location-sensitive deep network. In: International conference on medical image computing and computer-assisted intervention. Springer, Cham, pp 677–684

    Google Scholar 

  31. Jog A, Carass A, Roy S, Pham DL, Prince JL (2017) Random forest regression for magnetic resonance image synthesis. Med Image Anal 35:475–488

    Article  Google Scholar 

  32. Elad M, Aharon M (2006) Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans Image Process 15(12):3736–3745

    Article  MathSciNet  Google Scholar 

  33. Buades A, Coll B, Morel JM (2010) Image denoising methods: a new nonlocal principle. SIAM Rev 52(1):113–147

    Article  MathSciNet  MATH  Google Scholar 

  34. Dong W, Zhang L, Shi G, Li X (2013) Nonlocally centralized sparse representation for image restoration. IEEE Trans Image Process 22(4):1620–1630

    Article  MathSciNet  MATH  Google Scholar 

  35. Zhu F, Shao L, Lin M (2013) Multi-view action recognition using local similarity random forests and sensor fusion. Pattern Recognit Lett 34(1):20–24

    Article  Google Scholar 

  36. Yan R, Shao L, Cvetkovic SD, Klijn J (2012) Improved nonlocal means based on pre-classification and invariant block matching. J Display Technol 8(4):212–218

    Article  Google Scholar 

  37. Huang Y, Zhu F, Shao L, Frangi AF (2016) Color object recognition via cross-domain learning on RGB-D images. In: 2016 IEEE international conference on robotics and automation (ICRA). IEEE, pp 1672–1677

    Google Scholar 

  38. Mairal J, Elad M, Sapiro G (2008) Sparse representation for color image restoration. IEEE Trans Image Process 17(1):53–69

    Article  MathSciNet  MATH  Google Scholar 

  39. Shao L, Gao X, Li H (2014) Image restoration and enhancement: recent advances and applications. Signal Process 103:1–5

    Article  Google Scholar 

  40. Aharon M, Elad M, Bruckstein A (2006) K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans Signal Process 54(11):4311

    Article  MATH  Google Scholar 

  41. Frangi AF, Niessen WJ, Vincken KL, Viergever MA (1998) Multiscale vessel enhancement filtering. In: International conference on medical image computing and computer-assisted intervention. Springer, Berlin, pp 130-137

    Chapter  Google Scholar 

  42. Davis G, Mallat S, Avellaneda M (1997) Adaptive greedy approximations. Constr Approx 13(1):57–98

    Article  MathSciNet  MATH  Google Scholar 

  43. Chen SS, Donoho DL, Saunders MA (2001) Atomic decomposition by basis pursuit. SIAM Rev 43(1):129–159

    Article  MathSciNet  MATH  Google Scholar 

  44. Kainz B, Steinberger M, Wein W, Kuklisova-Murgasova M, Malamateniou C, Keraudren K, Torsney-Weir T, Rutherford M, Aljabar P, Hajnal JV, Rueckert D (2015) Fast volume reconstruction from motion corrupted stacks of 2D slices. IEEE Trans Med imaging 34(9):1901–1913

    Article  Google Scholar 

  45. Wang Z, Bovik AC, Sheikh HR, Simoncelli EP (2004) Image quality assessment: from error visibility to structural similarity. IEEE Trans Image Process 13(4):600–612

    Article  Google Scholar 

  46. Vemulapalli R, Van Nguyen H, Zhou kS (2015) Unsupervised cross-modal synthesis of subject-specific scans. In: Proceedings of the IEEE international conference on computer vision, pp 6309–638

    Google Scholar 

  47. Shenton ME, Dickey CC, Frumin M, McCarley RW (2001) A review of MRI findings in schizophrenia. Schizophr Res 49(1–2):1–52

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Malong Technologies, Shenzhen, China

    Yawen Huang

  2. Shenzhen Malong Artificial Intelligence Research Center, Shenzhen, China

    Yawen Huang

  3. Inception Institute of Artificial Intelligence (IIAI), Abu Dhabi, United Arab Emirates

    Ling Shao

  4. University of Leeds, Leeds, UK

    Alejandro F. Frangi

Authors
  1. Yawen Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ling Shao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alejandro F. Frangi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yawen Huang .

Editor information

Editors and Affiliations

  1. Bethesda Research Lab, PAII Inc., Bethesda, MD, USA

    Le Lu

  2. Nvidia Corporation, Bethesda, MD, USA

    Xiaosong Wang

  3. School of Computer Science, University of Adelaide, Adelaide, SA, Australia

    Gustavo Carneiro

  4. Department of Biomedical Engineering, University of Florida, Gainesville, FL, USA

    Lin Yang

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Huang, Y., Shao, L., Frangi, A.F. (2019). Simultaneous Super-Resolution and Cross-Modality Synthesis in Magnetic Resonance Imaging. In: Lu, L., Wang, X., Carneiro, G., Yang, L. (eds) Deep Learning and Convolutional Neural Networks for Medical Imaging and Clinical Informatics. Advances in Computer Vision and Pattern Recognition. Springer, Cham. https://doi.org/10.1007/978-3-030-13969-8_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-13969-8_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-13968-1

  • Online ISBN: 978-3-030-13969-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation