Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Article
  • Published:

A super-resolution strategy for mass spectrometry imaging via transfer learning

Nature Machine Intelligence volume 5, pages 656–668 (2023)Cite this article

Subjects

Abstract

High-spatial-resolution mass spectrometry imaging (HSR-MSI) provides precise spatial information on thousands of biomolecules without labelling across a tissue section. Deep learning methods, trained on large numbers of images, can be used to further improve resolution. However, the limited amount of HSR-MSI data that are publicly available mean that super-resolution reconstruction of images obtained by MSI using deep learning is still a challenge. Here we develop a deep learning framework based on transfer learning called MSI from optical super-resolution (MOSR) that substantially reduces the requirement for sample size. Needing only ten HSR-MSI images, the method transfers knowledge learned from abundant optical images (~15,000) to MSI tasks. Compared with the deep learning model without transfer learning, the MOSR model obtains better image quality with higher peak signal-to-noise ratios and structural similarity index values. It also achieves higher training efficiency and a stronger generalization performance. The MOSR model predicts HSR-MSI images with very small sample size and could transform applications with super-resolution MSI.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Fig. 1: Overview of MOSR.
Fig. 2: Image quality of the MOSR model compared with other models.
Fig. 3: Training efficiency of the MOSR model.
Fig. 4: Generalization performance of the MOSR model.
Fig. 5: Cross-modality image transformation from DESI to MALDI.

Similar content being viewed by others

Data availability

The public optical imaging data can be found at Allen Brain Atlas database (https://atlas.brain-map.org/) under dataset id numbers 67787347, 68162246, 68442901, 72270832 and so on. To facilitate access, we collated all the optical imaging data that we used to a public available data repository via Figshare at https://doi.org/10.6084/m9.figshare.22639936.v1 (ref. 69). Each image in the Figshare is named by its image id number in the Allen Brain Atlas database. The public MSI imaging data are available from the METASPACE Platform under dataset ids 100um_M2_003_Recal (https://metaspace2020.eu/datasets?q=2017-07-18_17h21m08s), FullBrain_Norh_neg (https://metaspace2020.eu/datasets?q=2020-09-01_10h04m59s), 20200904_FullBrain_Norh_pos_2 (https://metaspace2020.eu/datasets?q=2020-09-04_11h03m04s) and 20200827_Brain_Cer_Nor_neg_i (https://metaspace2020.eu/datasets?q=2021-04-15_13h34m52s).

Code availability

In this work we used open-source image and video restoration toolbox (BasicSR; https://github.com/xinntao/BasicSR) to build ESRGAN models. All custom code is available via GitHub at https://github.com/USTC-xlab/MOSR (ref. 70).

References

  1. Amstalden van Hove, E. R., Smith, D. F. & Heeren, R. M. A. A concise review of mass spectrometry imaging. J. Chromatogr. A 1217, 3946–3954 (2010).

    Google Scholar 

  2. Buchberger, A. R., DeLaney, K., Johnson, J. & Li, L. Mass spectrometry imaging: a review of emerging advancements and future insights. Anal. Chem. 90, 240–265 (2018).

    Google Scholar 

  3. Hansen, R. L. & Lee, Y. J. High-spatial resolution mass spectrometry imaging: toward single cell metabolomics in plant tissues. Chem. Rec. 18, 65–77 (2018).

    Google Scholar 

  4. Römpp, A. & Spengler, B. Mass spectrometry imaging with high resolution in mass and space. Histochem. Cell Biol. 139, 759–783 (2013).

    Google Scholar 

  5. Hsieh, Y. et al. Matrix-assisted laser desorption/ionization imaging mass spectrometry for direct measurement of clozapine in rat brain tissue. Rapid Commun. Mass Spectrom. 20, 965–972 (2006).

    Google Scholar 

  6. Bunch, J., Clench, M. R. & Richards, D. S. Determination of pharmaceutical compounds in skin by imaging matrix-assisted laser desorption/ionisation mass spectrometry. Rapid Commun. Mass Spectrom. 18, 3051–3060 (2004).

    Google Scholar 

  7. Prideaux, B. & Stoeckli, M. Mass spectrometry imaging for drug distribution studies. J. Proteom. 75, 4999–5013 (2012).

    Google Scholar 

  8. Takáts, Z., Wiseman Justin, M., Gologan, B. & Cooks, R. G. Mass spectrometry sampling under ambient conditions with desorption electrospray ionization. Science 306, 471–473 (2004).

    Google Scholar 

  9. Wiseman Justin, M. et al. Desorption electrospray ionization mass spectrometry: imaging drugs and metabolites in tissues. Proc. Natl Acad. Sci. USA 105, 18120–18125 (2008).

    Google Scholar 

  10. Nilsson, A. et al. Investigating nephrotoxicity of polymyxin derivatives by mapping renal distribution using mass spectrometry imaging. Chem. Res. Toxicol. 28, 1823–1830 (2015).

    Google Scholar 

  11. Li, B. et al. Interrogation of spatial metabolome of Ginkgo biloba with high-resolution matrix-assisted laser desorption/ionization and laser desorption/ionization mass spectrometry imaging. Plant Cell Environ. 41, 2693–2703 (2018).

    Google Scholar 

  12. Cobice, D. F. et al. Mass spectrometry imaging for dissecting steroid intracrinology within target tissues. Anal. Chem. 85, 11576–11584 (2013).

    Google Scholar 

  13. Yin, R., Burnum-Johnson, K. E., Sun, X., Dey, S. K. & Laskin, J. High spatial resolution imaging of biological tissues using nanospray desorption electrospray ionization mass spectrometry. Nat. Protoc. 14, 3445–3470 (2019).

    Google Scholar 

  14. Qi, K. et al. Cholesterol was identified as a biomarker in human melanocytic nevi using DESI and DESI/PI mass spectrometry imaging. Talanta 231, 122380 (2021).

    Google Scholar 

  15. Thoma, C. Making DESI-MSI desirable. Nat. Rev. Urol. 14, 325–325 (2017).

    Google Scholar 

  16. Balluff, B., Heeren, R. M. A. & Race, A. M. An overview of image registration for aligning mass spectrometry imaging with clinically relevant imaging modalities. J. Mass Spectrom. Adv. Clin. Lab 23, 26–38 (2022).

    Google Scholar 

  17. Bokhart, M. T., Nazari, M., Garrard, K. P. & Muddiman, D. C. MSiReader v1.0: evolving open-source mass spectrometry imaging software for targeted and untargeted analyses. J. Am. Soc. Mass. Spectrom. 29, 8–16 (2018).

    Google Scholar 

  18. Van de Plas, R., Yang, J., Spraggins, J. & Caprioli, R. M. Image fusion of mass spectrometry and microscopy: a multimodality paradigm for molecular tissue mapping. Nat. Methods 12, 366–372 (2015).

    Google Scholar 

  19. Van Malderen, S. J., van Elteren, J. T. & Vanhaecke, F. Submicrometer imaging by laser ablation-inductively coupled plasma mass spectrometry via signal and image deconvolution approaches. Anal. Chem. 87, 6125–6132 (2015).

    Google Scholar 

  20. Westerhausen, M. T. et al. Super-resolution reconstruction for two- and three-dimensional LA-ICP-MS bioimaging. Anal. Chem. 91, 14879–14886 (2019).

    Google Scholar 

  21. Titus, J. & Geroge, S. A comparison study on different interpolation methods based on satellite images. Int. J. Eng. Res. Technol. 2, 82–85 (2013).

  22. Dianyuan, H. Comparison of commonly used image interpolation methods. In Proc. 2nd International Conference on Computer Science and Electronics Engineering 10, 1556–1559 (Atlantis Press, 2013).

  23. Zhao, C., Guo, L., Dong, J. & Cai, Z. Mass spectrometry imaging-based multi-modal technique: next-generation of biochemical analysis strategy. Innovation 2, 100151–100151 (2021).

    Google Scholar 

  24. Porta Siegel, T. et al. Mass spectrometry imaging and integration with other imaging modalities for greater molecular understanding of biological tissues. Mol. Imag. Biol. 20, 888–901 (2018).

    Google Scholar 

  25. DeLaney, K., Phetsanthad, A. & Li, L. Advances in high-resolution maldi mass spectrometry for neurobiology. Mass Spectrom. Rev. 41, 194–214 (2022).

    Google Scholar 

  26. Holzlechner, M., Eugenin, E. & Prideaux, B. Mass spectrometry imaging to detect lipid biomarkers and disease signatures in cancer. Cancer Rep. 2, e1229–e1229 (2019).

    Google Scholar 

  27. Spraggins, J. M. & Caprioli, R. M. High-speed MALDI-TOF imaging mass spectrometry: rapid ion image acquisition and considerations for next generation instrumentation. J. Am. Soc. Mass. Spectrom. 22, 1022–1031 (2011).

    Google Scholar 

  28. Batson, J. & Royer, L. Noise2Self: blind denoising by self-supervision. In Proc. 36th International Conference on Machine Learning Vol. 97 (eds. Kamalika, C. & Ruslan, S.) 524–533 (PMLR, 2019).

  29. Krull, A., Buchholz, T.-O. & Jug, F. Noise2Void-learning denoising from single noisy images. In Proc. IEEE/CVF Conference on Computer Vision and Pattern Recognition 2129–2137 (IEEE, 2019).

  30. Li, Y. et al. DLBI: deep learning guided Bayesian inference for structure reconstruction of super-resolution fluorescence microscopy. Bioinformatics 34, i284–i294 (2018).

    Google Scholar 

  31. Weigert, M. et al. Content-aware image restoration: pushing the limits of fluorescence microscopy. Nat. Methods 15, 1090–1097 (2018).

    Google Scholar 

  32. Buchholz, T., Jordan, M., Pigino, G. & Jug, F. Cryo-CARE: content-aware image restoration for cryo-transmission electron microscopy data. In Proc. IEEE 16th International Symposium on Biomedical Imaging 502–506 (IEEE, 2019).

  33. de Haan, K., Ballard, Z. S., Rivenson, Y., Wu, Y. & Ozcan, A. Resolution enhancement in scanning electron microscopy using deep learning. Sci Rep. 9, 12050 (2019).

    Google Scholar 

  34. Heinrich, L., Bogovic, J. A. & Saalfeld, S. in Medical Image Computing and Computer Assisted Intervention – MICCAI 2017, Vol. 10434 (eds Descoteaux, M. et al.) 135–143 (Springer, Cham, 2017).

  35. Sreehari, S. et al. Multi-resolution data fusion for super-resolution electron microscopy. In Proc. IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 1084–1092 (IEEE, 2017).

  36. Park, J. et al. Computed tomography super-resolution using deep convolutional neural network. Phys. Med. Biol. 63, 145011 (2018).

    Google Scholar 

  37. You, C. et al. CT Super-Resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE). IEEE Trans. Med. Imag. 39, 188–203 (2020).

    Google Scholar 

  38. Zhang, Z. et al. CT super resolution via zero shot learning. Preprint at https://arxiv.org/abs/2012.08943 (2020).

  39. Chen, H. et al. Real-world single image super-resolution: a brief review. Inform. Fusion 79, 124–145 (2022).

    Google Scholar 

  40. Yang, W. et al. Deep learning for single image super-resolution: a brief review. IEEE Trans. Multimedia 21, 3106–3121 (2019).

    Google Scholar 

  41. Yang, L., Hanneke, S. & Carbonell, J. A theory of transfer learning with applications to active learning. Mach. Learn. 90, 161–189 (2013).

    MathSciNet  MATH  Google Scholar 

  42. Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Proc. 27th International Conference on Neural Information Processing Systems – Volume 2 3320–3328 (MIT Press, 2014).

  43. He, J. et al. A sensitive and wide coverage ambient mass spectrometry imaging method for functional metabolites based molecular histology. Adv. Sci. 5, 1800250 (2018).

    Google Scholar 

  44. Veselkov, K. A. et al. Chemo-informatic strategy for imaging mass spectrometry-based hyperspectral profiling of lipid signatures in colorectal cancer. Proc. Natl Acad. Sci. USA 111, 1216–1221 (2014).

    Google Scholar 

  45. Wang, X. et al. ESRGAN: enhanced super-resolution generative adversarial networks. In Proc. European Conference on Computer Vision – ECCV 2018 Workshops (eds Leal-Taixé, L. & Roth, S.) 63–79 (Springer, 2019).

  46. Lein, E. S. et al. Genome-wide atlas of gene expression in the adult mouse brain. Nature 445, 168–176 (2007).

    Google Scholar 

  47. Ledig, C. et al. Photo-realistic single image super-resolution using a generative adversarial network. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 105–114 (IEEE, 2017).

  48. Horé, A. & Ziou, D. Image quality metrics: PSNR vs. SSIM. In Proc. 20th International Conference on Pattern Recognition 2366–2369 (IEEE, 2010).

  49. Fang, L. et al. Deep learning-based point-scanning super-resolution imaging. Nat. Methods 18, 406–416 (2021).

    Google Scholar 

  50. Culley, S. et al. Quantitative mapping and minimization of super-resolution optical imaging artifacts. Nat. Methods 15, 263–266 (2018).

    Google Scholar 

  51. Aggarwal, C. C. Neural Networks and Deep Learning (Springer, 2018).

  52. Yasaka, K. & Abe, O. Deep learning and artificial intelligence in radiology: current applications and future directions. PLoS Med. 15, e1002707 (2018).

  53. Zhang, L. et al. Generalizing deep learning for medical image segmentation to unseen domains via deep stacked transformation. IEEE Trans. Med. Imag. 39, 2531–2540 (2020).

    Google Scholar 

  54. Large, R. & Knof, H. A comparison of negative and positive ion mass spectrometry. Org. Mass Spectrom. 11, 582–598 (1976).

    Google Scholar 

  55. Tyler, B. J. et al. Denoising of mass spectrometry images via inverse maximum signal factors analysis. Anal. Chem. 94, 2835–2843 (2022).

    Google Scholar 

  56. Belthangady, C. & Royer, L. A. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. Nat. Methods 16, 1215–1225 (2019).

    Google Scholar 

  57. Barbastathis, G., Ozcan, A. & Situ, G. On the use of deep learning for computational imaging. Optica 6, 921–943 (2019).

    Google Scholar 

  58. Moen, E. et al. Deep learning for cellular image analysis. Nat. Methods 16, 1233–1246 (2019).

    Google Scholar 

  59. Wang, H. et al. Deep learning enables cross-modality super-resolution in fluorescence microscopy. Nat. Methods 16, 103–110 (2019).

    MathSciNet  Google Scholar 

  60. Arentz, G. et al. in Advances in Cancer Research Vol. 134 (eds. Drake, R. R. & McDonnell, L. A.) 27–66 (Academic, 2017).

  61. Wolrab, D., Jirásko, R., Chocholoušková, M., Peterka, O. & Holčapek, M. Oncolipidomics: mass spectrometric quantitation of lipids in cancer research. Trends Anal. Chem. 120, 115480 (2019).

    Google Scholar 

  62. Palmer, A. et al. FDR-controlled metabolite annotation for high-resolution imaging mass spectrometry. Nat. Methods 14, 57–60 (2017).

    Google Scholar 

  63. Metodiev, M. D., Steven, R. T., Loizeau, X., Takats, Z. & Bunch, J. Modality agnostic model for spatial resolution in mass spectrometry imaging: application to MALDI MSI data. Anal. Chem. 93, 15295–15305 (2021).

    Google Scholar 

  64. Zubair, F., Prentice, B. M., Norris, J. L., Laibinis, P. E. & Caprioli, R. M. Standard reticle slide to objectively evaluate spatial resolution and instrument performance in imaging mass spectrometry. Anal. Chem. 88, 7302–7311 (2016).

    Google Scholar 

  65. Blau, Y., Mechrez, R., Timofte, R., Michaeli, T. & Zelnik-Manor, L. in Computer Vision – ECCV 2018 Workshops (eds Leal-Taixé, L. & Roth, S.) 334–355 (Springer, 2019).

  66. Ma, C., Yang, C.-Y., Yang, X. & Yang, M.-H. Learning a no-reference quality metric for single-image super-resolution. Comput. Vis. Image Und. 158, 1–16 (2017).

    Google Scholar 

  67. Mittal, A., Soundararajan, R. & Bovik, A. C. Making a ‘completely blind’ image quality analyzer. IEEE Signal Process. Lett. 20, 209–212 (2013).

    Google Scholar 

  68. Wang, X., Hou, Y., Hou, Z., Xiong, W. & Huang, G. Mass spectrometry imaging of brain cholesterol and metabolites with trifluoroacetic acid-enhanced desorption electrospray ionization. Anal. Chem. 91, 2719–2726 (2019).

    Google Scholar 

  69. Liao, T. Optical data for MOSR. Figshare https://doi.org/10.6084/m9.figshare.22639936.v1 (2023).

  70. USTC-xlab. USTC-xlab/MOSR: MOSR for MSI images (v2.0.1). Zenodo https://doi.org/10.5281/zenodo.7833505 (2023).

Download references

Acknowledgements

We thank National Key R&D Program of China (grant numbers 2021YFA0804900 and 2020YFA0112203), the National Natural Science Foundation of China (grant numbers 32225020, 91849206, 91942315, 92049304 and 32121002 to W.X. and 21974130 and 91849116 to H.Z.), the Strategic Priority Research Program of the Chinese Academy of Sciences (grant number XDB39050000), the Youth Innovation Promotion Association CAS, University Synergy Innovation Program of Anhui Province (grant number GXXT-2022-033), the Key Research Program of Frontier Science (CAS, grant number ZDBS-LY-SM002), the CAS Interdisciplinary Innovation Team (grant number JCTD-2018-20), the Fundamental Research Funds for the Central Universities, USTC Research Funds of the Double First-Class Initiative (grant numbers YD9100002001 to W.X. and YD9100002005 to H.Z.), the CAS Project for Young Scientists in Basic Research (grant number YSBR-013) and the CAS Collaborative Innovation Program of Hefei Science Center (grant number 2021HSC-CIP003).

Author information

Authors and Affiliations

  1. Department of Neurology, The First Affiliated Hospital of USTC, Hefei National Research Center for Physical Sciences at the Microscale, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China

    Tiepeng Liao, Zihao Ren, Man Yuan, Chenjian Miao, Qi Chen, Ziyi Wang, Lin Yi, Siyuan Ge, Wenwei Qian, Wei Xiong & Hongying Zhu

  2. Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China

    Zhaoliang Chai, Longfeng Shen, Zilei Wang, Wei Xiong & Hongying Zhu

  3. School of Information Science and Technology, University of Science and Technology of China, Hefei, China

    Junjie Li, Zhilin Li & Zilei Wang

  4. School of Computer Science and Technology, Huaibei Normal University, Huaibei, China

    Longfeng Shen

  5. CAS Key Laboratory of Brain Function and Disease, Hefei, China

    Wei Xiong & Hongying Zhu

  6. Anhui Province Key Laboratory of Biomedical Aging Research, Hefei, China

    Wei Xiong & Hongying Zhu

Authors
  1. Tiepeng Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zihao Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhaoliang Chai
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Man Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Chenjian Miao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Junjie Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Qi Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhilin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Ziyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Lin Yi
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Siyuan Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Wenwei Qian
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Longfeng Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Zilei Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Wei Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Hongying Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

H.Z. and W.X. designed research and supervised the project. T.L. performed the experiments with assistance from Z.R. and Z.C. T.L. analysed data with assistance from Z.L., J.L., M.Y., Q.C., Ziyi Wang, L.Y., S.G., L.S., Zilei Wang, C.M. and W.Q. T.L., H.Z. and W.X. wrote the paper.

Corresponding authors

Correspondence to Wei Xiong or Hongying Zhu.

Ethics declarations

Competing interests

The authors declare no competing interests.

Peer review

Peer review information

Nature Machine Intelligence thanks Dong Hye Ye and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liao, T., Ren, Z., Chai, Z. et al. A super-resolution strategy for mass spectrometry imaging via transfer learning. Nat Mach Intell 5, 656–668 (2023). https://doi.org/10.1038/s42256-023-00677-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-023-00677-7

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing