Skip to main content
SpringerLink
Log in

Pre-trained Diffusion Models for Plug-and-Play Medical Image Enhancement

  • Conference paper
  • First Online:
Medical Image Computing and Computer Assisted Intervention – MICCAI 2023 (MICCAI 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14222))

  • 4291 Accesses

Abstract

Deep learning-based medical image enhancement methods (e.g., denoising and super-resolution) mainly rely on paired data and correspondingly the well-trained models can only handle one type of task. In this paper, we address the limitation with a diffusion model-based framework that mitigates the requirement of paired data and can simultaneously handle multiple enhancement tasks by one pre-trained diffusion model without fine-tuning. Experiments on low-dose CT and heart MR datasets demonstrate that the proposed method is versatile and robust for image denoising and super-resolution. We believe our work constitutes a practical and versatile solution to scalable and generalizable image enhancement.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bernard, O., et al.: Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans. Med. Imaging 37(11), 2514–2525 (2018)

    Article  Google Scholar 

  2. Blau, Y., Michaeli, T.: The perception-distortion tradeoff. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  3. Campello, V.M., et al.: Multi-centre, multi-vendor and multi-disease cardiac segmentation: the m&ms challenge. IEEE Trans. Med. Imaging 40(12), 3543–3554 (2021)

    Article  Google Scholar 

  4. Chaudhari, A.S., et al.: Super-resolution musculoskeletal MRI using deep learning. Magn. Reson. Med. 80(5), 2139–2154 (2018)

    Article  Google Scholar 

  5. Chen, H., et al.: Low-dose CT denoising with convolutional neural network. In: International Symposium on Biomedical Imaging, pp. 143–146 (2017)

    Google Scholar 

  6. Choi, J., Kim, S., Jeong, Y., Gwon, Y., Yoon, S.: ILVR: conditioning method for denoising diffusion probabilistic models. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14347–14356 (2021)

    Google Scholar 

  7. Chung, H., Kim, J., Mccann, M.T., Klasky, M.L., Ye, J.C.: Diffusion posterior sampling for general noisy inverse problems. In: International Conference on Learning Representations (2023)

    Google Scholar 

  8. Dhariwal, P., Nichol, A.: Diffusion models beat GANs on image synthesis. In: Advances in Neural Information Processing Systems, pp. 8780–8794 (2021)

    Google Scholar 

  9. Diwakar, M., Kumar, M.: A review on CT image noise and its denoising. Biomed. Sig. Process. Control 42, 73–88 (2018)

    Article  Google Scholar 

  10. Dmitry, U., Vedaldi, A., Victor, L.: Deep image prior. Int. J. Comput. Vis. 128(7), 1867–1888 (2020)

    Article  Google Scholar 

  11. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851 (2020)

    Google Scholar 

  12. Kawar, B., Elad, M., Ermon, S., Song, J.: Denoising diffusion restoration models. In: Advances in Neural Information Processing Systems (2022)

    Google Scholar 

  13. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: International Conference on Learning Representations (2014)

    Google Scholar 

  14. Lin, D.J., Johnson, P.M., Knoll, F., Lui, Y.W.: Artificial intelligence for MR image reconstruction: an overview for clinicians. J. Magn. Reson. Imaging 53(4), 1015–1028 (2021)

    Article  Google Scholar 

  15. Liu, X., Gong, C., Liu, Q.: Flow straight and fast: learning to generate and transfer data with rectified flow. In: International Conference on Learning Representations (2023)

    Google Scholar 

  16. Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., Zhu, J.: DPM-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. In: Advances in Neural Information Processing Systems (2022)

    Google Scholar 

  17. Mason, A., et al.: Comparison of objective image quality metrics to expert radiologists’ scoring of diagnostic quality of MR images. IEEE Trans. Med. Imaging 39(4), 1064–1072 (2019)

    Article  Google Scholar 

  18. Mazurowski, M.A., Buda, M., Saha, A., Bashir, M.R.: Deep learning in radiology: an overview of the concepts and a survey of the state of the art with focus on MRI. J. Magn. Reson. Imaging 49(4), 939–954 (2019)

    Article  Google Scholar 

  19. McCollough, C.H., et al.: Low-dose CT for the detection and classification of metastatic liver lesions: Results of the 2016 low dose CT grand challenge. Med. Phys. 44(10), e339–e352 (2017)

    Article  Google Scholar 

  20. Moen, T.R., et al.: Low-dose CT image and projection dataset. Med. Phys. 48(2), 902–911 (2021)

    Article  Google Scholar 

  21. Nichol, A.Q., Dhariwal, P.: Improved denoising diffusion probabilistic models. In: International Conference on Machine Learning, pp. 8162–8171 (2021)

    Google Scholar 

  22. Peng, C., Guo, P., Zhou, S.K., Patel, V.M., Chellappa, R.: Towards performant and reliable undersampled MR reconstruction via diffusion model sampling. In: Medical Image Computing and Computer Assisted Intervention, pp. 623–633 (2022)

    Google Scholar 

  23. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 234–241 (2015)

    Google Scholar 

  24. Sheikh, H.R., Bovik, A.C.: Image information and visual quality. IEEE Trans. Image Process. 15(2), 430–444 (2006)

    Article  Google Scholar 

  25. Song, J., Meng, C., Ermon, S.: Denoising diffusion implicit models. In: International Conference on Learning Representations (2021)

    Google Scholar 

  26. Tirer, T., Giryes, R.: Image restoration by iterative denoising and backward projections. IEEE Trans. Image Process. 28(3), 1220–1234 (2018)

    Article  MathSciNet  MATH  Google Scholar 

  27. Wang, S., et al.: The extreme cardiac MRI analysis challenge under respiratory motion (cmrxmotion). arXiv preprint arXiv:2210.06385 (2022)

  28. Wang, Y., Yu, J., Zhang, J.: Zero-shot image restoration using denoising diffusion null-space model. In: International Conference on Learning Representations (2023)

    Google Scholar 

  29. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)

    Article  Google Scholar 

  30. Xia, W., Lyu, Q., Wang, G.: Low-dose CT using denoising diffusion probabilistic model for 20x times speedup. arXiv preprint arXiv:2209.15136 (2022)

  31. Xie, Y., Li, Q.: Measurement-conditioned denoising diffusion probabilistic model for under-sampled medical image reconstruction. In: Medical Image Computing and Computer Assisted Intervention, pp. 655–664 (2022)

    Google Scholar 

  32. Yi, X., Babyn, P.: Sharpness-aware low-dose CT denoising using conditional generative adversarial network. J. Digit. Imaging 31, 655–669 (2018)

    Article  Google Scholar 

  33. You, C., et al.: Ct super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-circle). IEEE Trans. Med. Imaging 39(1), 188–203 (2020)

    Article  Google Scholar 

  34. Zhang, K., Li, Y., Zuo, W., Zhang, L., Van Gool, L., Timofte, R.: Plug-and-play image restoration with deep denoiser prior. IEEE Trans. Pattern Anal. Mach. Intell. 44(10), 6360–6376 (2021)

    Article  Google Scholar 

  35. Zhang, K., Zuo, W., Gu, S., Zhang, L.: Learning deep CNN denoiser prior for image restoration. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3929–3938 (2017)

    Google Scholar 

Download references

Acknowledgement

This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC, RGPIN-2020-06189 and DGECR-2020-00294), Canadian Institute for Advanced Research (CIFAR) AI Catalyst Grants, and CIFAR AI Chair programs. We thank the IDDPM [21], guided-diffusion [8], and DDNM [28] team, as their implementation served as an important basis for our work. We want to especially mention Jiwen Yu, who provided invaluable guidance and support. We also thank the organizers of AAPM Low Dose CT Grand Challenge [20], ACDC [1], M &Ms1-2 [3], and CMRxMothion [27] for making the datasets publicly available.

Author information

Authors and Affiliations

  1. Peter Munk Cardiac Centre, University Health Network, Toronto, Canada

    Jun Ma & Bo Wang

  2. Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, Canada

    Jun Ma & Bo Wang

  3. Vector Institute for Artificial Intelligence, Toronto, Canada

    Jun Ma & Bo Wang

  4. Department of Information Technology and Electrical Engineering, ETH Zürich, Zürich, Switzerland

    Yuanzhi Zhu

  5. Department of Electrical Engineering, Yale University, New Haven, USA

    Chenyu You

  6. Department of Computer Science, University of Toronto, Toronto, Canada

    Bo Wang

  7. AI Hub, University Health Network, Toronto, Canada

    Bo Wang

Authors
  1. Jun Ma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yuanzhi Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Chenyu You
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bo Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Wang .

Editor information

Editors and Affiliations

  1. Icahn School of Medicine, Mount Sinai, NYC, NY, USA, Tel Aviv University, Tel Aviv, Israel

    Hayit Greenspan

  2. Emory University, Atlanta, GA, USA

    Anant Madabhushi

  3. Queen's University, Kingston, ON, Canada

    Parvin Mousavi

  4. The University of British Columbia, Vancouver, BC, Canada

    Septimiu Salcudean

  5. Yale University, New Haven, CT, USA

    James Duncan

  6. IBM Research, San Jose, CA, USA

    Tanveer Syeda-Mahmood

  7. Johns Hopkins University, Baltimore, MD, USA

    Russell Taylor

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 3004 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ma, J., Zhu, Y., You, C., Wang, B. (2023). Pre-trained Diffusion Models for Plug-and-Play Medical Image Enhancement. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14222. Springer, Cham. https://doi.org/10.1007/978-3-031-43898-1_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-43898-1_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-43897-4

  • Online ISBN: 978-3-031-43898-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation