Abstract
Generative image reconstruction algorithms such as measurement conditioned diffusion models are increasingly popular in the field of medical imaging. These powerful models can transform low signal-to-noise ratio (SNR) inputs into outputs with the appearance of high SNR. However, the outputs can have a new type of error called hallucinations. In medical imaging, these hallucinations may not be obvious to a Radiologist but could cause diagnostic errors. Generally, hallucination refers to error in estimation of object structure caused by a machine learning model, but there is no widely accepted method to evaluate hallucination magnitude. In this work, we propose a new image quality metric called the hallucination index. Our approach is to compute the Hellinger distance from the distribution of reconstructed images to a zero hallucination reference distribution. To evaluate our approach, we conducted a numerical experiment with electron microscopy images, simulated noisy measurements, and applied diffusion based reconstructions. We sampled the measurements and the generative reconstructions repeatedly to compute the sample mean and covariance. For the zero hallucination reference, we used the forward diffusion process applied to ground truth. Our results show that higher measurement SNR leads to lower hallucination index for the same apparent image quality. We also evaluated the impact of early stopping in the reverse diffusion process and found that more modest denoising strengths can reduce hallucination. We believe this metric could be useful for evaluation of generative image reconstructions or as a warning label to inform radiologists about the degree of hallucinations in medical images.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Barbano, R., et al.: Steerable conditional diffusion for out-of-distribution adaptation in imaging inverse problems. arXiv preprint arXiv:2308.14409 (2023)
Bhadra, S., Kelkar, V.A., Brooks, F.J., Anastasio, M.A.: On hallucinations in tomographic image reconstruction. IEEE Trans. Med. Imaging 40(11), 3249–3260 (2021)
Buban, J.P., Ramasse, Q., Gipson, B., Browning, N.D., Stahlberg, H.: High-resolution low-dose scanning transmission electron microscopy. J. Electron Microsc. 59(2), 103–112 (2010)
Chu, L.C., Anandkumar, A., Shin, H.C., Fishman, E.K.: The potential dangers of artificial intelligence for radiology and radiologists. J. Am. Coll. Radiol. 17(10), 1309–1311 (2020)
Cohen, J.P., Luck, M., Honari, S.: Distribution matching losses can hallucinate features in medical image translation. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part I, pp. 529–536. Springer (2018). https://doi.org/10.1007/978-3-030-00928-1_60
Consortium, M., et al.: Functional connectomics spanning multiple areas of mouse visual cortex. BioRxiv, 2021–07 (2021)
Denker, A., Schmidt, M., Leuschner, J., Maass, P., Behrmann, J.: Conditional normalizing flows for low-dose computed tomography image reconstruction. arXiv preprint arXiv:2006.06270 (2020)
Hajij, M., Zamzmi, G., Paul, R., Thukar, L.: Normalizing flow for synthetic medical images generation. In: 2022 IEEE Healthcare Innovations and Point of Care Technologies (HI-POCT), pp. 46–49. IEEE (2022)
Kazerouni, A., et al.: Diffusion models in medical imaging: a comprehensive survey. Med. Image Anal. 102846 (2023)
Khader, F., et al.: Denoising diffusion probabilistic models for 3D medical image generation. Sci. Rep. 13(1), 7303 (2023)
Mardani, M., et al.: Deep generative adversarial neural networks for compressive sensing MRI. IEEE Trans. Med. Imaging 38(1), 167–179 (2018)
Nikulin, M.S., et al.: Hellinger distance. Encycl. Math. 78 (2001)
Song, Y., Shen, L., Xing, L., Ermon, S.: Solving inverse problems in medical imaging with score-based generative models. arXiv preprint arXiv:2111.08005 (2021)
Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A., Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 (2020)
Suganthi, K., et al.: Review of medical image synthesis using GAN techniques. In: ITM Web of Conferences. vol. 37, pp. 01005. EDP Sciences (2021)
Teneggi, J., Tivnan, M., Stayman, W., Sulam, J.: How to trust your diffusion model: a convex optimization approach to conformal risk control. In: International Conference on Machine Learning, pp. 33940–33960. PMLR (2023)
Tivnan, M., et al.: Fourier diffusion models: a method to control MTF and NPS in score-based stochastic image generation. arXiv preprint arXiv:2303.13285 (2023)
Trampert, P., et al.: How should a fixed budget of dwell time be spent in scanning electron microscopy to optimize image quality? Ultramicroscopy 191, 11–17 (2018)
Xie, Y., Li, Q.: Measurement-conditioned denoising diffusion probabilistic model for under-sampled medical image reconstruction. In: International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 655–664. Springer (2022). https://doi.org/10.1007/978-3-031-16446-0_62
Yi, X., Walia, E., Babyn, P.: Generative adversarial network in medical imaging: a review. Med. Image Anal. 58, 101552 (2019)
Zhou, T., Li, Q., Lu, H., Cheng, Q., Zhang, X.: Gan review: models and medical image fusion applications. Inf. Fusion 91, 134–148 (2023)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Supplementary material 1 (mp4 3736 KB)
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tivnan, M., Yoon, S., Chen, Z., Li, X., Wu, D., Li, Q. (2024). Hallucination Index: An Image Quality Metric for Generative Reconstruction Models. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15010. Springer, Cham. https://doi.org/10.1007/978-3-031-72117-5_42
Download citation
DOI: https://doi.org/10.1007/978-3-031-72117-5_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72116-8
Online ISBN: 978-3-031-72117-5
eBook Packages: Computer ScienceComputer Science (R0)
