Abstract
Large amounts of labelled data are typically needed to develop robust deep learning methods for medical image analysis. However, issues related to the high costs of acquisition, time-consuming analysis, and patient privacy, have limited the number of publicly available datasets. Recently, latent diffusion models have been employed to generate synthetic data in several fields. Compared to other imaging modalities, the manipulation of 3D echocardiograms is particularly challenging due to the higher dimensionality and complex noise characteristics, and lack of objective ground truth. We present DIFF\(\cdot \)3, a latent diffusion model for synthesizing realistic 3D echocardiograms with high-quality labels from matching cardiovascular magnetic resonance imaging (CMR) scans. Using in vivo 3D echocardiograms from 134 participants and corresponding registered labels derived from CMR, source images and labels are initially compressed by a variational autoencoder, followed by diffusion in the latent space. Synthetic datasets were subsequently generated by randomly sampling from the latent distribution, and evaluated in terms of fidelity and diversity. DIFF\(\cdot \)3 may provide an effective and more efficient means of generating labelled 3D echocardiograms to supplement real patient data.
E. Ferdian and D. Zhao—Joint first authorship
M. P. Nash and A. A. Young—Joint senior authorship.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Code and data availability
Source code can be accessed in https://github.com/EdwardFerdian/diff-3. Synthetic datasets are available upon request.
References
Akrout, M., et al.: Diffusion-based Data Augmentation for Skin Disease Classification: Impact Across Original Medical Datasets to Fully Synthetic Images (2023). arXiv:2301.04802
Alessandrini, M., et al.: A pipeline for the generation of realistic 3D synthetic echocardiographic sequences: methodology and open-access database. IEEE Trans. Medical Imaging 34(7), 1436–1451 (2015). https://doi.org/10.1109/TMI.2015.2396632
Alzubaidi, L.,et al.: MedNet: pre-trained convolutional neural network model for the medical imaging tasks. CoRR abs/2110.0 (2021) arXiv:2110.06512
Chambon, P., Bluethgen, C., Langlotz, C.P., Chaudhari, A.: Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains (10 2022). arxiv:2210.04133
DuMont Schütte, A., et al.: Overcoming barriers to data sharing with medical image generation: a comprehensive evaluation. NPJ Digital Medicine 4(1), 141 (2021). https://doi.org/10.1038/s41746-021-00507-3
Evain, E., Faraz, K., Grenier, T., Garcia, D., Craene, M.D., Bernard, O.: A pilot study on convolutional neural networks for motion estimation from ultrasound images. IEEE Trans. Ultrason. Ferroelectr. Freq. Control 67(12), 2565–2573 (2020). https://doi.org/10.1109/TUFFC.2020.2976809
Gilbert, A., Marciniak, M., Rodero, C., Lamata, P., Samset, E., Mcleod, K.: Generating synthetic labeled data from existing anatomical models: an example with echocardiography segmentation. IEEE Trans. Med. Imaging 40(10), 2783–2794 (10 2021). https://doi.org/10.1109/TMI.2021.3051806, https://ieeexplore.ieee.org/document/9324763/
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs trained by a two time-scale update rule converge to a local nash equilibrium. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6629–6640. NIPS’17, Curran Associates Inc., Red Hook, NY, USA (2017)
Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. CoRR abs/2006.1 (2020). arXiv:2006.11239
Hong, S.: 3D-StyleGAN: a style-based generative adversarial network for generative modeling of three-dimensional medical images. In: Engelhardt, S., et al. (eds.) DGM4MICCAI/DALI -2021. LNCS, vol. 13003, pp. 24–34. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88210-5_3
Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: NNU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods 18(2), 203–211 (2021) https://doi.org/10.1038/s41592-020-01008-z
Jabri, A., Fleet, D.J., Chen, T.: Scalable Adaptive Computation for Iterative Generation. arXiv:2212.1 (2022)
Khader, F., et al.: Denoising diffusion probabilistic models for 3D medical image generation. Sci. Reports 13(1), 7303 (2023). https://doi.org/10.1038/s41598-023-34341-2
Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved Precision and Recall Metric for Assessing Generative Models. In: NeurIPS 2019. arXiv (2019). arXiv:1904.06991
Lucidrains: Denoising Diffusion Probabilistic Model, in Pytorch (2020). https://github.com/lucidrains/denoising-diffusion-pytorch
Moghadam, P.A., et al.: A Morphology Focused Diffusion Probabilistic Model for Synthesis of Histopathology Images (2022). arXiv:2209.13167
Østivk, A., et al.: Myocardial function imaging in echocardiography using deep learning. IEEE Trans. Med. Imaging 40(5), 1340–1351 (2021). https://doi.org/10.1109/TMI.2021.3054566
Pinaya, W.H.L., et al.: Brain imaging generation with latent diffusion models. In: Mukhopadhyay, A., Oksuz, I., Engelhardt, S., Zhu, D., Yuan, Y. (eds.) Deep Generative Models: Second MICCAI Workshop, DGM4MICCAI 2022, Held in Conjunction with MICCAI 2022, Singapore, September 22, 2022, Proceedings, pp. 117–126. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-18576-2_12
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-Resolution Image Synthesis with Latent Diffusion Models. CoRR abs/2112.1 (2021) arXiv:2112.10752
Song, J., Meng, C., Ermon, S.: Denoising Diffusion Implicit Models (2020). arXiv:2010.02502
Sun, L., Chen, J., Xu, Y., Gong, M., Yu, K., Batmanghelich, K.: Hierarchical amortized GAN for 3D high resolution medical image synthesis. IEEE J. Biomed. Health Inform. 26(8), 3966–3975 (2022). https://doi.org/10.1109/JBHI.2022.3172976
Thambawita, V., et al.: SinGAN-Seg: Synthetic training data generation for medical image segmentation. PLOS ONE 17(5), e0267976 (2022). https://doi.org/10.1371/journal.pone.0267976
Taigo, C., et al.: A data augmentation pipeline to generate synthetic labeled datasets of 3D echocardiography images using a GAN. IEEE Access 10, 98803–98815 (2022). https://doi.org/10.1109/ACCESS.2022.3207177
Trabucco, B., Doherty, K., Gurinas, M., Salakhutdinov, R.: Effective Data Augmentation With Diffusion Models (2023). arXiv:2302.07944
Xu, X., Kapse, S., Gupta, R., Prasanna, P.: ViT-DAE: Transformer-driven Diffusion Autoencoder for Histopathology Image Analysis (2023)
youngjung: improved-precision-and-recall-metric-pytorch (2019). https://github.com/youngjung/improved-precision-and-recall-metric-pytorch
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (2018). arXiv:1801.03924
Zhao, D., et al.: MITEA: A dataset for machine learning segmentation of the left ventricle in 3D echocardiography using subject-specific labels from cardiac magnetic resonance imaging. Front. Cardiovasc. Med. 9 (2023). https://doi.org/10.3389/fcvm.2022.1016703
Zhou, Y., et al.: A framework for the generation of realistic synthetic cardiac ultrasound and magnetic resonance imaging sequences from the same virtual patients. IEEE Trans. Med. Imaging 37(3), 741–754 (2018). https://doi.org/10.1109/TMI.2017.2708159
Acknowledgements
We gratefully acknowledge the staff at the Centre for Advanced MRI at the University of Auckland for their expertise and assistance with the imaging components of this study.
Funding
This study was funded by the Health Research Council of New Zealand (programme grant 17/608).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ferdian, E. et al. (2023). DIFF\(\cdot \)3: A Latent Diffusion Model for the Generation of Synthetic 3D Echocardiographic Images and Corresponding Labels. In: Wolterink, J.M., Svoboda, D., Zhao, C., Fernandez, V. (eds) Simulation and Synthesis in Medical Imaging. SASHIMI 2023. Lecture Notes in Computer Science, vol 14288. Springer, Cham. https://doi.org/10.1007/978-3-031-44689-4_13
Download citation
DOI: https://doi.org/10.1007/978-3-031-44689-4_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44688-7
Online ISBN: 978-3-031-44689-4
eBook Packages: Computer ScienceComputer Science (R0)
