Abstract
Medical education is essential for providing the best patient care in medicine, but creating educational materials using real-world data poses many challenges. For example, the diagnosis and treatment of a disease can be affected by small but significant differences in medical images; however, collecting images to highlight such differences is often costly. Therefore, medical image editing, which allows users to create their intended disease characteristics, can be useful for education. However, existing image-editing methods typically require manually annotated labels, which are labor-intensive and often challenging to represent fine-grained anatomical elements precisely. Herein, we present a novel algorithm for editing anatomical elements using segmentation labels acquired through self-supervised learning. Our self-supervised segmentation achieves pixel-wise clustering under the constraint of invariance to photometric and geometric transformations, which are assumed not to change the clinical interpretation of anatomical elements. The user then edits the segmentation map to produce a medical image with the intended detailed findings. Evaluation by five expert physicians demonstrated that the edited images appeared natural as medical images and that the disease characteristics were accurately reproduced.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, Y., et al.: Generative adversarial networks in medical image augmentation: a review. Comput. Biol. Med. 144, 105382 (2022). https://doi.org/10.1016/j.compbiomed.2022.105382
Dhariwal, P., Nichol, A.: Diffusion Models Beat GANs on Image Synthesis. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (NeurIPS). vol. 34, pp. 8780–8794. Curran Associates, Inc. (2021). https://proceedings.neurips.cc/paper_files/paper/2021/file/49ad23d1ec9fa4bd8d77d02681df5cfa-Paper.pdf
Duong, M.T., et al.: Artificial intelligence for precision education in radiology. Br. J. Radiol. 92(1103), 20190389 (2019). https://doi.org/10.1259/BJR.20190389
Fetty, L., et al.: Latent space manipulation for high-resolution medical image synthesis via the StyleGAN. Z. Med. Phys. 30(4), 305–314 (2020). https://doi.org/10.1016/j.zemedi.2020.05.001
Goodfellow, I., et al.: Generative Adversarial Nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems (NIPS). vol. 27 (2014). https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
Hyun Cho, J., Mall, U., Bala, K., Hariharan, B.: PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clustering. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16789–16799 (2021). https://doi.org/10.1109/CVPR46437.2021.01652
Ji, X., Vedaldi, A., Henriques, J.: Invariant Information Clustering for Unsupervised Image Classification and Segmentation. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9864–9873 (2019). https://doi.org/10.1109/ICCV.2019.00996
Jiang, L., Dai, B., Wu, W., Loy, C.C.: Focal Frequency Loss for Image Reconstruction and Synthesis. In: IEEE/CVF International Conference on Computer Vision (ICCV), pp. 13899–13909 (2021). https://doi.org/10.1109/ICCV48922.2021.01366
Kaiser, L., Roy, A., Vaswani, A., Parmar, N., Bengio, S., Uszkoreit, J., Shazeer, N.: Fast Decoding in Sequence Models using Discrete Latent Variables. In: Dy, J., Krause, A. (eds.) Proceedings of the 35th International Conference on Machine Learning (ICML). Proceedings of Machine Learning Research, 80, pp. 2390–2399 (2018). https://proceedings.mlr.press/v80/kaiser18a.html
Li, H., Wei, D., Cao, S., Ma, K., Wang, L., Zheng, Y.: Superpixel-guided label softening for medical image segmentation. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 227–237. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_23
Ling, H., Kreis, K., Li, D., Kim, S.W., Torralba, A., Fidler, S.: EditGAN: High-Precision Semantic Image Editing. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (NeurIPS) vol. 34, pp. 16331–16345 (2021). https://proceedings.neurips.cc/paper/2021/file/880610aa9f9de9ea7c545169c716f477-Paper.pdf
Paszke, A.,et al.: PyTorch: An Imperative Style, High-Performance Deep Learning Library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems (NeurIPS), vol. 32, pp. 8024–8035 (2019). https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html
Ronneberger, O., Fischer, P., Brox, T.: U-Net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28
Saboo, A., Gyawali, P.K., Shukla, A., Sharma, M., Jain, N., Wang, L.: Latent-optimization based disease-aware image editing for medical image augmentation. In: 32nd British Machine Vision Conference (BMVC). p. 181 (2021). https://www.bmvc2021-virtualconference.com/assets/papers/0840.pdf
Sasuga, S., et al.: Image Synthesis-Based Late Stage Cancer Augmentation and Semi-supervised Segmentation for MRI Rectal Cancer Staging. In: Nguyen, H.V., Huang, S.X., Xue, Y. (eds.) Data Augmentation, Labelling, and Imperfections. pp. 1–10. Springer Nature Switzerland, Cham (2022). https://link.springer.com/chapter/10.1007/978-3-031-17027-0_1
Schonfeld, E., Schiele, B., Khoreva, A.: A U-Net based discriminator for generative adversarial networks. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8207–8216 (2020). https://doi.org/10.1109/CVPR42600.2020.00823
Thermos, S., Liu, X., O’Neil, A., Tsaftaris, S.A.: Controllable cardiac synthesis via disentangled anatomy arithmetic. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12903, pp. 160–170. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87199-4_15
Tiago, C., Snare, S.R., Šprem, J., McLeod, K.: A domain translation framework with an adversarial denoising diffusion model to generate synthetic datasets of echocardiography images. IEEE Access 11, 17594–17602 (2023). https://doi.org/10.1109/ACCESS.2023.3246762
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 586–595 (2018). https://doi.org/10.1109/CVPR.2018.00068
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kobayashi, K. et al. (2023). Towards AI-Driven Radiology Education: A Self-supervised Segmentation-Based Framework for High-Precision Medical Image Editing. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14221. Springer, Cham. https://doi.org/10.1007/978-3-031-43895-0_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-43895-0_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43894-3
Online ISBN: 978-3-031-43895-0
eBook Packages: Computer ScienceComputer Science (R0)
