Abstract
Fundus image generation can serve training data diversity as well as clinical and medical education. To solve the problem of the controllability and flexibility of fundus image generation with multiple lesion features, we propose a controllable fundus image generation model (CFIGGAN) based on conditional generative adversarial networks (GAN) for medical data augmentation. The least square loss and the perceptual loss term are added to the final loss to make the generated images more realistic, and the spectral normalization is used as the normalization method of the discriminator to make the training process more stable. In the two-stage training of the model, the vascular tree image concatenates the real and generated images as positive and negative samples to train the model. CFIGGAN can generate diseased fundus images by using the annotations of vascular tree, field of vision(FOV), DR-related lesions as input and controlling the morphology of four types of lesions. Qualitative experimental evaluation shows that the fundus images generated by our model are clear and realistic and close to the real image data distribution. Quantitative experimental evaluation shows that the combination of the spectral norm and the perceptual loss can improve visual observation and quantitative indices, and data augmentation by image generation can further increase the classification accuracy. More importantly, CFIGGAN achieve the controllability of fundus image generation corresponding to DR-related lesions, and the proposed method can be extended to medical images generation of other diseases for broader prospects.













Similar content being viewed by others
Data Availibility Statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.
References
Fiorini S, Ballerini L, Trucco E, Ruggeri A (2014) Automatic generation of synthetic retinal fundus images. In: MIUA. p 7–12 https://diglib.eg.org/bitstream/handle/10.2312/stag.20141238.041-044/041-044.pdf?sequence=1 &isAllowed=yhttps://doi.org/10.1016/j.procs.2016.07.010
Costa P, Galdran A, Meyer MI, Abramoff MD, Niemeijer M, Mendonça AM, Campilho A (2017) Towards adversarial retinal image synthesis. arXiv:1701.08974https://doi.org/10.48550/arXiv.1701.08974
Ronneberger O, Fischer P, Brox T (2015) U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical image computing and computer-assisted intervention. Springer p 234–241. https://doi.org/10.1007/978-3-319-24574-4_28https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28
Costa P, Galdran A, Meyer MI, Niemeijer M, Abrámoff M, Mendonça AM, Campilho A (2017) End-to-end adversarial retinal image synthesis. IEEE Trans Med Imaging 37(3)781–791. https://doi.org/10.1109/TMI.2017.2759102https://ieeexplore.ieee.org/document/8055572
Mao X, Li Q, Xie H, Lau RY, Wang Z, Paul Smolley S (2017) Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision. p 2794–2802. https://doi.org/10.48550/arXiv.1611.04076arXiv:1611.04076
Isola P, Zhu J-Y, Zhou T, Efros AA (2017) Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p 1125–1134. https://doi.org/10.48550/arXiv.1611.07004arXiv:1611.07004v3
Miyato T, Kataoka T, Koyama M, Yoshida Y (2018) Spectral normalization for generative adversarial networks. arXiv:1802.05957https://doi.org/10.48550/arXiv.1802.05957
Srivastava A, Aggarwal AK (2018) Medical image fusion in spatial and transform domain: a comparative analysis. In: Handbook of research on advanced concepts in real-time image and video processing. IGI global p 281–300. https://doi.org/10.4018/978-1-5225-2848-7.ch011https://www.igi-global.com/chapter/medical-image-fusion-in-spatial-and-transform-domain/186281
Thukral R, Kumar A, Arora A, et al. (2020) Effects of different radiations of electromagnetic spectrum on human health. In: 2020 IEEE International Students’ Conference on Electrical, Electronics and Computer Science (SCEECS). IEEE p 1–6. https://doi.org/10.1109/SCEECS48394.2020.154https://ieeexplore.ieee.org/abstract/document/9086961
Thukral R, Kumar A, Arora A, et al. (2019) Effect of different thresholding techniques for denoising of emg signals by using different wavelets. In: 2019 2nd International conference on intelligent communication and computational techniques (ICCT). IEEE p 161–165. https://doi.org/10.1109/ICCT46177.2019.8969036https://ieeexplore.ieee.org/document/8969036
Srivastava A, Singhal V, Aggarawal AK (2017) Comparative analysis of multimodal medical image fusion using pca and wavelet transforms. Int J Latest Technol Eng Manag Appl Sci (IJLTEMAS) VI https://www.academia.edu/download/53002072/115-118.pdf
Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial nets. Adv Neural Inf Process Sys 27 https://proceedings.neurips.cc/paper_files/paper/2014/hash/5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html
Singh NK, Raza K (2021) Medical image generation using generative adversarial networks: a review. Health Inf Comput Perspec Healthcare 77–96. https://doi.org/10.1007/978-981-15-9735-0_5https://link.springer.com/chapter/10.1007/978-981-15-9735-0_5
Radford A, Metz L, Chintala S (2015) Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv:1511.06434https://doi.org/10.48550/arXiv.1511.06434
Ring M, Schlör D, Landes D, Hotho A (2019) Flow-based network traffic generation using generative adversarial networks. Comput Secur 82:156–172. https://doi.org/10.1016/j.cose.2018.12.012https://www.sciencedirect.com/science/article/pii/S0167404818308393
Karras T, Aila T, Laine S, Lehtinen J (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv:1710.10196https://doi.org/10.48550/arXiv.1710.10196
Chuquicusma MJ, Hussein S, Burt J, Bagci U (2018) How to fool radiologists with generative adversarial networks? a visual turing test for lung cancer diagnosis. In: 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018). IEEE p 240–244. https://doi.org/10.1109/ISBI.2018.8363564https://ieeexplore.ieee.org/document/8363564
Frid-Adar M, Diamant I, Klang E, Amitai M, Goldberger J, Greenspan H (2018) Gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification. Neurocomputing 321:321–331. https://doi.org/10.1016/j.neucom.2018.09.013https://www.sciencedirect.com/science/article/pii/S0925231218310749
Bermudez C, Plassard AJ, Davis LT, Newton AT, Resnick SM, Landman BA (2018) Learning implicit brain mri manifolds with deep learning. In: Medical Imaging 2018: Image Processing, Vol. 10574. International Society for Optics and Photonics p 105741L. https://doi.org/10.1117/12.2293515https://www.spiedigitallibrary.org/conference-proceedings-of-spie/10574/105741L/Learning-implicit-brain-MRI-manifolds-with-deep-learning/10.1117/12.2293515.short?SSO=1
Jin D, Xu Z, Tang Y, Harrison AP, Mollura DJ (2018) Ct-realistic lung nodule simulation from 3d conditional generative adversarial networks for robust lung segmentation. In: International conference on medical image computing and computer-assisted intervention. Springer p 732–740. https://doi.org/10.1007/978-3-030-00934-2_81https://link.springer.com/chapter/10.1007/978-3-030-00934-2_81
Mok TC, Chung A (2018) Learning data augmentation for brain tumor segmentation with coarse-to-fine generative adversarial networks. In: International MICCAI brainlesion workshop. Springer p 70–80. https://doi.org/10.1007/978-3-030-11723-8_7https://link.springer.com/chapter/10.1007/978-3-030-11723-8_7
Chartsias A, Joyce T, Dharmakumar R, Tsaftaris SA (2017) Adversarial image synthesis for unpaired multi-modal cardiac data. In: International workshop on simulation and synthesis in medical imaging. Springer p 3–13. https://doi.org/10.1007/978-3-319-68127-6_1https://link.springer.com/chapter/10.1007/978-3-319-68127-6_1
Maspero M, Savenije MH, Dinkla AM, Seevinck PR, Intven MP, Jurgenliemk-Schulz IM, Kerkmeijer LG, van den Berg CA (2018) Dose evaluation of fast synthetic-ct generation using a generative adversarial network for general pelvis mr-only radiotherapy. Phys Med Biol 63(18):185001. https://doi.org/10.1088/1361-6560/aada6dhttps://iopscience.iop.org/article/10.1088/1361-6560/aada6d/meta
Yang H, Sun J, Carass A, Zhao C, Lee J, Xu Z, Prince J (2018) Unpaired brain mr-to-ct synthesis using a structure-constrained cyclegan. In: Deep learning in medical image analysis and multimodal learning for clinical decision support. Springer p 174–182. https://doi.org/10.1007/978-3-030-00889-5_20https://link.springer.com/chapter/10.1007/978-3-030-00889-5_20
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z, et al. (2017) Photo-realistic single image super-resolution using a generative adversarial network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. p 4681–4690. https://doi.org/10.48550/arXiv.1609.04802https://arxiv.org/abs/1609.04802v5
Wang X, Yu K, Wu S, Gu J, Liu Y, Dong C, Qiao Y, Change Loy C (2018) Esrgan: Enhanced super-resolution generative adversarial networks. In: Proceedings of the European conference on computer vision (ECCV) workshops. https://doi.org/10.48550/arXiv.1809.00219arXiv:1809.00219v2
Guan J, Pan C, Li S, Yu D (2019) Srdgan: learning the noise prior for super resolution with dual generative adversarial networks. https://doi.org/10.48550/arXiv.1903.11821arXiv:1903.11821
Zhu JY, Park T, Isola P, Efros AA (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. In: Proceedings of the IEEE international conference on computer vision. p 2223–2232. https://doi.org/10.48550/arXiv.1703.10593arXiv:1703.10593v6
Yi Z, Zhang H, Tan P, Gong M (2017) Dualgan: Unsupervised dual learning for image-to-image translation. In: Proceedings of the IEEE international conference on computer vision. p 2849–2857. https://doi.org/10.48550/arXiv.1704.02510arXiv:1704.02510
Lin K, Li D, He X, Zhang Z, Sun M-T (2017) Adversarial ranking for language generation. Adv Neural Inf Process Sys 30 https://proceedings.neurips.cc/paper_files/paper/2017/file/bf201d5407a6509fa536afc4b380577e-Paper.pdf
Mogren O (2016) C-rnn-gan: Continuous recurrent neural networks with adversarial training. https://doi.org/10.48550/arXiv.1611.09904arXiv:1611.09904
Vondrick C, Pirsiavash H, Torralba A (2016) Generating videos with scene dynamics. Adv Neural Inf Process Sys 29 https://proceedings.neurips.cc/paper/2016/file/04025959b191f8f9de3f924f0940515f-Paper.pdf
Diaz-Pinto A, Colomer A, Naranjo V, Morales S, Xu Y, Frangi AF (2019) Retinal image synthesis and semi-supervised learning for glaucoma assessment. IEEE Trans Med Imaging 38(9)2211–2218. https://doi.org/10.1109/TMI.2019.2903434https://ieeexplore.ieee.org/document/8662628
Deshmukh A, Sivaswamy J (2019) Synthesis of optical nerve head region of fundus image. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE p 583–586. https://doi.org/10.1109/ISBI.2019.8759414https://ieeexplore.ieee.org/document/8759414
Wang X, Xu M, Li L, Wang Z, Guan Z (2019) Pathology-aware deep network visualization and its application in glaucoma image synthesis. In: International conference on medical image computing and computer-assisted intervention. Springer p 423–431. https://doi.org/10.1007/978-3-030-32239-7_47https://link.springer.com/chapter/10.1007/978-3-030-32239-7_47
Zhao H, Li H, Maurer-Stroh S, Cheng L (2018) Synthesizing retinal and neuronal images with generative adversarial nets. Med Image Anal 49:14–26. https://doi.org/10.1016/j.media.2018.07.001https://www.sciencedirect.com/science/article/pii/S1361841518304596
Zhou Y, He X, Cui S, Zhu F, Liu L, Shao L (2019) High-resolution diabetic retinopathy image synthesis manipulated by grading and lesions. In: International conference on medical image computing and computer-assisted intervention. Springer p 505–513. https://doi.org/10.1007/978-3-030-32239-7_56https://link.springer.com/chapter/10.1007/978-3-030-32239-7_56
Burlina PM, Joshi N, Pacheco KD, Liu TA, Bressler NM (2019) Assessment of deep generative models for high-resolution synthetic retinal image generation of age-related macular degeneration. JAMA Ophthalmol 137(3)258–264 https://doi.org/10.1001/jamaophthalmol.2018.6156https://jamanetwork.com/journals/jamaophthalmology/article-abstract/2720489
Altaf F, Islam SM, Akhtar N, Janjua NK (2019) Going deep in medical image analysis: concepts, methods, challenges, and future directions. IEEE Access 7:99540–99572. https://doi.org/10.1109/ACCESS.2019.2929365https://ieeexplore.ieee.org/document/8764525
Mahapatra D, Bozorgtabar B, Hewavitharanage S, Garnavi R (2017) Image super resolution using generative adversarial networks and local saliency maps for retinal image analysis, in: International conference on medical image computing and computer-assisted intervention. Springer p 382–390. https://doi.org/10.1007/978-3-319-66179-7_44https://link.springer.com/chapter/10.1007/978-3-319-66179-7_44
Staal J, Abràmoff MD, Niemeijer M, Viergever MA, Van Ginneken B (2004) Ridge-based vessel segmentation in color images of the retina. IEEE Trans Med Imaging 23(4):501–509. https://doi.org/10.1109/TMI.2004.825627https://ieeexplore.ieee.org/document/1282003
J Z J (2018) Laddernet: Multi-path networks based on u-net for medical image segmentation. https://doi.org/10.48550/arXiv.1810.07810arXiv:1810.07810
Li T, Gao Y, Wang K, Guo S, Liu H, Kang H (2019) Diagnostic assessment of deep learning algorithms for diabetic retinopathy screening. Inf Sci 501:511–522. https://doi.org/10.1016/j.ins.2019.06.011https://www.sciencedirect.com/science/article/pii/S0020025519305377
Hore A, Ziou D (2010) Image quality metrics: Psnr vs. ssim. In: 2010 20th international conference on pattern recognition. IEEE p 2366–2369. https://doi.org/10.1109/ICPR.2010.579https://ieeexplore.ieee.org/document/5596999
Heusel M, Ramsauer H, Unterthiner T, Nessler B, Hochreiter S (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv Neural Inf Process Sys 30. https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf
Bińkowski M, Sutherland DJ, Arbel M, Gretton A (2018) Demystifying mmd gans. https://doi.org/10.48550/arXiv.1801.01401arXiv:1801.01401
Guibas JT, Virdi TS, Li PS (2017) Synthetic medical images from dual generative adversarial networks. https://doi.org/10.48550/arXiv.1709.01872arXiv:1709.01872
Karras T, Laine S, Aittala M, Hellsten J, Lehtinen J, Aila T (2020) Analyzing and improving the image quality of stylegan. Proc IEEE/CVF Conf Comput Vis Pattern Recognit 2020:8110–8119
Dhariwal P, Nichol A (2021) Diffusion models beat gans on image synthesis. Adv Neural Inf Process Syst 34:8780–8794
Niu Y, Gu L, Zhao Y, Lu F (2021) Explainable diabetic retinopathy detection and retinal image generation. IEEE J Biomed Health Inform 26(1):44–55
Yu Z, Xiang Q, Meng J, Kou C, Ren Q, Lu Y (2019) Retinal image synthesis from multiple-landmarks input with generative adversarial networks. Biomed Eng Online 18(1):1–15
Acknowledgements
This work was supported by the National Natural Science Foundation of China under Grant 82071995, the Key Research And Development Program of Jilin Province under Grant 20220201141GX and the Natural Science Foundation of Jilin Province under Grant 20200201292JC.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of interest
The authors declare that they have no conflicts of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Guo, X., Li, X., Lin, Q. et al. Controllable fundus image generation based on conditional generative adversarial networks with mask guidance. Multimed Tools Appl 83, 46065–46085 (2024). https://doi.org/10.1007/s11042-023-17280-y
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-023-17280-y