Generating future fundus images for early age-related macular degeneration based on generative adversarial networks

doi:10.1016/j.cmpb.2022.106648

Computer Methods and Programs in Biomedicine

Volume 216, April 2022, 106648

https://doi.org/10.1016/j.cmpb.2022.106648 Get rights and content

Highlights

•
The first attempt to generate future fundus images based on current fundus images for early age-related macular degeneration (AMD) patients based on deep learning model.
•
Exploit the drusen segmentation mask for improving the performance.
•
Introduce a GAN-based model with two discriminators for preserving the identity and utilizing drusen masks.
•
Develop a fundus dataset for our task.

Abstract

Background and objective: Age-related macular degeneration (AMD) is one of the most common diseases that can lead to blindness worldwide. Recently, various fundus image analyzing studies are done using deep learning methods to classify fundus images to aid diagnosis and monitor AMD disease progression. But until now, to the best of our knowledge, no attempt was made to generate future synthesized fundus images that can predict AMD progression. In this paper, we developed a deep learning model using fundus images for AMD patients with different time elapses to generate synthetic future fundus images.

Method: We exploit generative adversarial networks (GANs) with additional drusen masks to maintain the pathological information. The dataset included 8196 fundus images from 1263 AMD patients. A proposed GAN-based model, called Multi-Modal GAN (MuMo-GAN), was trained to generate synthetic predicted-future fundus images.

Results: The proposed deep learning model indicates that the additional drusen masks can help to learn the AMD progression. Our model can generate future fundus images with appropriate pathological features. The drusen development over time is depicted well. Both qualitative and quantitative experiments show that our model is more efficient to monitor the AMD disease as compared to other studies.

Conclusion: This study could help individualized risk prediction for AMD patients. Compared to existing methods, the experimental results show a significant improvement in terms of tracking the AMD stage in both image-level and pixel-level.

Introduction

Age-related macular degeneration (AMD) is a disease of the central retina [12]. And one of the most common diseases that can cause blind worldwidely and expected to increase exponentially due to the aging population. Like other diseases, predicting the progression of AMD and identifying eyes with the high risk of progression is the crucial step to prevent blindness and minimize the socio-economic loss due to AMD. There are numerous researches on disease course and risk factors of AMD progression, but almost all study results are presented as numeric figures for general AMD population. For example, the 5-year progression of early AMD in Koreans is reported to be about 20% [21]. However, in the real clinical settings, for each AMD patients, the risk of progression differs based on their fundus findings like drusen numbers, size, location and accompanying pigmentary changes. So, for AMD patients, individualized risk estimation and graphic representation for their eyes can be important as much as treatment itself. Nevertheless, individualized risk prediction based on fundus findings AMD still remains an unresolved problem. Thus, we proposed a deep learning-based method in order to make future synthesized fundus images from the current fundus images.

There are several key problems in this task. Firstly, the future synthesized image must be realistic and it has to show accurate changes in comparison with the input image. The output image is supposed to have the same important feautures with the corresponding ground truth images. For AMD disease, the main factor for diagnosis are drusen. However, in the early stage of AMD, drusen are usually small and unclear. Therefore, it is hard to monitor the changes of drusen over time. In addition, the fundus image series of the same patient with different times can be taken by different conditions. It can cause not only the changes of position but also the style of fundus images. Moreover, to train a model to predict future images, we need a dataset that contains a series of images from the same subject at different times (e.g. AFAD dataset [14] for face aging task).

One of the possible approaches for this research is based on deep learning. Nowadays, it is true that deep learning models have outperformed over classical approaches in many tasks: classify/generate image, text, and speech. Especially, Generative Adversarial Networks (GANs) could be used to generate realistic images [4], [11]. GANs is a deep learning model for learning the distribution of data. The Conditional GANs (cGANs) [7] is a variant of GANs. It takes an image as an input instead of a random vector and then generates an output image. cGANs is popular in many applications such as image segmentation [19], style transfer [22], image-to-image translation [7], etc. Therefore, in this paper, we proposed a GAN-based model to generate future fundus images for AMD patients. To the best of our knowledge, this study is the first attempt to generate future fundus images based on current fundus images and the first attempt to develop an individualized and visually represented risk prediction algorithm. Overall, our contributions in this research are as follows:

•
We introduced a novel deep learning framework that can generate synthetic predicted-future fundus images for early AMD patients. Given the series of fundus images, our model predicts the future fundus image. By capturing AMD factors, the proposed model can provide helpful information of AMD progression and can be used for monitoring the disease.
•
We proposed a new GAN model for this task. We combine GAN and a segmentation model to extract and analyze the AMD factors. Our model exploits drusen masks along with image data to impose important features. Moreover, an extra discriminator is applied for preserving identity.
•
We develop a fundus dataset that fits into our task. The dataset contains fundus images of the 1263 AMD patients. Each patient has several fundus images taken at various times.

Section snippets

Related works

Until now, the most important features to monitor AMD progression are changes in fundus images and Optical Coherence Tomography (OCT) findings. Although there are recent studies regarding OCT images to predict early AMD conversion to wet age-related macular degeneration using deep learning [27], as same as previous risk estimation studies, most of these study results are presented as numeric figures and general estimates. Moreover, acquiring fundus images is more accessible and much cheaper

Image preprocessing

Our pre-processing steps are illustrated in Fig. 1. The important of fundus image for AMD diagnosis is the center region. Therefore, we decide to use only that region for experiments. In order to crop images, the first step is to locate the optic disc position by using the pre-trained model called DiscSeg [2], [3]. DiscSeg is the U-shape convolutional network trained to predict the position of optic disc at the given fundus image. After that, the region of interest (ROI) is cropped based on the

The experiment settings

The dataset includes 8196 of fundus images from 1263 AMD patients. The age of patients are from 30 to 84 years-old. We use 7156 images for training and 1040 images for testing. The details of our dataset are shown in Table 1. For training the main model, we use learning rate of 0.0002 and Adam optimizer with $β_{1} = 0.0$ and $β_{2} = 0.9$ . The model is trained with 200 epochs and a batch size of 32. The augmentation techniques are also applied (horizontal flip, shift, scale, and rotate). The Pix2pix model

Conclusions

In this study, we introduce a deep learning model for generating future fundus images of early AMD patients. Our GAN model with two discriminators shows that it can learn and produce the realistic fundus images. By exploiting drusen segmentation masks, we guide the model to focus on drusen changes which is the key factor for diagnosis AMD. The experimental results indicate that our method is able to catch the drusen changes over time.

Since this is the first study for generating future fundus

CRediT authorship contribution statement

Quang T.M. Pham: Conceptualization, Methodology, Software, Validation, Formal analysis, Writing – original draft, Writing – review & editing. Sangil Ahn: Methodology, Formal analysis, Writing – review & editing. Jitae Shin: Conceptualization, Formal analysis, Writing – review & editing, Supervision, Project administration, Funding acquisition. Su Jeong Song: Conceptualization, Data curation, Writing – review & editing.

Declaration of Competing Interest

The authors declare no conflict of interest.

Acknowledgement

This work was partly supported by the National Research Foundation of Korea(NRF) grant funded bythe Korea government(MSIT) (No. 2020R1F1A1065626) and was partly supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2021-2018-0-01798) supervised by the IITP (Institute for Information & communications Technology Promotion). It was also partly supported by the research fund from Biomedical Institute for Convergence

References (27)

L.S. Lim et al.
Age-related macular degeneration
The Lancet
(2012)
Y. Peng et al.
Deepseenet: a deep learning model for automated classification of patient-based age-related macular degeneration severity from color fundus photographs
Ophthalmology
(2019)
L.-C. Chen et al.
Encoder-decoder with atrous separable convolution for semantic image segmentation
H. Fu et al.
Joint optic disc and cup segmentation based on multi-label deep network and polar transformation
IEEE Trans Med Imaging
(2018)
H. Fu et al.
Disc-aware ensemble network for glaucoma screening from fundus image
IEEE Trans Med Imaging
(2018)
I.J. Goodfellow et al.
Generative adversarial nets
Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2
(2014)
K. He et al.
Deep residual learning for image recognition
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2016)
M. Heusel et al.
Gans trained by a two time-scale update rule converge to a local nash equilibrium
Proceedings of the 31st International Conference on Neural Information Processing Systems
(2017)
P. Isola et al.
Image-to-image translation with conditional adversarial networks
2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
(2017)
S.A. Kamran et al.
Fundus2angio: A conditional gan architecture for generating fluorescein angiography images from retinal fundus photography

S.A. Kamran et al.

Attention2angiogan: Synthesizing fluorescein angiography from retinal fundus images using generative adversarial networks

2020 25th International Conference on Pattern Recognition (ICPR)

(2021)

S.A. Kamran, K.F. Hossain, A. Tavakkoli, S.L. Zuckerbrod, S.A. Baker, Vtgan: Semi-supervised retinal image synthesis...

T. Karras, S. Laine, T. Aila, A Style-Based Generator Architecture for Generative Adversarial Networks (2018)....

Cited by (17)

A spatiotemporal convolution recurrent neural network for pixel-level peripapillary atrophy prediction using sequential fundus images
2024, Applied Soft Computing
The progression of peripapillary atrophy (PPA) is closely associated with the development of retinal diseases such as myopia and glaucoma. PPA prediction employing longitudinal images to obtain its progress trend can facilitate personalized treatment. Although existing studies have attempted to predict the persistence of PPA, such studies cannot provide quantitative measurement for personalized treatment. In this paper, we propose a spatiotemporal framework for pixel-level PPA prediction using sequential fundus images, including feature extractor, temporal memory, and spatiotemporal prediction modules. To take advantage of historical information, a temporal memory module is used, integrating current and prior features to build sequential data of features. To further enhance the prediction performance, the recurrent neural network states in a spatiotemporal prediction module transmit between different layers, enabling high-level states to guide the learning of low-level states. To handle missing data in clinical follow-up data, we use the predicted output of the spatiotemporal prediction module to substitute the missing data, and the scheduled-sampling strategy is employed in training. Extensive experiments conducted using a clinical dataset demonstrate that our proposed method achieves a satisfactory performance compared with the start-of-the-art models. The proposed approach can be applied using clinical data to obtain various quantitative indicators for personalized treatment and prevention of retinal disease.
Increasing-Margin Adversarial (IMA) training to improve adversarial robustness of neural networks
2023, Computer Methods and Programs in Biomedicine
Background and Objective: Deep neural networks (DNNs) are vulnerable to adversarial noises. Adversarial training is a general and effective strategy to improve DNN robustness (i.e., accuracy on noisy data) against adversarial noises. However, DNN models trained by the current existing adversarial training methods may have much lower standard accuracy (i.e., accuracy on clean data), compared to the same models trained by the standard method on clean data, and this phenomenon is known as the trade-off between accuracy and robustness and is commonly considered unavoidable. This issue prevents adversarial training from being used in many application domains, such as medical image analysis, as practitioners do not want to sacrifice standard accuracy too much in exchange for adversarial robustness. Our objective is to lift (i.e., alleviate or even avoid) this trade-off between standard accuracy and adversarial robustness for medical image classification and segmentation.
Methods: We propose a novel adversarial training method, named Increasing-Margin Adversarial (IMA) Training, which is supported by an equilibrium state analysis about the optimality of adversarial training samples. Our method aims to preserve accuracy while improving robustness by generating optimal adversarial training samples. We evaluate our method and the other eight representative methods on six publicly available image datasets corrupted by noises generated by AutoAttack and white-noise attack.
Results: Our method achieves the highest adversarial robustness for image classification and segmentation with the smallest reduction in accuracy on clean data. For one of the applications, our method improves both accuracy and robustness.
Conclusions: Our study has demonstrated that our method can lift the trade-off between standard accuracy and adversarial robustness for the image classification and segmentation applications. To our knowledge, it is the first work to show that the trade-off is avoidable for medical image segmentation.
Vision Transformers in medical computer vision—A contemplative retrospection
2023, Engineering Applications of Artificial Intelligence
Vision Transformers (ViTs), with the magnificent potential to unravel the information contained within images, have evolved as one of the most contemporary and dominant architectures that are being used in the field of computer vision. These are immensely utilized by plenty of researchers to perform new as well as former experiments. Here, in this article, we investigate the intersection of vision transformers and medical images. We proffered an overview of various ViT based frameworks that are being used by different researchers to decipher the obstacles in medical computer vision. We surveyed the applications of Vision Transformers in different areas of medical computer vision such as image-based disease classification, anatomical structure segmentation, registration, region-based lesion detection, captioning, report generation, and reconstruction using multiple medical imaging modalities that greatly assist in medical diagnosis and hence treatment process. Along with this, we also demystify several imaging modalities used in medical computer vision. Moreover, to get more insight and deeper understanding, the self-attention mechanism of transformers is also explained briefly. Conclusively, the ViT based solutions for each image analytics task are critically analyzed, open challenges are discussed and the pointers to possible solutions for future direction are deliberated. We hope this review article will open future research directions for medical computer vision researchers.
A fundus image enhancer based on illumination-guided attention and optic disc perception GAN
2023, Optik
High-quality fundus images are essential for ophthalmologists in clinical diagnosis of eye diseases. For reasons such as the acquisition process and the retinal disease, most fundus images may suffer poor illumination, blur, and low contrast. In order to obtain high-quality fundus images, a novel network model for fundus image enhancement is constructed in this paper. The model introduces a generated adversarial network (GAN) with illumination-guided attention and optic disc perception. A U-Net with illumination-guided attention is used as the generator, and a global discriminator and a local discriminator with optic disc segmentation are used in the learning process. The model can improve the exposure level of overexposed and underexposed fundus images and also stretch the contrast. Experiments are performed on multiple fundus image datasets taken by the handheld device and high-end device to verify the robustness of the proposed illumination-guided attention and optic disc perception GAN (IGAODPGAN). The retinal structure and pathological characteristics are highlighted significantly after enhancement. Our method is superior to other state-of-the-art algorithms in terms of both subjective and objective assessment, which can provide an efficient solution for fundus image enhancement, especially for the non-uniform illuminated fundus images.
Learn Single-horizon Disease Evolution for Predictive Generation of Post-therapeutic Neovascular Age-related Macular Degeneration
2023, Computer Methods and Programs in Biomedicine
Most of the existing disease prediction methods in the field of medical image processing fall into two classes, namely image-to-category predictions and image-to-parameter predictions.Few works have focused on image-to-image predictions. Different from multi-horizon predictions in other fields, ophthalmologists prefer to show more confidence in single-horizon predictions due to the low tolerance of predictive risk.
We propose a single-horizon disease evolution network (SHENet) to predictively generate post-therapeutic SD-OCT images by inputting pre-therapeutic SD-OCT images with neovascular age-related macular degeneration (nAMD). In SHENet, a feature encoder converts the input SD-OCT images to deep features, then a graph evolution module predicts the process of disease evolution in high-dimensional latent space and outputs the predicted deep features, and lastly, feature decoder recovers the predicted deep features to SD-OCT images. We further propose an evolution reinforcement module to ensure the effectiveness of disease evolution learning and obtain realistic SD-OCT images by adversarial training.
SHENet is validated on 383 SD-OCT cubes of 22 nAMD patients based on three well-designed schemes (P-0, P-1 and P-M) based on the quantitative and qualitative evaluations. Three metrics (PSNR, SSIM, 1-LPIPS) are used here for quantitative evaluations. Compared with other generative methods, the generative SD-OCT images of SHENet have the highest image quality (P-0: 23.659, P-1: 23.875, P-M: 24.198) by PSNR. Besides, SHENet achieves the best structure protection (P-0: 0.326, P-1: 0.337, P-M: 0.349) by SSIM and content prediction (P-0: 0.609, P-1: 0.626, P-M: 0.642) by 1-LPIPS. Qualitative evaluations also demonstrate that SHENet has a better visual effect than other methods.
SHENet can generate post-therapeutic SD-OCT images with both high prediction performance and good image quality, which has great potential to help ophthalmologists forecast the therapeutic effect of nAMD.
A GAN-based deep enhancer for quality enhancement of retinal images photographed by a handheld fundus camera
2022, Advances in Ophthalmology Practice and Research
Due to limited imaging conditions, the quality of fundus images is often unsatisfactory, especially for images photographed by handheld fundus cameras. Here, we have developed an automated method based on combining two mirror-symmetric generative adversarial networks (GANs) for image enhancement.
A total of 1047 retinal images were included. The raw images were enhanced by a GAN-based deep enhancer and another methods based on luminosity and contrast adjustment. All raw images and enhanced images were anonymously assessed and classified into 6 levels of quality classification by three experienced ophthalmologists. The quality classification and quality change of images were compared. In addition, image-detailed reading results for the number of dubiously pathological fundi were also compared.
After GAN enhancement, 42.9% of images increased their quality, 37.5% remained stable, and 19.6% decreased. After excluding the images at the highest level (level 0) before enhancement, a large number (75.6%) of images showed an increase in quality classification, and only a minority (9.3%) showed a decrease. The GAN-enhanced method was superior for quality improvement over a luminosity and contrast adjustment method (P＜0.001). In terms of image reading results, the consistency rate fluctuated from 86.6% to 95.6%, and for the specific disease subtypes, both discrepancy number and discrepancy rate were less than 15 and 15%, for two ophthalmologists.
Learning the style of high-quality retinal images based on the proposed deep enhancer may be an effective way to improve the quality of retinal images photographed by handheld fundus cameras.

View all citing articles on Scopus

View full text

Generating future fundus images for early age-related macular degeneration based on generative adversarial networks

Highlights

Abstract

Introduction

Section snippets

Related works

Image preprocessing

The experiment settings

Conclusions

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgement

The Lancet

Ophthalmology

Encoder-decoder with atrous separable convolution for semantic image segmentation

Joint optic disc and cup segmentation based on multi-label deep network and polar transformation

IEEE Trans Med Imaging

Disc-aware ensemble network for glaucoma screening from fundus image

IEEE Trans Med Imaging

Generative adversarial nets

Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2

Deep residual learning for image recognition

2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Gans trained by a two time-scale update rule converge to a local nash equilibrium

Proceedings of the 31st International Conference on Neural Information Processing Systems

Image-to-image translation with conditional adversarial networks

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

Fundus2angio: A conditional gan architecture for generating fluorescein angiography images from retinal fundus photography

Attention2angiogan: Synthesizing fluorescein angiography from retinal fundus images using generative adversarial networks

2020 25th International Conference on Pattern Recognition (ICPR)