Abstract
Fundus Fluorescein Angiography (FFA) is a critical tool for assessing retinal vascular dynamics and aiding in the diagnosis of eye diseases. However, its invasive nature and less accessibility compared to Color Fundus (CF) images pose significant challenges. Current CF to FFA translation methods are limited to static generation. In this work, we pioneer dynamic FFA video generation from static CF images. We introduce an autoregressive GAN for smooth, memory-saving frame-by-frame FFA synthesis. To enhance the focus on dynamic lesion changes in FFA regions, we design a knowledge mask based on clinical experience. Leveraging this mask, our approach integrates innovative knowledge mask-guided techniques, including knowledge-boosted attention, knowledge-aware discriminators, and mask-enhanced patchNCE loss, aimed at refining generation in critical areas and addressing the pixel misalignment challenge. Our method achieves the best FVD of 1503.21 and PSNR of 11.81 compared to other common video generation approaches. Human assessment by an ophthalmologist confirms its high generation quality. Notably, our knowledge mask surpasses supervised lesion segmentation masks, offering a promising non-invasive alternative to traditional FFA for research and clinical applications. The code is available at https://github.com/Michi-3000/Fundus2Video.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Chen, R., et al.: Translating color fundus photography to indocyanine green angiography using deep-learning for age-related macular degeneration screening. NPJ Digit. Med. 7(1), 34 (2024)
Chen, Y., et al.: Series-parallel generative adversarial network architecture for translating from fundus structure image to fluorescence angiography. Appl. Sci. 12(20), 10673 (2022)
Comin, C.H., Tsirukis, D.I., Sun, Y., Xu, X.: Quantification of retinal blood leakage in fundus fluorescein angiography in a retinal angiogenesis model. Sci. Rep. 11(1), 19903 (2021)
De Carlo, T.E., Romano, A., Waheed, N.K., Duker, J.S.: A review of optical coherence tomography angiography (octa). Int. J. Retina Vitreous 1, 1–15 (2015)
Dorjsembe, Z., Pao, H.K., Odonchimed, S., Xiao, F.: Conditional diffusion models for semantic 3D medical image synthesis. arXiv preprint arXiv:2305.18453 (2023)
Faust, O., Acharya, U.R., Ng, E.Y.K., Ng, K.H., Suri, J.S.: Algorithms for the automated detection of diabetic retinopathy using digital fundus images: a review. J. Med. Syst. 36, 145–157 (2012)
Freeman, W.R., Bartsch, D.U., Mueller, A.J., Banker, A.S., Weinreb, R.N.: Simultaneous indocyanine green and fluorescein angiography using a confocal scanning laser ophthalmoscope. Arch. Ophthalmol. 116(4), 455–463 (1998)
Huynh-Thu, Q., Ghanbari, M.: Scope of validity of PSNR in image/video quality assessment. Electron. Lett. 44(13), 800–801 (2008)
Iizuka, S., Simo-Serra, E., Ishikawa, H.: Globally and locally consistent image completion. ACM Trans. Graph. (ToG) 36(4), 1–14 (2017)
Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1125–1134 (2017)
Kamran, S.A., Hossain, K.F., Tavakkoli, A., Zuckerbrod, S.L.: Attention2Angiogan: synthesizing fluorescein angiography from retinal fundus images using generative adversarial networks. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9122–9129. IEEE (2021)
Kamran, S.A., Hossain, K.F., Tavakkoli, A., Zuckerbrod, S.L., Sanders, K.M., Baker, S.A.: RV-GAN: segmenting retinal vascular structure in fundus photographs using a novel multi-scale generative adversarial network. In: de Bruijne, M., et al. (eds.) MICCAI 2021, Part VIII. LNCS, vol. 12908, pp. 34–44. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_4
Kylstra, J.A., et al.: The importance of fluorescein angiography in planning laser treatment of diabetic macular edema. Ophthalmology 106(11), 2068–2073 (1999)
Li, F., Hu, Z., Chen, W., Kak, A.: Adaptive supervised PatchNCE loss for learning H &E-to-IHC stain translation with inconsistent groundtruth image pairs. In: Greenspan, H., et al. (eds.) MICCAI 2023. LNCS, vol. 14225, pp. 632–641. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43987-2_61
Pan, J., Wang, C., Jia, X., Shao, J., Sheng, L., Yan, J., Wang, X.: Video generation from single semantic label map. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3733–3742 (2019)
Park, K.B., Choi, S.H., Lee, J.Y.: M-GAN: retinal blood vessel segmentation by balancing losses through stacked deep fully convolutional networks. IEEE Access 8, 146308–146322 (2020)
Paszke, A., et al.: Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, vol. 32 (2019)
Ren, W., et al.: Consisti2V: enhancing visual consistency for image-to-video generation. arXiv preprint arXiv:2402.04324 (2024)
Shi, D., He, S., Yang, J., Zheng, Y., He, M.: One-shot retinal artery and vein segmentation via cross-modality pretraining. Ophthalmol. Sci. 4(2), 100363 (2024)
Shi, D., et al.: Translation of color fundus photography into fluorescein angiography using deep learning for enhanced diabetic retinopathy screening. Ophthalmol. Sci. 3(4), 100401 (2023)
Sinthanayothin, C., et al.: Automated detection of diabetic retinopathy on digital fundus images. Diabet. Med. 19(2), 105–112 (2002)
Song, F., Zhang, W., Zheng, Y., Shi, D., He, M.: A deep learning model for generating fundus autofluorescence images from color fundus photography. Adv. Ophthalmol. Pract. Res. 3(4), 192–198 (2023)
Tavakkoli, A., Kamran, S.A., Hossain, K.F., Zuckerbrod, S.L.: A novel deep learning conditional generative adversarial network for producing angiography images from retinal fundus photographs. Sci. Rep. 10(1), 1–15 (2020)
Unterthiner, T., van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., Gelly, S.: FVD: a new metric for video generation (2019)
Wang, T.C., Liu, M.Y., Zhu, J.Y., Tao, A., Kautz, J., Catanzaro, B.: High-resolution image synthesis and semantic manipulation with conditional GANs. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8798–8807 (2018)
Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13(4), 600–612 (2004)
Yannuzzi, L.A., et al.: Ophthalmic fundus imaging: today and beyond. Am. J. Ophthalmol. 137(3), 511–524 (2004)
Yannuzzi, L.A., et al.: Fluorescein angiography complication survey. Ophthalmology 93(5), 611–617 (1986)
Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)
Acknowledgments
The study was supported by the Global STEM Professorship Scheme (P0046113) and the Start-up Fund for RAPs under the Strategic Hiring Scheme (P0048623) from HKSAR. The sponsors or funding organizations had no role in the design or conduct of this research.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
A patent has been filed for this innovation (CN 202410360491.4).
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, W. et al. (2024). Fundus2Video: Cross-Modal Angiography Video Generation from Static Fundus Photography with Clinical Knowledge Guidance. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15001. Springer, Cham. https://doi.org/10.1007/978-3-031-72378-0_64
Download citation
DOI: https://doi.org/10.1007/978-3-031-72378-0_64
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72377-3
Online ISBN: 978-3-031-72378-0
eBook Packages: Computer ScienceComputer Science (R0)
