Fine-Grained Multi-modal Fundus Image Generation Based on Diffusion Models for Glaucoma Classification

Abstract

With the emergence of Foundation Model, the generation quality and generalisation ability of image generation method have been further improved. However, medical image generation is still a challenging and promising task. Recently, diffusion-based models are more prominent in multi-modal image generation for its flexibility. Therefore, in order to solve the problem of lack of high-quality medical images and high annotation costs, we propose a fine-grained multi-modal fundus image generation method based on foundation models to research an efficient way of data augmentation. First, we adopt optic fundus images, fundus vessel images and class textual information to form a weakly supervised fine-tuning dataset. Then, based on the Stable-Diffusion and Control-Net model, we fine-tune our method by LoRA model to generate high-resolution fundus images of special diseases in a targeted manner. Furthermore, we use these synthetic fundus images in conjunction with existing datasets for data augmentation or model fine-tuning to improve performance in the glaucoma classification task. Extensive experiments have shown that our method produces high quality medical fundus images and can be well applied to real-world medical imaging tasks. Moreover, experimental results show that we are able to generate fundus images that act as an augmentation, meaning that the generation of foundation models is effective in certain domains.