Bridge-Net: Context-involved U-net with patch-based loss weight mapping for retinal blood vessel segmentation
Introduction
Retinal diseases such as glaucoma, age-related macular degeneration, and diabetic retinopathy are one of the most important leading reasons that cause irreversible blindness (Tham et al., 2014). As a fundus structure that can be observed through non-invasive techniques, the morphology of retinal blood vessels such as length and width can play an important role in the diagnosis of ophthalmic diseases (Tomlins & Shah, 2008). Therefore, the analysis of retinal blood vessels is of great importance in the detection and diagnosis of retinal diseases (Fan et al., 2019, Hu et al., 2018).
In the whole process of analysis, retinal blood vessel segmentation is the first and necessary step (Srinidhi, Aparna, & Rajan, 2017). Ophthalmologists often solve this problem manually. However, manual retinal blood vessel segmentation is tedious and time-consuming (Fraz et al., 2012a), which encourages researchers to develop automatic segmentation methods to help computer-aided diagnostic (CAD) systems to perform automated diagnosis. Due to the existence of thin vessels, lesions, complicated blood vessel structure, and low contrast in fundus images, the segmentation remains a challenging task (Wang et al., 2015).
Deep learning, which can automatically learn discriminative features, has been widely applied in many computer vision problems including retinal blood vessel segmentation in recent years (LeCun, Bengio, & Hinton, 2015). In general, retinal vessel segmentation methods based on deep learning can be divided into two categories, i.e., end-to-end methods and patch-based methods (Arcadu et al., 2019, Dai et al., 2021, Moccia et al., 2018). Since fundus images usually have high resolution, the end-to-end methods often directly process the entire image by reducing the spatial resolution due to limited computing resources, which undoubtedly loses a lot of spatial information. The patch-based methods can expand data without a series of data augmentations that are applied in end-to-end methods, and alleviate the over-fitting phenomenon that tends to occur due to the small amount of data. However, the existing patch-based methods only perform follow-up processing on the extracted patch but ignore the surrounding spatial context information. Besides, the extreme imbalance between blood vessel pixels and background pixels in fundus images is not well considered, which makes it difficult to segment vessels, especially thin vessels, from the background.
To address the above issues, in this paper we propose a context-involved U-net with patch-based loss weight mapping for retinal blood vessel segmentation. Our motivations are two-fold.
(1) Existing patch-based methods do not consider the spatial context information around the extracted patch, which often plays an important role in determining the shape and location of blood vessels. In this paper, we propose a novel deep network architecture named Bridge-net to make use of the context of the retinal blood vessels efficiently. Our Bridge-net incorporates a recurrent neural network (RNN) into a convolutional neural network (CNN) to deliver the context and then to produce the probability map of the retinal blood vessels. Specifically, the context information around each patch is integrated into the feature extraction stage through the RNN to help determine the shape and location of the blood vessel, including the thickness and relative location of the blood vessel, to supplement the missing details. The experimental results show that the context information can be effectively used by Bridge-net to achieve accurate segmentation of blood vessels.
(2) The above two types of deep learning methods cannot well consider the imbalance between blood vessel pixels and background pixels, resulting in the inability to accurately identify thin blood vessels. In this paper, we propose a patch-based loss weight mapping by considering the distributions of different types of blood vessels to correct the imbalance, since there are large morphological differences between thick and thin blood vessels. Specifically, the patch classification algorithm is firstly proposed to define a patch as the thick vessel majority patch (TKp) or the thin vessel majority patch (TNp), and then different loss weights are assigned to the blood vessel pixels and background pixels in these two patches during training.
The contributions of this paper can be summarized as follows.
- •
We propose a novel network architecture, Bridge-net, for segmenting retinal blood vessels in fundus images by learning context-involved and non-context features simultaneously. Furthermore, a recurrent neural network is employed to deliver the context information of the target region.
- •
We present a patch classification algorithm to classify the thick and thin vessel patches and then propose a patch-based loss weight mapping based on the patch classification to correct the imbalance between blood vessels and background.
- •
We evaluate the proposed method on four publicly available datasets STARE, DRIVE, CHASEDB1, and HRF by comparing the results with those of several state-of-the-art methods. The results have shown the effectiveness and stability of our proposed method in retinal vessel segmentation.
The remainder of the paper is organized as follows. In Section 2, we review the retinal blood vessel segmentation methods over the past decades. Section 3 details the proposed method including Bridge-net architecture and the patch-based loss weight mapping. In Section 4, we introduce the experimental protocol and parameter settings. In Section 5, we evaluate the effectiveness of the proposed method through extensive experiments. Section 6 discusses the results and Section 7 concludes our work.
Section snippets
Related work
In the past decades, many works for retinal blood vessel segmentation have been developed. These approaches can be broadly divided into two categories: supervised methods and unsupervised methods (Fraz et al., 2012a).
Methodology
The flowchart of the proposed method is shown in Fig. 1. We propose a novel deep learning architecture to generate the probability map of the target region, and then train the architecture with patches randomly extracted from the preprocessed images. Meanwhile, a patch-based loss weight mapping is proposed to correct the imbalance between blood vessels and background according to the distributions of different types (thick and thin) of blood vessels.
Datasets
Four public datasets: STARE (Hoover, Kouznetsova, & Goldbaum, 1998), DRIVE (Staal et al., 2004), CHASEDB1 (Fraz et al., 2012b), and HRF (Odstrcilik et al., 2013) are used to evaluate the proposed Bridge-net architecture and the patch-based loss weight mapping.
The STARE1 dataset includes 20 images in which half of them are pathological while the others are normal. These images are taken by a TopCon camera at field of view (FOV) with a size of
Experiment for the bridge-net architecture
RU-net contains an RNN structure, but without the participation of the context of the target region, that is, and coincide with the target region , i.e., . Correspondingly, for and , they are the output of the same input through a network with same parameters, so . Therefore, the final reason for the difference between U-net and RU-net results lies in the fusion network . From (2) and Fig. 4c, we can find that is the feature map generated directly by ,
Discussion
In general, the end-to-end deep learning methods contain complete image information, which can assist in the segmentation of retinal blood vessels, and its calculation speed is fast. However, the whole image often contains a large number of vessel modes (Xia, Jiang, Deng, Xin, & Doss, 2018), and the data and labels of medical images are extremely difficult to obtain (Wu et al., 2020), which limits the accurate segmentation of deep learning methods, especially for thin blood vessels. In
Conclusion
In this paper, we analyze the limitations of the patch-based deep learning segmentation methods for retinal blood vessels and propose an efficient and automatic segmentation method. To effectively utilize the context information of the target region, a novel network architecture, Bridge-net, is developed. Firstly, we employ two descriptions on two patches respectively regarding the target region with and without context information to generate discriminative features through U-net. Then, we use
CRediT authorship contribution statement
Yuan Zhang: Conceptualization, Methodology, Visualization, Formal analysis, Validation, Software, Writing – review & editing. Miao He: Methodology, Visualization, Formal analysis, Validation, Software, Writing – review & editing. Zhineng Chen: Methodology, Visualization, Validation, Writing – review & editing. Kai Hu: Conceptualization, Supervision, Methodology, Visualization, Formal analysis, Validation, Writing – review & editing. Xuanya Li: Methodology, Visualization, Validation, Writing –
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
The authors would like to thank Dr. Huazhu Fu from the Inception Institute of Artificial Intelligence of Abu Dhabi in UAE for kindly providing the link to the related datasets and some helpful discussions about the experiment. This work was supported by the National Natural Science Foundation of China under Grants 61802328, 61972333, and 61771415, the Research Foundation of Education Department of Hunan Province of China under Grants 21B0172 and 19B561, the Natural Science Foundation of Hunan
References (59)
- et al.
Trainable cosfire filters for vessel delineation with application to retinal images
Medical Image Analysis
(2015) - et al.
Blood vessel segmentation methodologies in retinal images a survey
Computer Methods and Programs in Biomedicine
(2012) - et al.
Retinal vessel segmentation of color fundus images using multiscale convolutional neural network with an improved cross-entropy loss function
Neurocomputing
(2018) - et al.
Blood vessel segmentation algorithms—review of methods, datasets and evaluation metrics
Computer Methods and Programs in Biomedicine
(2018) - et al.
Adaptive histogram equalization and its variations
Computer Vision, Graphics, and Image Processing
(1987) - et al.
Strided fully convolutional neural network for boosting the sensitivity of retinal blood vessels segmentation
Expert Systems with Applications
(2019) - et al.
Global prevalence of glaucoma and projections of glaucoma burden through 2040: a systematic review and meta-analysis
Ophthalmology
(2014) - et al.
Retinal vessel segmentation using multiwavelet kernels and multiscale hierarchical decomposition
Pattern Recognition
(2013) - et al.
Blood vessel segmentation from fundus image by a cascade classification framework
Pattern Recognition
(2019) - et al.
Hierarchical retinal blood vessel segmentation based on feature and ensemble learning
Neurocomputing
(2015)
Scs-net: A scale and context sensitive network for retinal vessel segmentation
Medical Image Analysis
A hybrid deep segmentation network for fundus vessels via deep-learning framework
Neurocomputing
Segmentation of retinal blood vessels using the radial projection and semi-supervised approach
Pattern Recognition
Improving retinal vessel segmentation with joint local loss by matting
Pattern Recognition
A refined equilibrium generative adversarial network for retinal vessel segmentation
Neurocomputing
Retinal vessel segmentation in colour fundus images using extreme learning machine
Computerized Medical Imaging and Graphics
An active contour model for segmenting and measuring retinal vessels
IEEE Transactions on Medical Imaging
Deep learning algorithm predicts diabetic retinopathy progression in individual patients
NPJ Digital Medicine
Deep retinal image segmentation with regularization under geometric priors
IEEE Transactions on Image Processing
A deep learning system for detecting diabetic retinopathy across the disease spectrum
Nature Communications
A fully convolutional neural network based structured prediction approach towards the retinal vessel segmentation
A hierarchical image matting model for blood vessel segmentation in fundus images
IEEE Transactions on Image Processing
Patch-based fully convolutional neural network with skip connections for retinal blood vessel segmentation
An ensemble classification based approach applied to retinal blood vessel segmentation
IEEE Transactions on Biomedical Engineering
Deepvessel: Retinal vessel segmentation via deep learning and conditional random field
Retinal vessel segmentation via deep learning network and fully connected conditional random fields
Supervised sequence labelling with recurrent neural networks
Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response
Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response
IEEE Transactions on Medical Imaging
Cited by (61)
A retinal vessel segmentation network with multiple-dimension attention and adaptive feature fusion
2024, Computers in Biology and MedicineResMU-Net: Residual Multi-kernel U-Net for blood vessel segmentation in retinal fundus images
2024, Biomedical Signal Processing and ControlCoVi-Net: A hybrid convolutional and vision transformer neural network for retinal vessel segmentation
2024, Computers in Biology and MedicineCross-level collaborative context-aware framework for medical image segmentation
2024, Expert Systems with ApplicationsAn improved supervised and attention mechanism-based U-Net algorithm for retinal vessel segmentation
2024, Computers in Biology and Medicine