Bridge-Net: Context-involved U-net with patch-based loss weight mapping for retinal blood vessel segmentation

https://doi.org/10.1016/j.eswa.2022.116526Get rights and content

Highlights

  • We present a novel automatic method for the segmentation of retinal blood vessels.

  • We propose Bridge-net by joint learning context-involved and non-context features.

  • We develop a patch-based loss weight mapping to correct the imbalance of the image.

  • We evaluate the effectiveness of the proposed method on four public datasets.

  • The results have verified the effectiveness and stability of the proposed method.

Abstract

Retinal blood vessel segmentation in fundus images plays an important role in the early diagnosis and treatment of retinal diseases. In recent years, the segmentation methods based on deep neural networks have attracted the attention of experts and scholars. However, due to the complexity of the distribution of blood vessels in fundus images and the imbalance between blood vessels and background, retinal blood vessel segmentation remains challenging. In this paper, we present a retinal blood vessel segmentation method using deep neural networks. Firstly, we propose a novel deep network architecture named Bridge-net to make use of the context of the retinal blood vessels efficiently. Specifically, the architecture incorporates a recurrent neural network (RNN) into a convolutional neural network (CNN) to deliver the context and then to produce the probability map of the retinal blood vessels. Secondly, we propose a patch-based loss weight mapping by considering the distributions of different types of blood vessels to correct the imbalance, since there are large morphological differences between thick and thin blood vessels. Finally, we evaluate our method on three publicly datasets STARE, DRIVE, and CHASE_DB1, and compare the results to eighteen state-of-the-art approaches. We also compare our method with some existing approaches on a high-resolution dataset, i.e., HRF. The results show that our method achieves better/comparable performances when compared to the existing approaches. The results on various datasets also verify the effectiveness and stability of the proposed method.

Introduction

Retinal diseases such as glaucoma, age-related macular degeneration, and diabetic retinopathy are one of the most important leading reasons that cause irreversible blindness (Tham et al., 2014). As a fundus structure that can be observed through non-invasive techniques, the morphology of retinal blood vessels such as length and width can play an important role in the diagnosis of ophthalmic diseases (Tomlins & Shah, 2008). Therefore, the analysis of retinal blood vessels is of great importance in the detection and diagnosis of retinal diseases (Fan et al., 2019, Hu et al., 2018).

In the whole process of analysis, retinal blood vessel segmentation is the first and necessary step (Srinidhi, Aparna, & Rajan, 2017). Ophthalmologists often solve this problem manually. However, manual retinal blood vessel segmentation is tedious and time-consuming (Fraz et al., 2012a), which encourages researchers to develop automatic segmentation methods to help computer-aided diagnostic (CAD) systems to perform automated diagnosis. Due to the existence of thin vessels, lesions, complicated blood vessel structure, and low contrast in fundus images, the segmentation remains a challenging task (Wang et al., 2015).

Deep learning, which can automatically learn discriminative features, has been widely applied in many computer vision problems including retinal blood vessel segmentation in recent years (LeCun, Bengio, & Hinton, 2015). In general, retinal vessel segmentation methods based on deep learning can be divided into two categories, i.e., end-to-end methods and patch-based methods (Arcadu et al., 2019, Dai et al., 2021, Moccia et al., 2018). Since fundus images usually have high resolution, the end-to-end methods often directly process the entire image by reducing the spatial resolution due to limited computing resources, which undoubtedly loses a lot of spatial information. The patch-based methods can expand data without a series of data augmentations that are applied in end-to-end methods, and alleviate the over-fitting phenomenon that tends to occur due to the small amount of data. However, the existing patch-based methods only perform follow-up processing on the extracted patch but ignore the surrounding spatial context information. Besides, the extreme imbalance between blood vessel pixels and background pixels in fundus images is not well considered, which makes it difficult to segment vessels, especially thin vessels, from the background.

To address the above issues, in this paper we propose a context-involved U-net with patch-based loss weight mapping for retinal blood vessel segmentation. Our motivations are two-fold.

(1) Existing patch-based methods do not consider the spatial context information around the extracted patch, which often plays an important role in determining the shape and location of blood vessels. In this paper, we propose a novel deep network architecture named Bridge-net to make use of the context of the retinal blood vessels efficiently. Our Bridge-net incorporates a recurrent neural network (RNN) into a convolutional neural network (CNN) to deliver the context and then to produce the probability map of the retinal blood vessels. Specifically, the context information around each patch is integrated into the feature extraction stage through the RNN to help determine the shape and location of the blood vessel, including the thickness and relative location of the blood vessel, to supplement the missing details. The experimental results show that the context information can be effectively used by Bridge-net to achieve accurate segmentation of blood vessels.

(2) The above two types of deep learning methods cannot well consider the imbalance between blood vessel pixels and background pixels, resulting in the inability to accurately identify thin blood vessels. In this paper, we propose a patch-based loss weight mapping by considering the distributions of different types of blood vessels to correct the imbalance, since there are large morphological differences between thick and thin blood vessels. Specifically, the patch classification algorithm is firstly proposed to define a patch as the thick vessel majority patch (TKp) or the thin vessel majority patch (TNp), and then different loss weights are assigned to the blood vessel pixels and background pixels in these two patches during training.

The contributions of this paper can be summarized as follows.

  • We propose a novel network architecture, Bridge-net, for segmenting retinal blood vessels in fundus images by learning context-involved and non-context features simultaneously. Furthermore, a recurrent neural network is employed to deliver the context information of the target region.

  • We present a patch classification algorithm to classify the thick and thin vessel patches and then propose a patch-based loss weight mapping based on the patch classification to correct the imbalance between blood vessels and background.

  • We evaluate the proposed method on four publicly available datasets STARE, DRIVE, CHASE_DB1, and HRF by comparing the results with those of several state-of-the-art methods. The results have shown the effectiveness and stability of our proposed method in retinal vessel segmentation.

The remainder of the paper is organized as follows. In Section 2, we review the retinal blood vessel segmentation methods over the past decades. Section 3 details the proposed method including Bridge-net architecture and the patch-based loss weight mapping. In Section 4, we introduce the experimental protocol and parameter settings. In Section 5, we evaluate the effectiveness of the proposed method through extensive experiments. Section 6 discusses the results and Section 7 concludes our work.

Section snippets

Related work

In the past decades, many works for retinal blood vessel segmentation have been developed. These approaches can be broadly divided into two categories: supervised methods and unsupervised methods (Fraz et al., 2012a).

Methodology

The flowchart of the proposed method is shown in Fig. 1. We propose a novel deep learning architecture to generate the probability map of the target region, and then train the architecture with patches randomly extracted from the preprocessed images. Meanwhile, a patch-based loss weight mapping is proposed to correct the imbalance between blood vessels and background according to the distributions of different types (thick and thin) of blood vessels.

Datasets

Four public datasets: STARE (Hoover, Kouznetsova, & Goldbaum, 1998), DRIVE (Staal et al., 2004), CHASE_DB1 (Fraz et al., 2012b), and HRF (Odstrcilik et al., 2013) are used to evaluate the proposed Bridge-net architecture and the patch-based loss weight mapping.

The STARE1 dataset includes 20 images in which half of them are pathological while the others are normal. These images are taken by a TopCon camera at 35° field of view (FOV) with a size of

Experiment for the bridge-net architecture

RU-net contains an RNN structure, but without the participation of the context of the target region, that is, L and S coincide with the target region T, i.e., L=S=T. Correspondingly, for f(l) and f(t), they are the output of the same input through a network with same parameters, so f(l)=f(l). Therefore, the final reason for the difference between U-net and RU-net results lies in the fusion network Fu. From (2) and Fig. 4c, we can find that f(l) is the feature map generated directly by f(l),

Discussion

In general, the end-to-end deep learning methods contain complete image information, which can assist in the segmentation of retinal blood vessels, and its calculation speed is fast. However, the whole image often contains a large number of vessel modes (Xia, Jiang, Deng, Xin, & Doss, 2018), and the data and labels of medical images are extremely difficult to obtain (Wu et al., 2020), which limits the accurate segmentation of deep learning methods, especially for thin blood vessels. In

Conclusion

In this paper, we analyze the limitations of the patch-based deep learning segmentation methods for retinal blood vessels and propose an efficient and automatic segmentation method. To effectively utilize the context information of the target region, a novel network architecture, Bridge-net, is developed. Firstly, we employ two descriptions on two patches respectively regarding the target region with and without context information to generate discriminative features through U-net. Then, we use

CRediT authorship contribution statement

Yuan Zhang: Conceptualization, Methodology, Visualization, Formal analysis, Validation, Software, Writing – review & editing. Miao He: Methodology, Visualization, Formal analysis, Validation, Software, Writing – review & editing. Zhineng Chen: Methodology, Visualization, Validation, Writing – review & editing. Kai Hu: Conceptualization, Supervision, Methodology, Visualization, Formal analysis, Validation, Writing – review & editing. Xuanya Li: Methodology, Visualization, Validation, Writing –

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

The authors would like to thank Dr. Huazhu Fu from the Inception Institute of Artificial Intelligence of Abu Dhabi in UAE for kindly providing the link to the related datasets and some helpful discussions about the experiment. This work was supported by the National Natural Science Foundation of China under Grants 61802328, 61972333, and 61771415, the Research Foundation of Education Department of Hunan Province of China under Grants 21B0172 and 19B561, the Natural Science Foundation of Hunan

References (59)

  • WuH. et al.

    Scs-net: A scale and context sensitive network for retinal vessel segmentation

    Medical Image Analysis

    (2021)
  • YangL. et al.

    A hybrid deep segmentation network for fundus vessels via deep-learning framework

    Neurocomputing

    (2021)
  • YouX. et al.

    Segmentation of retinal blood vessels using the radial projection and semi-supervised approach

    Pattern Recognition

    (2011)
  • ZhaoH. et al.

    Improving retinal vessel segmentation with joint local loss by matting

    Pattern Recognition

    (2020)
  • ZhouY. et al.

    A refined equilibrium generative adversarial network for retinal vessel segmentation

    Neurocomputing

    (2021)
  • ZhuC. et al.

    Retinal vessel segmentation in colour fundus images using extreme learning machine

    Computerized Medical Imaging and Graphics

    (2017)
  • Al-DiriB. et al.

    An active contour model for segmenting and measuring retinal vessels

    IEEE Transactions on Medical Imaging

    (2009)
  • ArcaduF. et al.

    Deep learning algorithm predicts diabetic retinopathy progression in individual patients

    NPJ Digital Medicine

    (2019)
  • CherukuriV. et al.

    Deep retinal image segmentation with regularization under geometric priors

    IEEE Transactions on Image Processing

    (2020)
  • DaiL. et al.

    A deep learning system for detecting diabetic retinopathy across the disease spectrum

    Nature Communications

    (2021)
  • DasguptaA. et al.

    A fully convolutional neural network based structured prediction approach towards the retinal vessel segmentation

  • FanZ. et al.

    A hierarchical image matting model for blood vessel segmentation in fundus images

    IEEE Transactions on Image Processing

    (2019)
  • FengZ. et al.

    Patch-based fully convolutional neural network with skip connections for retinal blood vessel segmentation

  • FrazM.M. et al.

    An ensemble classification based approach applied to retinal blood vessel segmentation

    IEEE Transactions on Biomedical Engineering

    (2012)
  • FuH. et al.

    Deepvessel: Retinal vessel segmentation via deep learning and conditional random field

  • FuH. et al.

    Retinal vessel segmentation via deep learning network and fully connected conditional random fields

  • GravesA.

    Supervised sequence labelling with recurrent neural networks

    (2012)
  • HooverA. et al.

    Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response

  • HooverA. et al.

    Locating blood vessels in retinal images by piecewise threshold probing of a matched filter response

    IEEE Transactions on Medical Imaging

    (2000)
  • Cited by (61)

    View all citing articles on Scopus
    View full text