Elsevier

Neurocomputing

Volume 309, 2 October 2018, Pages 179-191
Neurocomputing

Retinal vessel segmentation of color fundus images using multiscale convolutional neural network with an improved cross-entropy loss function

https://doi.org/10.1016/j.neucom.2018.05.011Get rights and content

Highlights

  • We propose a novel retinal vessel segmentation method of fundus images.

  • The method is combined with CNN and fully CRFs.

  • We develop a multiscale CNN with an improved cross-entropy loss function.

  • We evaluate the effectiveness of the proposed method on DRIVE and STARE datasets.

  • The results of our method are competitive in the sensitivity while ensuring accuracy.

Abstract

Retinal vessel analysis of fundus images is an indispensable method for the screening and diagnosis of related diseases. In this paper, we propose a novel retinal vessel segmentation method of the fundus images based on convolutional neural network (CNN) and fully connected conditional random fields (CRFs). The segmentation process is mainly divided into two steps. Firstly, a multiscale CNN architecture with an improved cross-entropy loss function is proposed to produce the probability map from image to image. We construct the multiscale network by combining the feature map of each middle layer to learn more detail information of the retinal vessels. Meanwhile, our proposed cross-entropy loss function ignores the slightest loss of relatively easy samples in order to take more attention to learn the hard examples. Secondly, CRFs is applied to get the final binary segmentation result which makes use of more spatial context information by taking into account the interactions among all of the pixels in the fundus images. The effectiveness of the proposed method has been evaluated on two public datasets, i.g., DRIVE and STARE with comparisons against eleven state-of-the-art approaches including five deep learning based methods. Results show that our method allows for detection of more tiny blood vessels and more precise locating of the edges.

Introduction

There are a lot of people in the world who are suffering from blindness. Some ophthalmologic diseases such as diabetes, hypertension, and arteriosclerosis [1] are main causes of blindness that cannot be ignored. These diseases are related to morphological changes of the vessels in diameter, tortuosity, branching pattern or angles [2]. Experts can analyze changes in vascular morphology by segmenting the retinal vessels. Therefore, the segmentation task of the retinal vessels is of great importance for the diagnosis of related diseases. The medical images most commonly used for the ophthalmic diseases are two-dimensional (2-D) color fundus images and three-dimensional (3-D) optic coherence tomography (OCT) images [3]. As the color fundus image can be obtained more conveniently just through a fundus camera by contrast to OCT image, it has been widely used in the analysis of some ophthalmologic diseases.

Usually, specialists segment the retinal vessels of fundus images manually. However, manual segmentation by experts requires expertise and is time consuming [4]. It is not applicable for large-scale screening or diagnosis work regardless of time or cost. Therefore, an effective and efficient automatic vessel segmentation manner is particularly meaningful. With the development of computer-aided diagnosis (CAD) system, various vessel segmentation methods have been proposed. These methods can segment automatically without depending on the experts and provide reliable and robust segmentation. However, affected by many factors such as the lesion area, the complicated vessel structure and the low contrast between the target and the background [1], the segmentation task remains challenging.

The existing segmentation methods can be divided into supervised or unsupervised categories [4] according to whether the manual labeled ground truths are used or not. Obviously, the unsupervised approaches are designed according to the inherent characteristics of the blood vessels without relying on the manual labeled map. Salem et al. [5] proposed an unsupervised approach by using RAdius based Clustering ALgorithm (RACAL) which mapped the distribution of image pixels through the distance based principle. Ng et al. [6] developed a maximum likelihood inversion model for the retinal vessel segmentation task. Neto et al. [7] proposed an unsupervised coarse-to-fine algorithm. In this approach, they segmented vascular structure relied on mathematical morphology, spatial dependency, and curvature. Khan et al. [8] proposed a contrast-sensitive segmentation method using background normalization, second-order Gaussian filter and region growing. They pay attention to the low-contrast region in a contrast-sensitive manner. Zhao et al. [9] proposed an infinite perimeter active contour model which employed Lebesgue measurement to detect small vessels. Their method combined different region information to guarantee the detection of vessels and vessels edges. Salazar-Gonzalez et al. [10] first carried out a preprocessing for the image by adaptive histogram equalization and robust distance transform, and then segmented the vessels with graph cut. Yin et al. [11] proposed a segmentation method using hessian matrix and thresholded entropy. They used post-processing to eliminate noise and the central light reflex. These unsupervised methods performed well when detecting retinal vessel according to the vascular structure and some prior knowledge without using ground truths.

Meanwhile, the supervised approaches extract retinal blood vessels by learning rules from the annotation results. Marin et al. [12] treated the segmentation task as a pixel classification problem. They first extracted gray-level features and moment invariant features for every pixel and then put these features into neural networks to classify the pixel. Zhu et al. [2] constructed a 39-D discriminative feature vector such as local features, Hessian, divergence of vector field and so on for every pixel based on prior knowledge, and then classify them into vessel or not using the extreme learning machine (ELM) classifier. Generally, most of these methods depend on prior knowledge or a series of complex processing to get discriminative features.

Convolutional neural network (CNN) can automatically learn hierarchical features by some convolution and pooling operations without prior knowledge and extra preprocess. In recent years, due to the efficiency of CNN, it has been widely used in computer vision, medical image processing, and pattern recognition [13], [14].

Wang et al. [1] segmented the retinal vessels based on a pixel classification mode. They proposed a hierarchical segmentation method which used CNN as a trainable feature extractor and employed the ensemble random forests (RFs) as a trainable classifier. In order to determine what each pixel belongs to, they extracted features of a square subwindow centered on the pixel that need to be classified with CNN and then predicted with ensemble RFs. This patch-based method is time-consuming since it has a large number of repeatable calculations. Furthermore, most of existing image-to-image CNN methods performed better when detecting large objects. However, retinal vessel structures are thin elongated [10] and therefore it is difficult to segment the vessels accurately. In addition, the imbalance problem further increases the difficulty of the vessel segmentation, since the vessel pixels account for only about 10% of the pixels of the entire image. Inspired by the holistically-nested edge detection (HED) [15] system, Fu et al. [16] formulated the retinal vessel segmentation task into a boundary detection task. They first applied HED to get a probability map and then utilized fully connected conditional random fields (CRFs) to refine the segmentation. This image-to-image method was more efficient and the segmentation accuracy achieved at the state-of-the-art level. With the increase of HED network layers, the receptive field was gradually increased to learn more global discriminative features to distinguish vessels and non-vessels.

However, the top layers contain more global information while the bottom layers contribute more details. Fu et al.’s [16] method lost some vascular edges and thin vessels inevitably by only adopting part of the top layers to ensure the accuracy of the overall structure and the majority of the vascular. Therefore, although the existing CNN methods usually learn the discriminative information by increasing the depth of the network, the details contained in the shallow layers cannot be ignored for detecting thin elongated vascular structure. More specifically, the image pixels can be divided into two categories: easy to detect and hard to detect. As the easy pixels occupy the majority in the fundus images, they often dominated the direction of parameter update in the learning process of CNN. There are still some hard examples that have not been detected accurately when the loss stabilizes and the CNN model stops to update.

In this paper, in order to segment the retinal vessels more efficiently and effectively, we propose a supervised image-to-image retinal vessel segmentation system by combining multiscale CNN and CRFs. Firstly, we used a multiscale CNN to get a probability map. The proposed CNN model uses a multiscale architecture to fuse richer multiscale information for comprehensive vascular feature description. Meanwhile, we develop an improved loss function for emphasizing hard examples to effectively detect small objects and overcome the imbalance problem. Later, to smooth the coarse probability map, we define the CNN output probability map as unary potentials of the CRFs model to obtain the final segmentation map. Finally, we carry out experiments on two public datasets and conduct performance evaluations to verify the segmentation effect. Results on both datasets show that our method outperformed most of the recent approaches in terms of sensitivity and accuracy. The contributions of this work are mainly reflected in two aspects of the CNN construction.

  • 1)

    We put all of the middle convolution features into the final CNN classifier to learn a richer multiscale information [17] so that we can locate vessel edges precisely and detect more tiny vessels. Especially, by combining the features in each layer, we take full advantage of the information of every scale to improve the detection accuracy of the vessels edges and the tiny thin vessels.

  • 2)

    We propose an improved class-balanced cross-entropy loss function that further improves the segmentation performance by addressing the segmentation of hard samples such as the vascular edge and the tiny thin vessels.

The remainder of this paper is organized as follows. Section 2 provides a detailed description of the proposed method including multiscale CNN architecture, the improved loss function, and the CRFs. Section 3 introduces the image datasets, the experimental settings and the evaluation metrics. In Section 4, we discuss and compare our experimental results from many aspects. Finally, a conclusion with future work is drawn in Section 5.

Section snippets

Method

The flowchart of the proposed method is shown in Fig. 1. We first propose a multiscale CNN to learn probability map. The multiscale CNN architecture is used to learn discriminatory features from the color fundus images. Meanwhile, an improved cross-entropy loss function is proposed to emphasize the hard examples. Furthermore, we use CRF to mitigate noise and edge blurring, thus refining the probability map to get the final segmentation result. The details of the above steps are presented as

Datasets

The evaluation is conducted on two public datasets for retinal vessel segmentation of fundus images: DRIVE [23] and STARE [24].

DRIVE contains 40 images obtained at 45° field of view (FOV) with a resolution of 565 × 584 pixels. 20 images of them are selected as the training set and another 20 images are used as the testing set. Each training image has one ground truth segmented by a specialist. Each testing image has two manual segmentation results segmented by two specialists with the first one

Results and discussion

We conduct a series of experiments on two datasets to confirm the validity of our method with a variety of indicators and illustrations. We also compare the results of the proposed approach with a series of state-of-the-art methods including traditional methods and deep learning methods.

Conclusion

Since the existing CNN-based image-to-image retinal vessel segmentation methods fail to locate some details and hard examples such as vascular edges and tiny thin vessels, in this paper we proposed a novel retinal vessel segmentation method by combining multiscale CNN with an improved loss function and CRFs to detect more details and hard examples. Both the improvements have optimized the performance of the segmentation to some extent. On the one hand, the multiscale manner has detected more

Acknowledgments

The authors would like to thank Dr. Huazhu Fu from Agency for Science, Technology and Research (A*STAR) of Singapore for kindly providing the link to the related dataset and some helpful discussions about the experiment. The authors would also like to thank the reviewers for their insightful comments, which have greatly helped to improve the quality of this paper. This work was supported by the National Natural Science Foundation of China under Grant no. 61771415 and the Cernet Innovation

Kai Hu received the B.S. degree in Computer Science and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2007 and 2013, respectively. He was a Visiting Scholar at the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, from 2016 to 2017. His current research interests include machine learning, pattern recognition, and medical image processing.

References (33)

  • Y. Zhao et al.

    Automated vessel segmentation using infinite perimeter active contour model with hybrid region information with application to retinal images

    IEEE Trans. Med. Imag.

    (2015)
  • A. Salazar-Gonzalez et al.

    Segmentation of the blood vessels and optic disk in retinal images

    IEEE J. Biomed. Health Inf.

    (2014)
  • X. Yin et al.

    Accurate image analysis of the retina using hessian matrix and binarisation of thresholded entropy with application of texture mapping

    Plos One

    (2014)
  • D. Marín et al.

    A new supervised method for blood vessel segmentation in retinal images by using gray-level and moment invariants-based features

    IEEE Trans. Med. Imag.

    (2011)
  • G. Litjens et al.

    A survey on deep learning in medical image analysis

    Med. Image Anal.

    (2017)
  • Qikui Zhu et al.

    Deeply-supervised cnn for prostate segmentation

    Proceedings of the International Joint Conference on Neural Networks

    (2017)
  • Cited by (283)

    • A review of retinal vessel segmentation for fundus image analysis

      2024, Engineering Applications of Artificial Intelligence
    View all citing articles on Scopus

    Kai Hu received the B.S. degree in Computer Science and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2007 and 2013, respectively. He was a Visiting Scholar at the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, from 2016 to 2017. His current research interests include machine learning, pattern recognition, and medical image processing.

    Zhenzhen Zhang received the B.S. degree in Electronic Information Science and Technology from Yantai University, Shandong, China, in 2014. Now she is pursuing the M.S. degree in Information and Communication Engineering from Xiangtan University, China. Her current research interests are deep learning and medical image processing.

    Xiaorui Niu received the B.S. degree in Communication Engineering from Huanghuai University, Henan, China, in 2015. Now she is pursuing the M.S. degree in Information and Communication Engineering from Xiangtan University, China. Her current research interests are machine learning and medical image processing.

    Yuan Zhang received the B.S. degree in Biomedical Engineering from Zhengzhou University, Henan, China, in 2009, and the M.S. degree in Information and Communication Engineering from Xiangtan University, Hunan, China, in 2012. She is currently pursuing the Ph.D. degree in Computational Mathematics from Xiangtan University. Her research interests focus on wavelet analysis, machine learning, and biomedical signal processing.

    Chunhong Cao received the B.S. degree in Computational Mathematics and Applied Software from Central South University, Hunan, China, in 1999, the M.S. degree in Computer Software and Theory and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2005 and 2017, respectively. She is an Associate Professor in the College of Information Engineering, Xiangtan University. Her current research interests include compressed sensing, pattern recognition, and image processing.

    Fen Xiao received the B.S. degree in Computer Science and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2002 and 2008, respectively. She was a Visiting Scholar at the Pacific Northwest National Laboratory, United States, from 2015 to 2016. She is currently a Professor in the College of Information Engineering, Xiangtan University. Her research interests are wavelet analysis theory, neural network, and image processing.

    Xieping Gao was born in 1965. He received the B.S. and M.S. degrees from Xiangtan University, China, in 1985 and 1988, respectively, and the Ph.D. degree from Hunan University, China, in 2003. He is a Professor in the College of Information Engineering, Xiangtan University, China. He was a visiting scholar at the National Key Laboratory of Intelligent Technology and Systems, Tsinghua University, China, from 1995 to 1996, and at the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, from 2002 to 2003. He is a regular reviewer for several journals and he has been a member of the technical committees of several scientific conferences. He has authored and co-authored over 80 journal papers, conference papers, and book chapters. His current research interests are in the areas of wavelets analysis, neural networks, image processing, computer network, mobile communication, and bioinformatics.

    View full text