Retinal vessel segmentation of color fundus images using multiscale convolutional neural network with an improved cross-entropy loss function
Introduction
There are a lot of people in the world who are suffering from blindness. Some ophthalmologic diseases such as diabetes, hypertension, and arteriosclerosis [1] are main causes of blindness that cannot be ignored. These diseases are related to morphological changes of the vessels in diameter, tortuosity, branching pattern or angles [2]. Experts can analyze changes in vascular morphology by segmenting the retinal vessels. Therefore, the segmentation task of the retinal vessels is of great importance for the diagnosis of related diseases. The medical images most commonly used for the ophthalmic diseases are two-dimensional (2-D) color fundus images and three-dimensional (3-D) optic coherence tomography (OCT) images [3]. As the color fundus image can be obtained more conveniently just through a fundus camera by contrast to OCT image, it has been widely used in the analysis of some ophthalmologic diseases.
Usually, specialists segment the retinal vessels of fundus images manually. However, manual segmentation by experts requires expertise and is time consuming [4]. It is not applicable for large-scale screening or diagnosis work regardless of time or cost. Therefore, an effective and efficient automatic vessel segmentation manner is particularly meaningful. With the development of computer-aided diagnosis (CAD) system, various vessel segmentation methods have been proposed. These methods can segment automatically without depending on the experts and provide reliable and robust segmentation. However, affected by many factors such as the lesion area, the complicated vessel structure and the low contrast between the target and the background [1], the segmentation task remains challenging.
The existing segmentation methods can be divided into supervised or unsupervised categories [4] according to whether the manual labeled ground truths are used or not. Obviously, the unsupervised approaches are designed according to the inherent characteristics of the blood vessels without relying on the manual labeled map. Salem et al. [5] proposed an unsupervised approach by using RAdius based Clustering ALgorithm (RACAL) which mapped the distribution of image pixels through the distance based principle. Ng et al. [6] developed a maximum likelihood inversion model for the retinal vessel segmentation task. Neto et al. [7] proposed an unsupervised coarse-to-fine algorithm. In this approach, they segmented vascular structure relied on mathematical morphology, spatial dependency, and curvature. Khan et al. [8] proposed a contrast-sensitive segmentation method using background normalization, second-order Gaussian filter and region growing. They pay attention to the low-contrast region in a contrast-sensitive manner. Zhao et al. [9] proposed an infinite perimeter active contour model which employed Lebesgue measurement to detect small vessels. Their method combined different region information to guarantee the detection of vessels and vessels edges. Salazar-Gonzalez et al. [10] first carried out a preprocessing for the image by adaptive histogram equalization and robust distance transform, and then segmented the vessels with graph cut. Yin et al. [11] proposed a segmentation method using hessian matrix and thresholded entropy. They used post-processing to eliminate noise and the central light reflex. These unsupervised methods performed well when detecting retinal vessel according to the vascular structure and some prior knowledge without using ground truths.
Meanwhile, the supervised approaches extract retinal blood vessels by learning rules from the annotation results. Marin et al. [12] treated the segmentation task as a pixel classification problem. They first extracted gray-level features and moment invariant features for every pixel and then put these features into neural networks to classify the pixel. Zhu et al. [2] constructed a 39-D discriminative feature vector such as local features, Hessian, divergence of vector field and so on for every pixel based on prior knowledge, and then classify them into vessel or not using the extreme learning machine (ELM) classifier. Generally, most of these methods depend on prior knowledge or a series of complex processing to get discriminative features.
Convolutional neural network (CNN) can automatically learn hierarchical features by some convolution and pooling operations without prior knowledge and extra preprocess. In recent years, due to the efficiency of CNN, it has been widely used in computer vision, medical image processing, and pattern recognition [13], [14].
Wang et al. [1] segmented the retinal vessels based on a pixel classification mode. They proposed a hierarchical segmentation method which used CNN as a trainable feature extractor and employed the ensemble random forests (RFs) as a trainable classifier. In order to determine what each pixel belongs to, they extracted features of a square subwindow centered on the pixel that need to be classified with CNN and then predicted with ensemble RFs. This patch-based method is time-consuming since it has a large number of repeatable calculations. Furthermore, most of existing image-to-image CNN methods performed better when detecting large objects. However, retinal vessel structures are thin elongated [10] and therefore it is difficult to segment the vessels accurately. In addition, the imbalance problem further increases the difficulty of the vessel segmentation, since the vessel pixels account for only about 10% of the pixels of the entire image. Inspired by the holistically-nested edge detection (HED) [15] system, Fu et al. [16] formulated the retinal vessel segmentation task into a boundary detection task. They first applied HED to get a probability map and then utilized fully connected conditional random fields (CRFs) to refine the segmentation. This image-to-image method was more efficient and the segmentation accuracy achieved at the state-of-the-art level. With the increase of HED network layers, the receptive field was gradually increased to learn more global discriminative features to distinguish vessels and non-vessels.
However, the top layers contain more global information while the bottom layers contribute more details. Fu et al.’s [16] method lost some vascular edges and thin vessels inevitably by only adopting part of the top layers to ensure the accuracy of the overall structure and the majority of the vascular. Therefore, although the existing CNN methods usually learn the discriminative information by increasing the depth of the network, the details contained in the shallow layers cannot be ignored for detecting thin elongated vascular structure. More specifically, the image pixels can be divided into two categories: easy to detect and hard to detect. As the easy pixels occupy the majority in the fundus images, they often dominated the direction of parameter update in the learning process of CNN. There are still some hard examples that have not been detected accurately when the loss stabilizes and the CNN model stops to update.
In this paper, in order to segment the retinal vessels more efficiently and effectively, we propose a supervised image-to-image retinal vessel segmentation system by combining multiscale CNN and CRFs. Firstly, we used a multiscale CNN to get a probability map. The proposed CNN model uses a multiscale architecture to fuse richer multiscale information for comprehensive vascular feature description. Meanwhile, we develop an improved loss function for emphasizing hard examples to effectively detect small objects and overcome the imbalance problem. Later, to smooth the coarse probability map, we define the CNN output probability map as unary potentials of the CRFs model to obtain the final segmentation map. Finally, we carry out experiments on two public datasets and conduct performance evaluations to verify the segmentation effect. Results on both datasets show that our method outperformed most of the recent approaches in terms of sensitivity and accuracy. The contributions of this work are mainly reflected in two aspects of the CNN construction.
- 1)
We put all of the middle convolution features into the final CNN classifier to learn a richer multiscale information [17] so that we can locate vessel edges precisely and detect more tiny vessels. Especially, by combining the features in each layer, we take full advantage of the information of every scale to improve the detection accuracy of the vessels edges and the tiny thin vessels.
- 2)
We propose an improved class-balanced cross-entropy loss function that further improves the segmentation performance by addressing the segmentation of hard samples such as the vascular edge and the tiny thin vessels.
The remainder of this paper is organized as follows. Section 2 provides a detailed description of the proposed method including multiscale CNN architecture, the improved loss function, and the CRFs. Section 3 introduces the image datasets, the experimental settings and the evaluation metrics. In Section 4, we discuss and compare our experimental results from many aspects. Finally, a conclusion with future work is drawn in Section 5.
Section snippets
Method
The flowchart of the proposed method is shown in Fig. 1. We first propose a multiscale CNN to learn probability map. The multiscale CNN architecture is used to learn discriminatory features from the color fundus images. Meanwhile, an improved cross-entropy loss function is proposed to emphasize the hard examples. Furthermore, we use CRF to mitigate noise and edge blurring, thus refining the probability map to get the final segmentation result. The details of the above steps are presented as
Datasets
The evaluation is conducted on two public datasets for retinal vessel segmentation of fundus images: DRIVE [23] and STARE [24].
DRIVE contains 40 images obtained at 45° field of view (FOV) with a resolution of 565 × 584 pixels. 20 images of them are selected as the training set and another 20 images are used as the testing set. Each training image has one ground truth segmented by a specialist. Each testing image has two manual segmentation results segmented by two specialists with the first one
Results and discussion
We conduct a series of experiments on two datasets to confirm the validity of our method with a variety of indicators and illustrations. We also compare the results of the proposed approach with a series of state-of-the-art methods including traditional methods and deep learning methods.
Conclusion
Since the existing CNN-based image-to-image retinal vessel segmentation methods fail to locate some details and hard examples such as vascular edges and tiny thin vessels, in this paper we proposed a novel retinal vessel segmentation method by combining multiscale CNN with an improved loss function and CRFs to detect more details and hard examples. Both the improvements have optimized the performance of the segmentation to some extent. On the one hand, the multiscale manner has detected more
Acknowledgments
The authors would like to thank Dr. Huazhu Fu from Agency for Science, Technology and Research (A*STAR) of Singapore for kindly providing the link to the related dataset and some helpful discussions about the experiment. The authors would also like to thank the reviewers for their insightful comments, which have greatly helped to improve the quality of this paper. This work was supported by the National Natural Science Foundation of China under Grant no. 61771415 and the Cernet Innovation
Kai Hu received the B.S. degree in Computer Science and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2007 and 2013, respectively. He was a Visiting Scholar at the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, from 2016 to 2017. His current research interests include machine learning, pattern recognition, and medical image processing.
References (33)
- et al.
Hierarchical retinal blood vessel segmentation based on feature and ensemble learning
Neurocomputing
(2015) - et al.
Retinal vessel segmentation in colour fundus images using extreme learning machine
Comput. Med. Imag. Graph.
(2017) - et al.
Blood vessel segmentation methodologies in retinal images–a survey
Comput. Methods Progr. Biomed.
(2012) - et al.
Maximum likelihood estimation of vessel parameters from scale space analysis
Image Vis. Comput.
(2010) - et al.
Trainable COSFIRE filters for vessel delineation with application to retinal images
Med. Image Anal.
(2015) - et al.
Segmentation of optic disc, fovea and retinal vasculature using a single convolutional neural network
J. Comput. Sci.
(2017) - et al.
Retinal imaging and image analysis
IEEE Rev. Biomed. Eng.
(2010) - et al.
Segmentation of retinal blood vessels using a novel clustering algorithm (RACAL) with a partial supervision strategy
Med. Biol. Eng. Comput.
(2007) - et al.
An unsupervised coarse-to-fine algorithm for blood vessel segmentation in fundus images
Expert Syst. Appl. Int. J.
(2017) - et al.
Automatic retinal vessel extraction algorithm based on contrast-sensitive schemes
Proceedings of the International Conference on Image and Vision Computing New Zealand
(2016)
Automated vessel segmentation using infinite perimeter active contour model with hybrid region information with application to retinal images
IEEE Trans. Med. Imag.
Segmentation of the blood vessels and optic disk in retinal images
IEEE J. Biomed. Health Inf.
Accurate image analysis of the retina using hessian matrix and binarisation of thresholded entropy with application of texture mapping
Plos One
A new supervised method for blood vessel segmentation in retinal images by using gray-level and moment invariants-based features
IEEE Trans. Med. Imag.
A survey on deep learning in medical image analysis
Med. Image Anal.
Deeply-supervised cnn for prostate segmentation
Proceedings of the International Joint Conference on Neural Networks
Cited by (283)
Global and local multi-modal feature mutual learning for retinal vessel segmentation
2024, Pattern RecognitionSEG-LUS: A novel ultrasound segmentation method for liver and its accessory structures based on muti-head self-attention
2024, Computerized Medical Imaging and GraphicsSatellite image recognition using ensemble neural networks and difference gradient positive-negative momentum
2024, Chaos, Solitons and FractalsA review of retinal vessel segmentation for fundus image analysis
2024, Engineering Applications of Artificial IntelligenceMulti-scale Perception and Feature Refinement Network for multi-class segmentation of intracerebral hemorrhage in CT images
2024, Biomedical Signal Processing and Control
Kai Hu received the B.S. degree in Computer Science and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2007 and 2013, respectively. He was a Visiting Scholar at the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, from 2016 to 2017. His current research interests include machine learning, pattern recognition, and medical image processing.
Zhenzhen Zhang received the B.S. degree in Electronic Information Science and Technology from Yantai University, Shandong, China, in 2014. Now she is pursuing the M.S. degree in Information and Communication Engineering from Xiangtan University, China. Her current research interests are deep learning and medical image processing.
Xiaorui Niu received the B.S. degree in Communication Engineering from Huanghuai University, Henan, China, in 2015. Now she is pursuing the M.S. degree in Information and Communication Engineering from Xiangtan University, China. Her current research interests are machine learning and medical image processing.
Yuan Zhang received the B.S. degree in Biomedical Engineering from Zhengzhou University, Henan, China, in 2009, and the M.S. degree in Information and Communication Engineering from Xiangtan University, Hunan, China, in 2012. She is currently pursuing the Ph.D. degree in Computational Mathematics from Xiangtan University. Her research interests focus on wavelet analysis, machine learning, and biomedical signal processing.
Chunhong Cao received the B.S. degree in Computational Mathematics and Applied Software from Central South University, Hunan, China, in 1999, the M.S. degree in Computer Software and Theory and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2005 and 2017, respectively. She is an Associate Professor in the College of Information Engineering, Xiangtan University. Her current research interests include compressed sensing, pattern recognition, and image processing.
Fen Xiao received the B.S. degree in Computer Science and the Ph.D. degree in Computational Mathematics from Xiangtan University, Hunan, China, in 2002 and 2008, respectively. She was a Visiting Scholar at the Pacific Northwest National Laboratory, United States, from 2015 to 2016. She is currently a Professor in the College of Information Engineering, Xiangtan University. Her research interests are wavelet analysis theory, neural network, and image processing.
Xieping Gao was born in 1965. He received the B.S. and M.S. degrees from Xiangtan University, China, in 1985 and 1988, respectively, and the Ph.D. degree from Hunan University, China, in 2003. He is a Professor in the College of Information Engineering, Xiangtan University, China. He was a visiting scholar at the National Key Laboratory of Intelligent Technology and Systems, Tsinghua University, China, from 1995 to 1996, and at the School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore, from 2002 to 2003. He is a regular reviewer for several journals and he has been a member of the technical committees of several scientific conferences. He has authored and co-authored over 80 journal papers, conference papers, and book chapters. His current research interests are in the areas of wavelets analysis, neural networks, image processing, computer network, mobile communication, and bioinformatics.