3D dense connectivity network with atrous convolutional feature pyramid for brain tumor segmentation in magnetic resonance imaging of human heads

Highlights

• A densely connected convolutional neural network is developed as the backbone to learn dense features of brain tumors via MR images.

• A 3D atrous convolution feature pyramid is connected with the end of the backbone to learn multiscale features of brain tumors.

• A 3D deep supervision mechanism is further proposed to guide the optimization of the gradient flow during training.

Abstract

The existing deep convolutional neural networks (DCNNs) based methods have achieved significant progress regarding automatic glioma segmentation in magnetic resonance imaging (MRI) data. However, there are two main problems affecting the performance of traditional DCNNs constructed by simply stacking convolutional layers, namely, exploding/vanishing gradients and limitations to the feature computations. To address these challenges, we propose a novel framework to automatically segment brain tumors. First, a three-dimensional (3D) dense connectivity architecture is used to build the backbone for feature reuse. Second, we design a new feature pyramid module using 3D atrous convolutional layers and add this module to the end of the backbone to fuse multiscale contexts. Finally, a 3D deep supervision mechanism is equipped with the network to promote training. On the multimodal brain tumor image segmentation benchmark (BRATS) datasets, our method achieves Dice similarity coefficient values of 0.87, 0.72, and 0.70 on the BRATS 2013 Challenge, 0.84, 0.70, and 0.61 on the BRATS 2013 LeaderBoard, 0.83, 0.70, and 0.62 on the BRATS 2015 Testing, 0.8642, 0.7738, and 0.7525 on the BRATS 2018 Validation in terms of whole tumors, tumor cores, and enhancing cores, respectively. Compared to the published state-of-the-art methods, the proposed method achieves promising accuracy and fast processing, demonstrating good potential for clinical medicine.

Introduction

Brain tumors are a common clinical cancer of the nervous system that occurs in the cranial cavity and can be classified into two categories from their initial origin, i.e., primary brain tumors and metastatic brain tumors [1]. Brain tumors in the first category originate from brain lesions, while those in the other category originate from cancerous tissues in any other part of the body. Brain tumors are severely dangerous diseases that divide and grow uncontrollably, gradually damaging the nervous system and ultimately leading to death. To protect healthy tissues but damage tumor cells during therapy, segmenting the tumor becomes crucial before applying any therapy. Therefore, it is crucial to detect brain tumors early for timely treatment and prolonging the life of the patient. Since the advantage of MRI is its high capability to present extensive information about the soft tissue of the brain with both noninvasive and radiation-free characteristics, MRI is a popular technique to diagnose brain tumors. However, the tumor boundary is usually unclear, discontinuous, and irregular. Moreover, the intensities or other variations in MR images usually vary since both the imaging devices and environments seriously differ. These conditions make manually annotating a large number of multimodal MRI subjects impractical. Therefore, developing robust methods for automatic or semiautomatic brain tumor segmentation has become an interesting and challenging research topic.

With the rapid development of artificial intelligence (AI), especially the breakthroughs in deep learning techniques and the rise of medical big data, many researchers have started to focus on how to efficiently study medical images via AI [2]. The methods can be classified into none DCNNs-based and DCNN-based methods. The none DCNNs-based methods usually employ hand-crafted features with specific classifiers for processing medical images. Soltaninejad et al. [3] used a support vector machine (SVM) on statistical features for brain tumor grading. Superpixels/supervoxels were generated from a multimodal MRI dataset to extract statistical features, and then random forests were implemented to classify each superpixel/supervoxel [4], [5]. Yang et al. [6] proposed a 3D morphological analysis for tumor segmentation. Raschke et al. [7] proposed a retrospective analysis of MRI from patients with a histological diagnosis of glioma that used Bayesian inference to achieve the whole brain tumor tissue-type maps and automated segmentation. For classifying the tumor type, Timothy et al. [8] proposed whole-brain diffusion tensor imaging segmentation to delineate the tumor volumes of interest. Similar to tensor imaging, Yang et al. [9] presented a computerized decision support framework that discriminates between high-grade glioblastoma multiform and solitary metastasis using MRI. Due to the state-of-the-art performance achieved by DCNNs in the field of computer vision, DCNNs-based methods have been gradually applied in medical image segmentation tasks. In contrast to the previous none DCNNs-based methods, DCNNs-based techniques are a category of data-driven methods that build complex models via representation learning mechanisms in an end-to-end manner. The existing DCNNs-based methods applied to medical images can be classified into 2D DCNNs and 3D DCNNs. The first category of networks [10], [11], [12], [13] usually uses a multistream (pathway) architecture. These architectures are also referred to as cascaded networks. In these networks, the input images are sampled from the same original subject and then input separately for feature learning and dense prediction. In contrast to multistream architecture, another U-Net-based method [14], whose lower layers are responsible for transforming the input into low-resolution representations whereas deeper layers produce pixelwise predictions using upsampling operations, has received more attention for brain tumor segmentation [15], [16], [17], [18], [19], [20], [21]. Although the 2D DCNNs-based methods have achieved promising performance, they only take into account the local dependencies of the ground truth but ignore the appearance and spatial consistency. The use of common 2D DCNNs-based methods on 3D MR images fails to detect some important spatial information. To address this issue, 3D DCNNs-based models [22], [23], [24], [25], [26], whose convolutional layers consist of 3D filters, were developed. Similar to 2D DCNNs-based methods, the general architectures of 3D DCNNs are classified into multistream and U-Net. Different from 2D DCNNs-based methods, these 3D DCNNs-based approaches directly extract features from MRI data without fusing the final results. Hence, the processing pipeline of 3D DCNNs is easier to implement than that of 2D DCNNs.

Although the aforementioned networks have greatly promoted the development of medical image segmentation, some challenges still exist in brain tumor segmentation. On the one hand, most existing DCNNs-based approaches strengthen the representative ability by stacking more layers. However, simply increasing layers results in exploding/vanishing gradients, which has a negative influence on training. Additionally, the networks usually feed the feature maps produced by the last hidden layer to the final classifier. Under this stacked method, the features of previous layers are indirectly input to the end of the network, and thus, the gradient flow may be impeded during training. On the other hand, brain tumors appear in a variety of sizes and shapes in MR images. This requires DCNNs-based models to be able to analyze and handle tumors of various scales. Therefore, single-scale-based networks have limitations in feature computation to address this challenge. In this manuscript, we propose a 3D dense connectivity convolutional neural network equipped with an atrous convolutional feature pyramid and 3D deep supervision mechanism to comprehensively segment brain tumors. The proposed network is referred to as DenseAFPNet. The contributions of this study are summarized as follows:

• To alleviate the difficulties in training deep networks and improve the efficiency of the gradient flow, a densely connected convolutional neural network is developed as the backbone for learning dense features for brain tumors via MR images.

• A 3D hierarchical feature pyramid using 3D atrous convolutional layers is connected with the end of the backbone to learn multiscale features of brain tumors. The proposed pyramid provides the network with different receptive fields and thus fuses the contextual information with the lesions to improve the segmentation performance.

• Through the introduction of an auxiliary loss function for each pyramidal hierarchy, a 3D deep supervision mechanism is further proposed. Such a mechanism guides the optimization of both the lower and upper hierarchies to reinforce the propagation of the gradient flow during training, and thus this mechanism makes the network learn more discriminating and powerful features. The mechanism is not only a strong regularization technique as a prompter to improve the convergence of the network but also provides an effective strategy for addressing the challenge of overfitting.