Multi-view longitudinal CNN for multiple sclerosis lesion segmentation

Highlights

• A convolutional neural network based method for multiple sclerosis lesion segmentation is proposed.

• The network utilizes longitudinal data, a novel contribution in the domain of MS lesion analysis.

• The use of longitudinal data significantly improves segmentation accuracy.

• State-of-the-art results are obtained on a public benchmark dataset.

• Expert human level segmentation accuracy is obtained by the proposed method.

Abstract

In this work, a deep-learning based automated method for Multiple Sclerosis (MS) lesion segmentation is presented. Automatic segmentation of MS lesions is a challenging task due to their variability in shape, size, location and texture in Magnetic Resonance (MR) images. In the proposed scheme, MR intensities and White Matter (WM) priors are used to extract candidate lesion voxels, following which Convolutional Neural Networks (CNN) are utilized for false positive reduction and final segmentation result. The proposed network uses longitudinal data, a novel contribution in the domain of MS lesion analysis. The method obtained state-of-the-art results on the 2015 Longitudinal MS Lesion Segmentation Challenge dataset, and achieved a performance level equivalent to a trained human rater. Automatic segmentation methods, such as the one proposed, once proven in accuracy and robustness, can help diagnosis and patient follow-up while reducing the time consuming need of manual segmentation.

Introduction

Multiple Sclerosis is one of the most common non-traumatic neurological diseases in young adults. It is a chronic inflammatory disease in which the immune system attacks the central nervous system (CNS) and damages myelin, myelin producing cells and underlying nerve fibers. Damages of the myelin cause scarring of brain tissues, mostly in the white matter, which are termed MS Lesions. The impairment of the CNS due to MS ultimately leads to deficiency in sensation, movement and cognition.

MRI plays an important role in the diagnosis and treatment of MS. The Revised McDonald Criteria, which incorporate the combination of clinical characteristics and MRI features, have been devised for the purpose of diagnosis of MS (Polman et al., 2005). According to these criteria, MS can be diagnosed after a single clinical episode when MS lesions are visible in MRI. Early diagnosis is important due to the availability of therapies that slow the progression of the disease. Once MS has been diagnosed, subsequent MR scans, usually performed on a yearly basis, are used to track the progress of the disease and make further treatment decisions.

Lesions may appear, disappear or change size and shape between consecutive MR scans (Guttmann et al., 1995). It is therefore the case that temporal variability in WM tissues may provide a strong indication for the presence of MS lesions, as depicted in Fig. 1. Recently, lesion segmentation algorithms that utilize longitudinal information were shown to obtain enhanced results over algorithms that segment each time point independently (Roy et al., 2015).

Due to the clinical importance and the challenging nature of automatic MS lesion detection and segmentation, several challenges have been organized, such as the MS lesion segmentation challenge in MICCAI 2008 (Styner et al., 2008) and the Longitudinal MS lesion segmentation challenge in ISBI 2015 (Carass et al., 2017). In the latter challenge, competing teams were able to make use of longitudinal data as well as multiple contrast images in order to provide accurate automatic lesion segmentations. A variety of algorithms were proposed. Top performing methods used supervised classification frameworks, such as Random Forests (Geremia et al., 2010) and Deep Neural Networks (Brosch et al., 2016).

Convolutional Neural Networks have become increasingly popular following the 2012 ImageNet classification challenge, in which Alex Krizhevsky’s network won by a large margin (Krizhevsky et al., 2012). Since then, CNNs have been successfully used for additional applications, such as object detection and segmentation. Due to the complex structure and the enormous number of parameter of CNNs, understanding why they perform so well is not straightforward, and several works have been dedicated for this purpose Zeiler and Fergus (2014), Simonyan et al. (2013). In recent years, CNNs have also been used successfully for medical image analysis, in which volumetric data is commonly available. A multi-view CNN, in which candidate 3D volumes are decomposed to axial, coronal and sagittal views, achieved state-of-the-art accuracy in lymph node detection. This multi-view framework, compared to 3D CNNs, was able to reduce the computations required for training and testing and was robust to overfitting due to the reduced number of network weights (Roth et al., 2014). Both 2D and 3D CNNs were recently proposed for the segmentation of MS lesions (Carass et al., 2017).

This work presents a longitudinal Multi-View CNN for the MS lesion segmentation task. The input to the CNN are patches from multiple views, multiple contrast images and multiple time points. To the best of our knowledge, this is the first CNN that takes advantage of longitudinal data for MS lesion segmentation. The proposed segmentation method was evaluated on the dataset provided in the 2015 ISBI challenge, and achieved state-of-the-art accuracy on the test set. This method, which was trained on the relatively small number of patients available in the challenge training set, was able to generalize well on the test set and achieve human level performance.

The rest of the paper is organized as follows: The proposed segmentation method is detailed in Section 2. Evaluations on the 2015 ISBI dataset are presented in Section 3. Experimental results are detailed in Section 4. Finally, Discussion of various aspects of the segmentation method and concluding remarks are provided in Section 5.