Elsevier

Neurocomputing

Volume 358, 17 September 2019, Pages 10-19
Neurocomputing

MMAN: Multi-modality aggregation network for brain segmentation from MR images

https://doi.org/10.1016/j.neucom.2019.05.025Get rights and content

Abstract

Brain tissue segmentation from Magnetic resonance (MR) image is significant for assessing both neurologic conditions and brain disease. Manual brain tissue segmentation is time-consuming, tedious and subjective which indicates a need for more efficiently automated approaches. However, due to ambiguous boundaries, anatomically complex structure and individual differences, conventional automated segmentation methods performed poorly. Therefore, more effective feature extraction techniques and advanced segmentation models are in essential demand. Inspired by deep learning concepts, we propose a multi-modality aggregation network (MMAN), which is able to extract multi-scale features of brain tissues and harness complementary information from multi-modality MR images for fast and accurate segmentation. Extensive experiments on the well-known MRBrainS Challenge database corroborate the efficiency of the proposed model. Within approximately thirteen seconds, the MMAN can segment three different brain tissues from MRI data of each individual, that is faster than many existing methods. For the segmentation of gray matter, white matter, and cerebrospinal fluid, the MMAN achieved dice coefficients of 86.40%, 89.70% and 84.86%, respectively. Consequently, the proposed model outperformed many state-of-the-art methods and got the second place in the MRBrainS Challenge. Therefore, the proposed MMAN is promising for automated brain segmentation in clinical applications.

Introduction

The segmentation of brain tissues from magnetic resonance (MR) images is a prerequisite for quantifying structural brain volumes, evaluating neurologic condition and diagnosing brain diseases like Alzheimer’s disease, dementia, focal epilepsy, parkinsonism, and multiple sclerosis [1], [2], [3], [4]. Based on magnetic resonance imaging (MRI) technique, key brain tissues like gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF) become visible [5]. The segmentation of key brain tissues is significant for visualizing and analyzing brain tissue growth [6], measuring age-related cortical thickness changes [7], studying pathological regions of brain tumors [8], performing image-guided interventions and surgical planning [9]. In early clinical studies, the contours of brain tissues were manually segmented by clinical experts, which was time-consuming, laborious and very sensitive to high intra-expert and inter-expert variability [10]. Therefore, effective automated segmentation approaches for brain tissues are highly desired in clinical applications.

In order to facilitate the development of automated brain segmentation methods, the MRBrainS Challenge was launched at the medical image computing and computer assisted intervention (MICCAI) conference that the benchmark MRBrainS database was released [11]. The MRBrainS database was acquired by multi-sequence (multi-modality) 3T MRI scans including modalities of T1-weighted, T1-weighted inversion recovery (T1-IR) and T2-weighted fluid attenuated inversion recovery (T2-FLAIR). In MRBrainS database, clinical experts manually segmented MR images into GM, WM and CSF, which was then used as ground truth. As shown in Fig. 1, MR images of different modalities and the corresponding ground truth of brain tissue segmentation are presented. Consequently, different automated segmentation methods were proposed and evaluated on common benchmark MRBrainS database.

Recently, many teams took part in the MRBrainS Challenge that a series of segmentation methods and strategies were utilized. A novel method based on an atlas of classifiers (AOC) was proposed [12]. Compared with classical probabilistic atlas or multi-atlas approaches, AOC was more informative and economical in efficient memory utilization and computational complexity. Compared with conventional brain segmentation methods [12], [13], deep learning approaches have recently achieved tremendous success in biomedical image processing tasks [14], [15], [16]. Moreover, deep learning approaches achieved better performance than the conventional methods in the MRBrainS Challenge [11]. A deep network, termed parallel multi-dimensional long short-term memory (PyraMiD-LSTM), was proposed which utilized contexts within each slice of MRI data for segmentation [17]. This aforementioned PyraMiD-LSTM achieved better performance than those atlas-based methods. Compared with PyraMiD-LSTM, multi-dimensional gated recurrent units (MDGRU) improved performance in areas of speed, accuracy and memory efficiency [18]. Conventional deep networks such as densely convolution networks and fully convolution network could be extended into 3D form for volumetric brain segmentation. The 3D densely convolution networks, 3D fully convolution network and 3D U-net also achieved impressive performance in the MRBrainS Challenge [19], [20], [21]. A deep voxelwise residual network (VoxResNet) was proposed to utilize contextual information of multi-modality MR images from volumetric segmentation [22]. Moreover, a hyper-densely connected network (HyperDenseNet) was proposed to combine multi-modal MR images for accurate segmentation [23]. Nevertheless, the existing deep segmentation networks did not study multi-scale features of brain tissues in-depth. Multi-scale features played important roles in object detection [24], visual recognition [25] and semantic segmentation [26]. Multi-scale features extraction approach also showed advantages in analyzing brain MRI data of patients with mild cognitive impairment [27]. Additionally, MR images of different modalities may contain complementary information of contours between different brain tissues. If we can effectively learn multi-scale features of brain tissues while extracting complementary information from multi-modality MR images, brain segmentation performance can be further improved.

In this paper, we propose a novel network to extract and aggregate multi-scale features of brain tissues from multi-modality MR images for accurate segmentation. In summary, the contributions of this paper are as follows:

  • 1.

    A novel network, termed multi-modality aggregation network (MMAN), is proposed for brain segmentation.

  • 2.

    MMAN can effectively learn multi-scale features and extract complementary information from multi-modality MR images for fast and accurate segmentation.

  • 3.

    Extensive experiments on the benchmark MRBrainS Challenge database corroborate the efficiency of MMAN, which outperformed most of participants and got the second place in the challenge.

The remainder of this paper is organized as follows: The multi-modality aggregation network (MMAN) is presented in Section 2. In Section 3, numerical experiments are carried out, and corresponding segmentation results are presented. Some discussions are presented in Section 4. The conclusions are given in Section 5.

Section snippets

Proposed method

In the past few years, deep learning approaches have achieved significant success in many areas [28]. With deeper structure, deep learning models were able to learn more representative features and achieve better performance than conventional models in brain segmentation [11], [29], [30]. In this section, we propose a novel deep network (termed multi-modality aggregation network, MMAN) to extract and aggregate multi-scale features of brain tissues from multi-modality MR images for more accurate

Experiments

In this section, the experiments of the proposed MMAN on the benchmark MRBrainS database are presented. Detailed data normalization procedures and training parameter settings of the proposed method are also illustrated. Additionally, the experimental results of MMAN are presented and compared with other state-of-the-art methods.

All tests were performed on hardware with following specifications: Desktop, Intel i7 3.6GHz CPU, 16G DDR3 RAM, Nvidia Titan X, Ubuntu 16.04, Keras with tensorflow

Discussion

In the previous section, the experimental results demonstrate the MMAN’s ability to achieve state-of-the-art performance in brain segmentation. In this section, the performance of MMAN based on MRI data of different modalities are presented. And the influence of different modalities for brain segmentation is also discussed. Additionally, the features of different levels in MMAN are analyzed. A few segmentation examples are presented for ease of analysis. These examples can help to analyze the

Conclusion

In this paper, we develop a multi-modality aggregation network (MMAN) for brain segmentation from multi-modality MR images. Our method combines dilated convolution and Inception structure to effectively extract and aggregate multi-scale features of brain tissues for accurate segmentation. Extensive experiments on benchmark MRBrainS Chanllenge database corroborate the superiority of the proposed method. The segmentation of brain tissues is a prerequisite for the quantification of brain

Conflict of Interest

The authors have declared that no conflict of interest exists.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 61836003, 61573150, 61573152, 61633010.

Jingcong Li received the B.S. degree in automation from South China University of Technology, Guangzhou, China, in 2013. He is currently working toward the Ph.D. degree in pattern recognition and intelligent systems at the South China University of Technology, Guangzhou, China. His research interests include signal processing, machine learning and their applications in biomedical engineering.

References (40)

  • HuK. et al.

    Multi-scale features extraction from baseline structure mri for MCI patient classification and ad early diagnosis

    Neurocomputing

    (2016)
  • J.R. Petrella et al.

    Neuroimaging and early diagnosis of alzheimer disease: a look to the future

    Radiology

    (2003)
  • A. Giorgio et al.

    Clinical use of brain volumetry

    JMRI

    (2013)
  • C. Vijayakumar et al.

    Development of image-processing software for automatic segmentation of brain tumors in MR images

    Med. Phys.

    (2011)
  • M.A. Balafar et al.

    Review of brain MRI image segmentation methods

    Artif. Intell. Rev.

    (2010)
  • I. Despotovic et al.

    MRI segmentation of the human brain: challenges, methods, and applications

    Comput. Math. Methods Med.

    (2015)
  • A.M. Mendrik et al.

    MRBrainS challenge: Online evaluation framework for brain image segmentation in 3T MRI scans

    Comput. Intell. Neurosci.

    (2015)
  • B. Kodner et al.

    Atlas of classifiers for brain MRI segmentation

    Proceedings of the MICCAI

    (2017)
  • M.F. Stollenga et al.

    Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation

    Proceedings of the NIPS

    (2015)
  • S. Andermatt et al.

    Multi-dimensional gated recurrent units for the segmentation of biomedical 3D-data

    Proceedings of the MICCAI

    (2016)
  • Cited by (44)

    • MR brain segmentation based on DE-ResUnet combining texture features and background knowledge

      2022, Biomedical Signal Processing and Control
      Citation Excerpt :

      Dolz et al. [29] proposed HyperDenseNet, a 3D fully convolutional neural network that extended the definition of dense connectivity to multi-modal segmentation problems. Li et al. [30] developed a multi-modal aggregation network (MMAN) for brain segmentation from multi-modal MR images. In [31], the authors developed a 3D spatially-weighted network for brain MRI tissue segmentation called SW-3D-UNet, and extended it to the multi-modal MRI environment.

    • A robust discriminative multi-atlas label fusion method for hippocampus segmentation from MR image

      2021, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      Early detection of changes in the hippocampus is essential for the early prevention and diagnosis of neural diseases. Hence, the automatic and accurate extraction of hippocampus from Magnetic Resonance (MR) images has become a pivotal task in medical image analysis [2,3]. Due to the irregular shape of the hippocampus and the blurred boundary with surrounding tissues, the automatic and accurate segmentation of the hippocampus is a challenging task.

    View all citing articles on Scopus

    Jingcong Li received the B.S. degree in automation from South China University of Technology, Guangzhou, China, in 2013. He is currently working toward the Ph.D. degree in pattern recognition and intelligent systems at the South China University of Technology, Guangzhou, China. His research interests include signal processing, machine learning and their applications in biomedical engineering.

    Zhu Liang Yu received his BSEE in 1995 and MSEE in 1998, both in electronic engineering from the Nanjing University of Aeronautics and Astronautics, China. He received his Ph.D. in 2006 from Nanyang Technological University, Singapore. He joined Center for Signal Processing, Nanyang Technological University from 2000 as a research engineer, then as a Group Leader from 2001. In 2008, he joined the College of Automation Science and Engineering, South China University of Technology and was promoted to be a full professor in 2011. His research interests include signal processing, pattern recognition, machine learning and their applications in communications, biomedical engineering, etc.

    Zhenghui Gu received the Ph.D. degree from Nanyang Technological University in 2003. From 2002 to 2008, she was with Institute for Infocomm Research, Singapore. She joined the College of Automation Science and Engineering, South China University of Technology, in 2009 as an associate professor. She was promoted to be a full professor in 2015. Her research interests include the fields of signal processing and pattern recognition.

    Yuanqing Li was born in Hunan Province, China, in 1966. He received the B.S. degree in applied mathematics from Wuhan University, Wuhan, China, in 1988, the M.S. degree in applied mathematics from South China Normal University, Guangzhou, China, in 1994, and the Ph.D. degree in control theory and applications from South China University of Technology, Guangzhou, China, in 1997. Since 1997, he has been with South China University of Technology, where he became a full professor in 2004. In 2002-04, he worked at the Laboratory for Advanced Brain Signal Processing, RIKEN Brain Science Institute, Saitama, Japan, as a researcher. In 2004-08, he worked at the Laboratory for Neural Signal Processing, Institute for Inforcomm Research, Singapore, as a research scientist. His research interests include, blind signal processing, sparse representation, machine learning, brain computer interface, EEG and fMRI data analysis. He is the author or coauthor of more than 60 scientific papers in journals and conference proceedings.

    View full text