MMAN: Multi-modality aggregation network for brain segmentation from MR images
Introduction
The segmentation of brain tissues from magnetic resonance (MR) images is a prerequisite for quantifying structural brain volumes, evaluating neurologic condition and diagnosing brain diseases like Alzheimer’s disease, dementia, focal epilepsy, parkinsonism, and multiple sclerosis [1], [2], [3], [4]. Based on magnetic resonance imaging (MRI) technique, key brain tissues like gray matter (GM), white matter (WM), and cerebrospinal fluid (CSF) become visible [5]. The segmentation of key brain tissues is significant for visualizing and analyzing brain tissue growth [6], measuring age-related cortical thickness changes [7], studying pathological regions of brain tumors [8], performing image-guided interventions and surgical planning [9]. In early clinical studies, the contours of brain tissues were manually segmented by clinical experts, which was time-consuming, laborious and very sensitive to high intra-expert and inter-expert variability [10]. Therefore, effective automated segmentation approaches for brain tissues are highly desired in clinical applications.
In order to facilitate the development of automated brain segmentation methods, the MRBrainS Challenge was launched at the medical image computing and computer assisted intervention (MICCAI) conference that the benchmark MRBrainS database was released [11]. The MRBrainS database was acquired by multi-sequence (multi-modality) 3T MRI scans including modalities of T1-weighted, T1-weighted inversion recovery (T1-IR) and T2-weighted fluid attenuated inversion recovery (T2-FLAIR). In MRBrainS database, clinical experts manually segmented MR images into GM, WM and CSF, which was then used as ground truth. As shown in Fig. 1, MR images of different modalities and the corresponding ground truth of brain tissue segmentation are presented. Consequently, different automated segmentation methods were proposed and evaluated on common benchmark MRBrainS database.
Recently, many teams took part in the MRBrainS Challenge that a series of segmentation methods and strategies were utilized. A novel method based on an atlas of classifiers (AOC) was proposed [12]. Compared with classical probabilistic atlas or multi-atlas approaches, AOC was more informative and economical in efficient memory utilization and computational complexity. Compared with conventional brain segmentation methods [12], [13], deep learning approaches have recently achieved tremendous success in biomedical image processing tasks [14], [15], [16]. Moreover, deep learning approaches achieved better performance than the conventional methods in the MRBrainS Challenge [11]. A deep network, termed parallel multi-dimensional long short-term memory (PyraMiD-LSTM), was proposed which utilized contexts within each slice of MRI data for segmentation [17]. This aforementioned PyraMiD-LSTM achieved better performance than those atlas-based methods. Compared with PyraMiD-LSTM, multi-dimensional gated recurrent units (MDGRU) improved performance in areas of speed, accuracy and memory efficiency [18]. Conventional deep networks such as densely convolution networks and fully convolution network could be extended into 3D form for volumetric brain segmentation. The 3D densely convolution networks, 3D fully convolution network and 3D U-net also achieved impressive performance in the MRBrainS Challenge [19], [20], [21]. A deep voxelwise residual network (VoxResNet) was proposed to utilize contextual information of multi-modality MR images from volumetric segmentation [22]. Moreover, a hyper-densely connected network (HyperDenseNet) was proposed to combine multi-modal MR images for accurate segmentation [23]. Nevertheless, the existing deep segmentation networks did not study multi-scale features of brain tissues in-depth. Multi-scale features played important roles in object detection [24], visual recognition [25] and semantic segmentation [26]. Multi-scale features extraction approach also showed advantages in analyzing brain MRI data of patients with mild cognitive impairment [27]. Additionally, MR images of different modalities may contain complementary information of contours between different brain tissues. If we can effectively learn multi-scale features of brain tissues while extracting complementary information from multi-modality MR images, brain segmentation performance can be further improved.
In this paper, we propose a novel network to extract and aggregate multi-scale features of brain tissues from multi-modality MR images for accurate segmentation. In summary, the contributions of this paper are as follows:
- 1.
A novel network, termed multi-modality aggregation network (MMAN), is proposed for brain segmentation.
- 2.
MMAN can effectively learn multi-scale features and extract complementary information from multi-modality MR images for fast and accurate segmentation.
- 3.
Extensive experiments on the benchmark MRBrainS Challenge database corroborate the efficiency of MMAN, which outperformed most of participants and got the second place in the challenge.
The remainder of this paper is organized as follows: The multi-modality aggregation network (MMAN) is presented in Section 2. In Section 3, numerical experiments are carried out, and corresponding segmentation results are presented. Some discussions are presented in Section 4. The conclusions are given in Section 5.
Section snippets
Proposed method
In the past few years, deep learning approaches have achieved significant success in many areas [28]. With deeper structure, deep learning models were able to learn more representative features and achieve better performance than conventional models in brain segmentation [11], [29], [30]. In this section, we propose a novel deep network (termed multi-modality aggregation network, MMAN) to extract and aggregate multi-scale features of brain tissues from multi-modality MR images for more accurate
Experiments
In this section, the experiments of the proposed MMAN on the benchmark MRBrainS database are presented. Detailed data normalization procedures and training parameter settings of the proposed method are also illustrated. Additionally, the experimental results of MMAN are presented and compared with other state-of-the-art methods.
All tests were performed on hardware with following specifications: Desktop, Intel i7 3.6GHz CPU, 16G DDR3 RAM, Nvidia Titan X, Ubuntu 16.04, Keras with tensorflow
Discussion
In the previous section, the experimental results demonstrate the MMAN’s ability to achieve state-of-the-art performance in brain segmentation. In this section, the performance of MMAN based on MRI data of different modalities are presented. And the influence of different modalities for brain segmentation is also discussed. Additionally, the features of different levels in MMAN are analyzed. A few segmentation examples are presented for ease of analysis. These examples can help to analyze the
Conclusion
In this paper, we develop a multi-modality aggregation network (MMAN) for brain segmentation from multi-modality MR images. Our method combines dilated convolution and Inception structure to effectively extract and aggregate multi-scale features of brain tissues for accurate segmentation. Extensive experiments on benchmark MRBrainS Chanllenge database corroborate the superiority of the proposed method. The segmentation of brain tissues is a prerequisite for the quantification of brain
Conflict of Interest
The authors have declared that no conflict of interest exists.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grants 61836003, 61573150, 61573152, 61633010.
Jingcong Li received the B.S. degree in automation from South China University of Technology, Guangzhou, China, in 2013. He is currently working toward the Ph.D. degree in pattern recognition and intelligent systems at the South China University of Technology, Guangzhou, China. His research interests include signal processing, machine learning and their applications in biomedical engineering.
References (40)
- et al.
Scalable joint segmentation and registration framework for infant brain images
Neurocomputing
(2017) - et al.
Early diagnosis of Alzheimer’s disease based on partial least squares, principal component analysis and support vector machine using segmented MRI images
Neurocomputing
(2015) - et al.
Automatic quantification of normal cortical folding patterns from fetal brain MRI
NeuroImage
(2014) - et al.
Links: learning-based multi-source integration framework for segmentation of infant brain images
NeuroImage
(2015) - et al.
Longitudinal changes in cortical thickness associated with normal aging
NeuroImage
(2010) - et al.
Brain MRI image segmentation based on learning local variational Gaussian mixture models
Neurocomputing
(2016) - et al.
A deep convolutional neural network for segmenting and classifying epithelial and stromal regions in histopathological images
Neurocomputing
(2016) - et al.
A survey on deep learning in medical image analysis.
Med. Image Anal.
(2017) - et al.
Segmentation of glioma tumors in brain using deep convolutional neural network
Neurocomputing
(2018) - et al.
VoxResNet: Deep voxelwise residual networks for brain segmentation from 3D MR images
NeuroImage
(2018)
Multi-scale features extraction from baseline structure mri for MCI patient classification and ad early diagnosis
Neurocomputing
Neuroimaging and early diagnosis of alzheimer disease: a look to the future
Radiology
Clinical use of brain volumetry
JMRI
Development of image-processing software for automatic segmentation of brain tumors in MR images
Med. Phys.
Review of brain MRI image segmentation methods
Artif. Intell. Rev.
MRI segmentation of the human brain: challenges, methods, and applications
Comput. Math. Methods Med.
MRBrainS challenge: Online evaluation framework for brain image segmentation in 3T MRI scans
Comput. Intell. Neurosci.
Atlas of classifiers for brain MRI segmentation
Proceedings of the MICCAI
Parallel multi-dimensional LSTM, with application to fast biomedical volumetric image segmentation
Proceedings of the NIPS
Multi-dimensional gated recurrent units for the segmentation of biomedical 3D-data
Proceedings of the MICCAI
Cited by (44)
A novel cross-layer dual encoding-shared decoding network framework with spatial self-attention mechanism for hippocampus segmentation
2023, Computers in Biology and MedicineEfficient self-attention mechanism and structural distilling model for Alzheimer's disease diagnosis
2022, Computers in Biology and MedicineMR brain segmentation based on DE-ResUnet combining texture features and background knowledge
2022, Biomedical Signal Processing and ControlCitation Excerpt :Dolz et al. [29] proposed HyperDenseNet, a 3D fully convolutional neural network that extended the definition of dense connectivity to multi-modal segmentation problems. Li et al. [30] developed a multi-modal aggregation network (MMAN) for brain segmentation from multi-modal MR images. In [31], the authors developed a 3D spatially-weighted network for brain MRI tissue segmentation called SW-3D-UNet, and extended it to the multi-modal MRI environment.
Weakly supervised segmentation with cross-modality equivariant constraints
2022, Medical Image AnalysisA novel M-SegNet with global attention CNN architecture for automatic segmentation of brain MRI
2021, Computers in Biology and MedicineA robust discriminative multi-atlas label fusion method for hippocampus segmentation from MR image
2021, Computer Methods and Programs in BiomedicineCitation Excerpt :Early detection of changes in the hippocampus is essential for the early prevention and diagnosis of neural diseases. Hence, the automatic and accurate extraction of hippocampus from Magnetic Resonance (MR) images has become a pivotal task in medical image analysis [2,3]. Due to the irregular shape of the hippocampus and the blurred boundary with surrounding tissues, the automatic and accurate segmentation of the hippocampus is a challenging task.
Jingcong Li received the B.S. degree in automation from South China University of Technology, Guangzhou, China, in 2013. He is currently working toward the Ph.D. degree in pattern recognition and intelligent systems at the South China University of Technology, Guangzhou, China. His research interests include signal processing, machine learning and their applications in biomedical engineering.
Zhu Liang Yu received his BSEE in 1995 and MSEE in 1998, both in electronic engineering from the Nanjing University of Aeronautics and Astronautics, China. He received his Ph.D. in 2006 from Nanyang Technological University, Singapore. He joined Center for Signal Processing, Nanyang Technological University from 2000 as a research engineer, then as a Group Leader from 2001. In 2008, he joined the College of Automation Science and Engineering, South China University of Technology and was promoted to be a full professor in 2011. His research interests include signal processing, pattern recognition, machine learning and their applications in communications, biomedical engineering, etc.
Zhenghui Gu received the Ph.D. degree from Nanyang Technological University in 2003. From 2002 to 2008, she was with Institute for Infocomm Research, Singapore. She joined the College of Automation Science and Engineering, South China University of Technology, in 2009 as an associate professor. She was promoted to be a full professor in 2015. Her research interests include the fields of signal processing and pattern recognition.
Yuanqing Li was born in Hunan Province, China, in 1966. He received the B.S. degree in applied mathematics from Wuhan University, Wuhan, China, in 1988, the M.S. degree in applied mathematics from South China Normal University, Guangzhou, China, in 1994, and the Ph.D. degree in control theory and applications from South China University of Technology, Guangzhou, China, in 1997. Since 1997, he has been with South China University of Technology, where he became a full professor in 2004. In 2002-04, he worked at the Laboratory for Advanced Brain Signal Processing, RIKEN Brain Science Institute, Saitama, Japan, as a researcher. In 2004-08, he worked at the Laboratory for Neural Signal Processing, Institute for Inforcomm Research, Singapore, as a research scientist. His research interests include, blind signal processing, sparse representation, machine learning, brain computer interface, EEG and fMRI data analysis. He is the author or coauthor of more than 60 scientific papers in journals and conference proceedings.