Reliability-based robust multi-atlas label fusion for brain MRI segmentation

https://doi.org/10.1016/j.artmed.2019.03.004Get rights and content

Highlights

  • We define two kinds of reliabilities for each voxel in the target image, including label reliability and spatial reliability, which are estimated based on the soft label and the spatial structure from the initial segmentation, respectively.

  • We use the voxels with high reliabilities to help refine the label fusion process of those voxels with low reliabilities in the target image.

  • We validate our proposed framework on four well-known label fusion methods (i.e., LWV, PBM, JLF and SPBM) and perform experiments on three public datasets (i.e., NIREP, LONILPBA40, and ADNI), with experimental results demonstrating the outperformance of our methods over three state-of-the-art methods in multi-atlas image segmentation.

Abstract

Label fusion is one of the key steps in multi-atlas based segmentation of structural magnetic resonance (MR) images. Although a number of label fusion methods have been developed in literature, most of those existing methods fail to address two important problems, i.e., (1) compared with boundary voxels, inner voxels usually have higher probability (or reliability) to be correctly segmented, and (2) voxels with high segmentation reliability (after initial segmentation) can help refine the segmentation of voxels with low segmentation reliability in the target image. To this end, we propose a general reliability-based robust label fusion framework for multi-atlas based MR image segmentation. Specifically, in the first step, we perform initial segmentation for MR images using a conventional multi-atlas label fusion method. In the second step, for each voxel in the target image, we define two kinds of reliability, including the label reliability and spatial reliability that are estimated based on the soft label and spatial information from the initial segmentation, respectively. Finally, we employ voxels with high label-spatial reliability to help refine the label fusion process of those with low reliability in the target image. We incorporate our proposed framework into four well-known label fusion methods, including locally-weighted voting (LWV), non-local mean patch-based method (PBM), joint label fusion (JLF) and sparse patch-based method (SPBM), and obtain four novel label-spatial reliability-based label fusion approaches (called ls-LWV, ls-PBM, ls-JLF, and ls-SPBM). We validate the proposed methods in segmenting ROIs of brain MR images from the NIREP, LONI-LPBA40 and ADNI datasets. The experimental results demonstrate that our label-spatial reliability-based label fusion methods outperform the state-of-the-art methods in multi-atlas image segmentation.

Introduction

MR imaging widely used in real-world clinical applications. Accurate segmentation of brain MR images provides quantitative analysis of brain structures, thus facilitating MRI-based pathology detection and brain parcellation. For instance, many clinical applications need the segmentation of MR images to describe how brain structures change during the disease progression. As an example, the hippocampus is known to be related with the Alzheimer's disease [1], [2], and thus, it is critical to accurately segment hippocampus from the whole brain for computer-aided brain disease diagnosis. On the other hand, we usually segment a brain MR image into multiple regions-of-interest (ROIs) before constructing brain networks for subsequent analysis in brain network analysis [3], [4], [5], [6], [7]. However, it is time-consuming and usually error-prone for experts to manually segment those large amounts of MR images [8], [9], [10]. Hence, there is a largely unmet need to develop advanced automatic methods for brain ROI segmentation.

Recently, multi-atlas based segmentation methods have shown great successes in segmenting medical images [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], [26]. The assumption of multi-atlas segmentation is that a voxel in the target image should have the same label as its corresponding voxel in the atlas image, if their local tissue shapes or appearances are similar. Typically, there are two main steps for multi-atlas segmentation, i.e., (1) image registration [27], [28], [29], [30], [31], [32], and (2) label fusion. Specifically, in the image registration step, each atlas image is warped onto the target image. Then, in the label fusion step, labels from different atlases will be propagated to the target image to obtain the final labels. In this work, we focus on the label fusion step under the multi-atlas segmentation framework.

In literature, numerous label fusion strategies have been proposed for multi-atlas based brain MRI segmentation. Among them, majority voting (MV) is the simplest one, where each atlas image is treated equally when assigning labels to the target image [11]. As a more advanced strategy, locally-weighted voting (LWV) considers patch-wise similarity between the target image and each atlas as the voting weight for label assignment, and it has shown that LWV outperforms MV when segmenting brain MRI [12]. To alleviate the registration errors, the non-local mean patch-based method (PBM) has been proposed to propagate labels from not only the same location in the atlases, but also the neighboring patches in the atlases. PBM seeks multiple candidates by the pair-wise similarity between the target image patch and atlas image patches within a search region, showing improved accuracy and robustness of the labeling results [13], [14]. More recently, the sparse representation based PBM method (SPBM) is proposed for label fusion, where only a small number of image patches (with high similarity to the target image patch) will be selected for the subsequent label fusion by using the l1-norm based sparsity constraint [15], [16]. In addition, several joint label fusion methods for brain MRI segmentation are proposed to measure the joint labeling risk between two patches in atlases, thus reducing the risks of labeling error [18], [19]. Several multi-layer dictionary learning methods [24], [25] have been proposed for multi-atlas segmentation. Song et al. progressively construct dynamic multi-layer dictionary to reduce the gap between the image domain and the label domain. Zu et al. [25] use a tree-like multi-layer dictionary to represent the hierarchical patch for ROI segmentation. Instead of capturing the complex brain with intensity features, deep learning method is used for learning the representation of the original image patches, and use these learnt features for label fusion [26]. However, most of the existing methods treat each voxel in the target image equally and independently in segmentation, without considering the specific location and reliability of each voxel in the target image.

Previous studies [13], [17], [33] have shown that most of misclassified voxels locate at the boundary of ROIs, while voxels far from the boundary of ROIs are easier to be segmented correctly. In light of this, we define the reliability for each voxel to measure whether a specific voxel is easy to be correctly segmented. If a voxel has high reliability, we assume that it is easy to correctly segment this voxel; and vice versa. Consequently, those voxels with high reliability in the target image can be used to help refine the label fusion of voxels with low reliability.

With this assumption, in this paper, we present a general reliability-based robust label fusion framework for multi-atlas based MR image segmentation, including three main steps: (1) initial segmentation, (2) estimating voxel reliability, and (3) reliability-based robust label fusion. Specifically, we first perform the initial segmentation using conventional multi-atlas label fusion method (e.g., LWV and PBM, etc.), and thus can obtain a normalized voting result (i.e., a soft label with a value between 0 and 1) for each voxel after initial segmentation. In the second step, for each voxel in the target image, we define two kinds of reliability: (1) label reliability, and (2) spatial reliability. The label reliability is estimated based on the soft label, where we assume that a voxel has higher label reliability to be correctly segmented if its soft label has lower entropy. Meanwhile, the spatial reliability is estimated based on the spatial structure of the target image label map from the initial segmentation, where we assume a voxel has high reliability to be correctly segmented if the label map around this voxel is continuous. In the third step, we use voxels with the high label and spatial reliability to help refine the label fusion process of those voxels with low reliability in the target image. Our method is a general framework and can easily be combined with existing state-of-the-art methods. To validate the effectiveness of the proposed framework, we apply the proposed reliability-based strategy to four well-known label fusion approaches, i.e., LWV, PBM, JLF and SPBM, and obtain four novel reliability-based robust label fusion approaches, called label-spatial reliability-based LWV (ls-LWV), label-spatial reliability-based PBM (ls-PBM), label-spatial reliability-based JLF (ls-JLF) and label-spatial reliability-based SPBM (ls-SPBM), respectively. Experimental results on the NIREP, LONI-LPBA40 and ADNI datasets show that our method yields improved performance in ROI segmentation of brain MRI, compared with several state-of-the-art methods.

The major contributions of this work can be summarized as follows. First, we estimate the reliability (i.e., label reliability and spatial reliability) of each voxel based on results of initial segmentation, where most of existing conventional multi-atlas label fusion method can be used for initial segmentation. This helps us easily embed conventional label fusion methods into our framework. Second, voxels with high reliability are used to help refine the label fusion process of the voxels with low reliability in our framework. To the best of our knowledge, this is among the first attempt to utilize the voxel reliability as the prior knowledge for brain MRI segmentation. Third, we apply the proposed reliability-based strategy to several state-of-the-art methods on the NIREP, LONI-LPBA40 and ADNI datasets, with experimental results demonstrating the superior performance of our method over the state-of-the-art approaches in brain MRI segmentation. The code is now publicly available.1

The remainder of the paper is organized as follows. In Section 2, we describe the proposed reliability-based robust label fusion framework. We present the materials used in the experiments, experimental settings, and experimental results on the NIREP, LONI-LPBA40 and ADNI datasets in Section 3. In Section 4, we compare the proposed method with the state-of-the-art methods and investigate the influence of parameters. Finally, a conclusion of this paper and the limitations of our method as well as possible future work are presented in Section 5.

Section snippets

Method

In this section, we first introduce the notations used in this work, and then present the definitions of the proposed label reliability and spatial reliability. Finally, we elaborate our reliability-based robust label fusion framework.

Data and image pre-processing

In the experiments, we validate our proposed methods and those competing methods on the segmentation of regions-of-interest (ROIs) in brain MR images.

  • (1)

    NIREP dataset [34]: This dataset consists of 16 subjects with T1-weighted MR images, including 8 normal male adults and 8 female adults. The MR images were obtained in a General Electric Signa scanner operating at 1.5 T, using the following protocol: SPGR/50, TR 24, TE 7, NEX 1 matrix 256 × 192, FOV 24 cm. 124 contiguous coronal slices were

Discussion

In this section, we first compare our proposed framework with a corrective learning method [40]. We then analyze the statistical significance of the difference between our methods and each of competing methods, and analyze the robustness of the proposed method. We also investigate the influence of the proposed label reliability and spatial reliability, as well as the effect of the search region in the target image and the parameter λ in Eq. (11).

Conclusion

In this paper, we proposed a novel label-spatial reliability-based robust label fusion framework for multi-atlas MRI segmentation. Specifically, we first perform initial segmentation using conventional label fusion methods for the target image. Then, for each voxel in the target image, we define the label reliability and spatial reliability-based on the soft label and the spatial structure from the initial segmentation, respectively. We then estimate the label-spatial reliability for each voxel

References (41)

  • M. Kim et al.

    Improved image registration by sparse patch-based deformation estimation

    NeuroImage

    (2015)
  • C. Lian et al.

    An evidential classifier based on feature selection and two-step classification strategy

    Pattern Recognit

    (2015)
  • D.W. Shattuck et al.

    Construction of a 3D probabilistic atlas of human cortical structures

    NeuroImage

    (2008)
  • F. Shi et al.

    LABEL: pediatric brain extraction using learning-based meta-algorithm

    NeuroImage

    (2012)
  • M. Liu et al.

    Pairwise constraint-guided sparse learning for feature selection

    IEEE Trans Cybern

    (2016)
  • M. Liu et al.

    Joint binary classifier learning for ECOC-based multi-class classification

    IEEE Trans Pattern Anal Mach Intell

    (2016)
  • C. Zu et al.

    Label-aligned multi-task feature learning for multimodal classification of Alzheimer's disease and mild cognitive impairment

    Brain Imaging Behav

    (2016)
  • B. Jie et al.

    Sub-network kernels for measuring similarity of brain connectivity networks in disease diagnosis

    IEEE Trans Image Process

    (2018)
  • B. Jie et al.

    Topological graph kernel on multiple thresholded functional connectivity networks for mild cognitive impairment classification

    Hum Brain Mapp

    (2014)
  • D. Zhang et al.

    Ordinal pattern: a new descriptor for brain connectivity networks

    IEEE Trans Med Imaging

    (2018)
  • Cited by (16)

    • Information fusion and artificial intelligence for smart healthcare: a bibliometric study

      2023, Information Processing and Management
      Citation Excerpt :

      Even though many label fusion methods exist, the majority ignores the higher probability of inner voxels being correctly segmented and the ability of high-reliable voxels to refine the segmentation of low-reliable voxels. To resolve the above issues, Sun et al., 2019b developed an innovative label-spatial reliability-driven robust label fusion approach to segment multi-atlas MRIs. Specifically, original segmentation based on conventional label fusion approaches for targeted images was first performed.

    • Multi-rater label fusion based on an information bottleneck for fundus image segmentation

      2023, Biomedical Signal Processing and Control
      Citation Excerpt :

      Similarly, Guan et al. [28] studied the performance of individual raters and reconstructed the weight of each rater. To enhance the credibility of the label, Sun et al. [29] proposed a general reliability-based robust label fusion framework, which employs voxels with high label-spatial reliability to help refine the label fusion process of those with low reliability in the target image. Nowadays, Brown et al. [30] proposed using a fully Bayesian spatial regression model for label fusion that facilitates direct incorporation of covariate information while making accessible the entire posterior distribution.

    • A robust discriminative multi-atlas label fusion method for hippocampus segmentation from MR image

      2021, Computer Methods and Programs in Biomedicine
      Citation Excerpt :

      The earliest segmentation of specific anatomical structures is achieved by expert manual labeling. But there are long-term and error-prone defects, which limit the application of manual marking in big data [4–6]. Therefore, the automatic segmentation of MR images has become a hot topic in medical image analysis.

    • Multi-Atlas Based Methods in Brain MR Image Segmentation

      2019, Chinese Medical Sciences Journal
    View all citing articles on Scopus

    This work was supported in part by the National Natural Science Foundation of China (Nos: 61876082, 61861130366, 61703301, and 61473149), Taishan Scholar Program of Shandong Province in China, Scientific Research Foundation of Taishan University (No. Y-01-2018019) and China Council Scholarship.

    View full text