Elsevier

Neurocomputing

Volume 455, 30 September 2021, Pages 242-254
Neurocomputing

Unsupervised brain tumor segmentation using a symmetric-driven adversarial network

https://doi.org/10.1016/j.neucom.2021.05.073Get rights and content

Abstract

The aim of this study was to computationally model, in an unsupervised manner, a manifold of symmetry variations in normal brains, such that the learned manifold can be used to segment brain tumors from magnetic resonance (MR) images that fail to exhibit symmetry. An unsupervised brain tumor segmentation method, named as symmetric driven generative adversarial network (SD-GAN), was proposed. SD-GAN model was trained to learn a non-linear mapping between the left and right brain images, and thus being able to present the variability of the (symmetry) normal brains. The trained SD-GAN was then used to reconstruct normal brains and to segment brain tumors based on higher reconstruction errors arising from their being unsymmetrical. SD-GAN was evaluated on two public benchmark datasets (Multi-modal Brain Tumor Image Segmentation (BRATS) 2012 and 2018). SD-GAN provided best performance with tumor segmentation accuracy superior to the state-of-the-art unsupervised segmentation methods and performed comparably (<3% lower in Dice score) to the supervised U-Net (the most widely used supervised method for medical images). This study demonstrated that symmetric features presenting variations (i.e., inherent anatomical variations) can be modelled using unannotated normal MR images and thus be used in segmenting tumors.

Introduction

Brain tumor segmentation is fundamental for clinical decision support systems (CDSS) where the paradigm is that CDSSs can provide a second opinion to assist image interpretation. In clinical workflows the usual approach is to segment tumors manually. Manual segmentation is tedious, time-consuming and can be prone to intra- and inter-observer differences [1]. Many investigators have thus developed automated segmentation methods. Deep learning (DL) methods are the state-of-the-art for automated brain tumor segmentation. This is primarily attributed to the ability of DL to leverage large labelled datasets to derive feature representations with high-level semantics. There is a scarcity of annotated brain tumor training data, however, due to costs involved in labelling the multiple Magnetic Resonance (MR) imaging scans / sequences that are usually carried out. Further, various MRI scanning vendors/manufacturers employ different naming conventions for the data acquisitions and there are also differences in how scans are performed from site-to-site [2]. In addition, primary brain tumors differ in size, shape, location and degree of enhancement after intravenous contrast (see Fig. 1). Primary brain tumors comprise approximately 2% of all malignancies in adults and 20% in children. They are usually separated into low-grade and high grade gliomas and in most cases they are unilateral. Low grade gliomas (LGGs) tend to slowly infiltrate normal brain tissue whereas high-grade gliomas (HGGs) grow rapidly, destroy normal brain, enhance with contrast, have associated vasogenic edema and may be hemorrhagic/necrotic. Hence, without training data that include all these variations, DL has difficulty in generating effective feature representations for these tumors.

We suggest that modeling variations in the normal brain, with constrained anatomical variability (i.e. bilateral symmetry), can be used to segment primary brain tumors and may remove the reliance of large annotated training data. Our hypothesis is that because the normal brain is generally symmetrical, a methodology that identifies asymmetry will be able to detect primary brain tumors. In this study, we propose an unsupervised DL method that models variations in symmetry. An unsupervised approach offers the advantage that it can exploit the abundant amount of unlabeled data generated during routine clinical imaging.

Our work is related to unsupervised brain tumor segmentation methods that can be separated into 2 main groups [10]: a) Local based methods [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], where local features e.g., intensity values, are first calculated and then used for classification by an unsupervised classifier. Commonly used features and unsupervised classifiers include: thresholding [11], region growing [12], [13], K-means [14], Fuzzy C-means [15], [16], [17], [18], Markov Random Field (MRF) methods [19], [20] and methods where multiple methods are combined [21], [22]. Symmetry analysis [5], [6], [45], [46] is a local based method that segments tumors based on their extracted symmetry features. b) Image based discriminative methods [23], [24], [25], [26], [27], where a common approach is unsupervised anomaly detection (UAD). UAD assumes that normal data have constrained variability whereas abnormal data (e.g. tumors) have diverse appearances and can be differentiated from normal. UAD constrains normal variations as a manifold and uses it to detect tumors that cannot be fitted into the ‘learned’ manifold. This parallels the human process of tumor detection which can be considered as searching for anomalies compared to prior learned knowledge of what is normal. The aim of UAD is to find a lower dimensional embedding of the input data where the distance between anomalies and normal data are large [8].

The elements that directy relate to our method are anomaly detection and symmetry analysis. The UAD approach we used is based on the approach used by An and Cho [8]. The current UAD methods used in brain tumor segmentation are auto-encoder (AE) methods [[38], [39], [40], [47]]. Baur et al. [23] investigated deep spatial auto-encoding (SAE) models on 2D whole brain MR images at an image level by comparing the input to the reconstructed image. Chen and Konukoglu [24] argued that the high variability across brain MR images, i.e., different slices of 3D brain volume, can cause a situation where the dissimilarity between two normal images could be larger than the dissimilarity between an abnormal brain and its ‘normal’ version. So, they enhanced the representative ability of an auto-encoder based model by imposing a consistency in the latent space to constrain the encoder to find a latent space where the projections of the input image and the reconstructed image are close to each other. Zimmerer et al. [43] used a variational auto-encoder with the Kullback-Leibler divergence to measure reconstruction errors. The AEs are able to simulate non-linear transformations from the latent space to input data, and then to detect anomalies as a deviation from the transforms by measuring the reconstruction error. The reliance on AEs means that these methods may not have sufficient feature representation to describe the brain regions. AEs also tend to reconstruct blurred images for the hand-engineering measurement of similarity between the reconstructions and the inputs, e.g., element-wise L1 or L2 distance [27]. UAD has also been applied to detect abnormalities with other medical imaging modalities using GANs [27], [9]. Schlegl et al. proposed AnoGAN [25] and f-AnoGAN [26] to model a latent space on normal 2D optical coherence tomography (OCT) samples. Unlike AEs, GANs can produce detailed images as, rather than predefining a similarity measurement, a GAN learns an objective function by itself via the adversarial training process.

With respect to symmetry analysis, human brains are relatively symmetric. The brain is constrained by the bony skull so that any mass lesion in the brain, such as a primary brain tumor, results in asymmetry because it displaces normal brain tissue. Thus, many existing algorithms capture the asymmetry induced by brain tumors. These algorithms combine symmetry analysis with traditional supervised methods such as Support Vector Machine (SVM) [49], [50], AdaBoost [48], Fuzzy [51], decision forest [3], [7]. Anthony et al. [3] estimated the mid-sagittal plane (MSP) by locating symmetric interest points as proposed by Yu et al. [4]. Then, they calculated symmetric texture and symmetric intensity features to measure the difference between the regions reflected across the MSP. Consequently, they incorporated these hand-crafted symmetric features to improve the performance of a traditional decision forest classifier. In unsupervised brain tumor segmentation, investigators segment tumors based on an asymmetry score, or symmetry map calculated from their estimation of the distance between hemispheres. Manu et al. [5] implemented a registration plus a simple intensity extraction between flipped brains to calculate asymmetrical areas as candidate tumor regions for further unsupervised region growing segmentation. Zhao et al., [53] conducted a 3D registration [54] on two hemispheres before detecting asymmetric areas. Saha et al. [45] used the Bhattacharya coefficient, which was calculated based on intensity values, to measure the difference between symmetrical brains to detect tumors. Erihov et al. [46] detected tumor regions as a salient object which were distinctive locally and across its symmetrical region. Hassan et al. [6] estimated the MSP to locate symmetrical areas based on intensity features and then used a deformation model and spatial relations between the tumor and other tissues to segment the brain tumor.

In summary, all the works referred to above first estimated the MSP or registered the brains to mitigate asymmetry introduced by motion. Asymmetrical candidate regions were detected based on hand-crafted symmetry features (e.g., intensity). Then, classical supervised or unsupervised classifiers were further applied to refine the segmentation. However, symmetry property in normal human brain is a high-level semantic concept measuring the visual similarity between two hesmihperes. Therefore, the reliance on using hand-crafted features means that these methods have limited representative capacity of capturing symmetry.

We propose a symmetry driven GAN (SD-GAN) for the unsupervised segmentation of brain tumors. We model symmetry variations as normal brain patterns and then use them to differentiate tumors. We use a conditional GAN (cGAN) [42] to model the transformation between the normal left and right hemispheres, where the normal symmetry variations are embedded in our learning model. Our method, compared to existing methods, contributes the following: a) It tolerates large normal variations e.g., alignment and movement; consequently, it eliminates reliance on annotated training data. b) It provides detailed and realistic brain volumes that resemble the input images; the reconstructed images are conditioned by a latent space of symmetry variations that is learned through the symmetric transformation training. The capacity for generating detailed and realistic brain volumes enables segment brain tumors to show asymmetry. c) We leverage a cGAN to iteratively learn the normal brain appearance in an end-to-end manner by incorporating the symmetry analysis into our voxel-wise classifier. In addition, different from the existing symmetry-based methods [5], [6], [7], [53] which rely on additional registration steps or estimating the MSP, our SD-GAN learns the symmetry only according to the middle vertical line. Thus, our method removes the reliance on applying additional registration steps and estimating the MSP.

Section snippets

Materials

We used the public Brain Tumor Segmentation Challenge Datasets (BraTSs) from 2012 and 2018 that included LLGs and HGGs. The BraTS 2012 dataset had 20 HGGs, 10 LGGs and 50 synthesized brain images that were HGGs or LGGs [37]. The 2018 dataset [35], [36] had 285 patients with LGGs and HGGs from 19 institutions where different protocols were used to acquire the MR data. Each sample, however, had T1 (T1), T1W + contrast (T1c), T2-weighted (T2W) and FLAIR images. These two datasets were used because

Results

In Table 2 we present the evaluation metrics of our method when compared to the state-of-the-art UAD methods and the 3D U-Net. Our approach had the best in Dice at 61.9%, and Sensitivity at 61.3%. Our method was competitive when compared to the supervised 3D U-Net with a Dice of 61.9% versus 64.9%). Our result of AAE using BraTS dataset was 1.5% lower in Dice Score than those reported in the referenced paper [44] (our AAE: 39.5%, AAE [44]: 41%). Our result of VAE using BraTS dataset is 0.9%

Discussion

Our main findings are that our SD-GAN: a) outperformed comparative deep UAD brain tumor segmentation methods; b) perform competitively to supervised DL method and symmetry analysis methods; c) tolerated most normal aymmetry variations thus improving segmentation.

Model variants

We conducted additional experiments on three smaller model variants. The detailed model configurations and the segmentation results are provided in Table 5. The “small” and “medium” models are variants in which fewer channels were adopted in the generator networks (with using the same discriminator network of a size of 2.9 M). The other medium network (“medium-2”) is a variant when we only use a 4-layer U-Net in the generator network. The medium sized network obtained a slightly decrease in

Conclusions

We present our findings using an unsupervised symmetric-driven adversarial network method for segmentation of brain tumors. The novelty was in the modelling of a manifold of symmetry variations on normal brains that can be used to detect tumors based on their fitness to the modelled manifold. Our results on public brain tumor datasets show that our method achieved the best unsupervised segmentation performance and performed competitively to the supervised methods.

CRediT authorship contribution statement

Xinheng Wu: Conceptualization, Methodology, Data curation, Software, Visualization, Formal analysis, Validation, Investigation, Writing - original draft. Lei Bi: Conceptualization, Investigation, Writing - review & editing, Project administration. Michael Fulham: Visualization, Formal analysis, Writing - review & editing. David Dagan Feng: Supervision, Funding acquisition. Luping Zhou: Writing - review & editing. Jinman Kim: Supervision, Writing - review & editing, Project administration,

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Xinheng Wu received her Master of Philosophy (research) from the University of Sydney in 2020. She obtained her master of statistics degree, B.S. degree from the University of Hong Kong and the China University of Mining & Technology, Beijing, respectively. Her research interests include deep learning, computer vision and medical image analysis.

References (57)

  • B. Anthony et al.

    Brain tumor segmentation with symmetric texture and symmetric intensity-based decision forests

    IEEE International Symposium on Biomedical Imaging

    (2013)
  • S. Yu et al.

    Reflection symmetry-integrated image segmentation

    IEEE transactions on pattern analysis and machine intelligence

    (2011)
  • G. Manu et al.

    Brain tumor segmentation by integrating symmetric property with region growing approach

    Annual IEEE India Conference

    (2015)
  • K. Hassan, O. Colliot, and I. Bloch, “Automatic brain tumor segmentation using symmetry analysis and deformable...
  • G. Ezequiel, B. H. Menze, and N. Ayache, “Spatial decision forests for glioma segmentation in multi-channel MR images,”...
  • J. An, and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” Special Lecture...
  • P. Isola et al.

    Image-to-image translation with conditional adversarial networks

  • M. Angulakshmi et al.

    Automated brain tumour segmentation techniques—A review

    International Journal of Imaging Systems and Technology

    (2017)
  • N.M. Saad et al.

    Segmentation of brain lesions in diffusion-weighted MRI using thresholding technique

    IEEE International Conference on Signal and Image Processing Applications

    (2011)
  • K.S.A. Viji et al.

    Modified texture-based region growing segmentation of MR brain images

    IEEE Conference on Information & Communication Technologies

    (2013)
  • K.M. Nimeesha et al.

    Brain tumour segmentation using K-means and fuzzy c-means clustering algorithm

    Int J Comput Sci Inf Technol Res Excell

    (2013)
  • C. Militello et al.

    Gamma Knife treatment planning: MR brain tumor segmentation and volume measurement based on unsupervised Fuzzy C-Means clustering

    International Journal of Imaging Systems and Technology.

    (2015 Sep)
  • N. Subbanna et al.

    Iterative multilevel MRF leveraging context and voxel information for brain tumour segmentation in MRI

  • N. Subbanna et al.

    Probabilistic gabor and markov random fields segmentation of brain tumours in mri volumes

    Proc MICCAI Brain Tumor Segmentation Challenge

    (2012)
  • A. Moran et al.
  • C. Baur et al.

    Deep autoencoding models for unsupervised anomaly segmentation in brain mr images

    International MICCAI Brainlesion Workshop

    (2018)
  • X. Chen et al.

    Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders

    In MIDL Conference book

    (2018)
  • T. Schlegl et al.

    Unsupervised anomaly detection with generative adversarial networks to guide marker discovery

    International Conference on Information Processing in Medical Imaging

    (2017)
  • Cited by (38)

    • SAR-CGAN: Improved generative adversarial network for EIT reconstruction of lung diseases

      2023, Biomedical Signal Processing and Control
      Citation Excerpt :

      They have also been used to address the low-spatial problem of EIT imaging [14–17]. As a classical deep learning technology, conditional generative adversarial network (CGAN) is used to reconstruct the medical image [18–21]. It adds extra information to render image generation process controlled, and improves the detailed features of the generated images effectively through adversarial learning between the generator and discriminator [22].

    • Hybrid transformer UNet for thyroid segmentation from ultrasound scans

      2023, Computers in Biology and Medicine
      Citation Excerpt :

      Moreover, the gland appearances vary significantly from one patient to another, making the challenge more arduous. Recently, deep learning based methods, including auto encoder decoder neural networks (AED) [4], fully convolutional neural networks (FCN) [5], generative adversarial networks (GAN) [6], UNet [7], and its variants [8–12] have greatly facilitated the medical image segmentation. In the field of thyroid segmentation, many deep learning frameworks [13–18] have been proposed and outperformed conventional methods [19–26] remarkably.

    View all citing articles on Scopus

    Xinheng Wu received her Master of Philosophy (research) from the University of Sydney in 2020. She obtained her master of statistics degree, B.S. degree from the University of Hong Kong and the China University of Mining & Technology, Beijing, respectively. Her research interests include deep learning, computer vision and medical image analysis.

    Lei Bi received his Master of Information Technology, Master of Philosophy (research) and PhD from the University of Sydney, in 2011, 2013 and 2018, respectively. Currently, he is a research fellow with the Australia Research Council Training Centre in Innovative BioEngineering, the University of Sydney. His research interests include in using deep learning technologies for computer-aided diagnosis and medical image analysis.

    Dagan Feng received the M. E. degree in Electrical Engineering & Computer Science (EECS) from Shanghai Jiao Tong University, Shanghai, China, in 1982, and the M. S. in Biocybernetics and Ph. D. in Computer Science from the University of California, Los Angeles, CA, USA, in 1985 and 1988 respectively, where he received the Crump Prize for Excellence in Medical Engineering. He is currently a Professor in the School of Information Technologies, Director of Biomedical & Multimedia Information Technology (BMIT) Research Group and Director of the Institute of Biomedical Engineering and Technology at the University of Sydney. He has published over 800 scholarly research papers, pioneered several new research directions, and made a number of landmark contributions in his field. Prof. Feng's research in the areas of biomedical and multimedia information technology seeks to address the major challenges in “big data science” and provide innovative solutions for stochastic data acquisition, compression, storage, management, modelling, fusion, visualization and communication. Prof. Feng is Fellow of ACS, HKIE, IET, IEEE and Australian Academy of Technological Sciences and Engineering.

    Michael Fulham is currently the Director of the Department of Molecular Imaging and Senior Neurologist at the Royal Prince Alfred Hospital, Clinical Director of Medical Imaging for Sydney South West Area Health Service, Sydney; and Adjunct Professor in the School of Computer Science, the University of Sydney, Australia. His research interests include neuroimaging, PET-CT imaging, and the application of information technologies for patient management.

    Luping Zhou is a Senior Lecturer in School of Electrical and Information Engineering, University of Sydney, Australia. She obtained her PhD, MSc, and BEng from Australian National University, National University of Singapore and Southeast University, China, respectively. Her research interests include medical image analysis, machine learning, computer vision.

    Jinman Kim received the B.S. (honours) degree in computer science and PhD degree from the University of Sydney, Australia, in 2001 and 2006, respectively. Since his PhD, he has been a Research Associate at the leading teaching hospital, the Royal Prince Alfred. In 2008 until 2012, he was an ARC postdoc research fellow, one year leave (2009-2010) to join MIRALab research group, Geneva, Switzerland, as a Marie Curie senior research fellow. Since 2013, he has been with the School of Computer Science, The University of Sydney, where he was a Senior Lecturer, and became an Associate Professor in 2016. His research interests include medical image analysis and visualization, computer aided diagnosis, and Telehealth technologies.

    View full text