Unsupervised brain tumor segmentation using a symmetric-driven adversarial network
Introduction
Brain tumor segmentation is fundamental for clinical decision support systems (CDSS) where the paradigm is that CDSSs can provide a second opinion to assist image interpretation. In clinical workflows the usual approach is to segment tumors manually. Manual segmentation is tedious, time-consuming and can be prone to intra- and inter-observer differences [1]. Many investigators have thus developed automated segmentation methods. Deep learning (DL) methods are the state-of-the-art for automated brain tumor segmentation. This is primarily attributed to the ability of DL to leverage large labelled datasets to derive feature representations with high-level semantics. There is a scarcity of annotated brain tumor training data, however, due to costs involved in labelling the multiple Magnetic Resonance (MR) imaging scans / sequences that are usually carried out. Further, various MRI scanning vendors/manufacturers employ different naming conventions for the data acquisitions and there are also differences in how scans are performed from site-to-site [2]. In addition, primary brain tumors differ in size, shape, location and degree of enhancement after intravenous contrast (see Fig. 1). Primary brain tumors comprise approximately 2% of all malignancies in adults and 20% in children. They are usually separated into low-grade and high grade gliomas and in most cases they are unilateral. Low grade gliomas (LGGs) tend to slowly infiltrate normal brain tissue whereas high-grade gliomas (HGGs) grow rapidly, destroy normal brain, enhance with contrast, have associated vasogenic edema and may be hemorrhagic/necrotic. Hence, without training data that include all these variations, DL has difficulty in generating effective feature representations for these tumors.
We suggest that modeling variations in the normal brain, with constrained anatomical variability (i.e. bilateral symmetry), can be used to segment primary brain tumors and may remove the reliance of large annotated training data. Our hypothesis is that because the normal brain is generally symmetrical, a methodology that identifies asymmetry will be able to detect primary brain tumors. In this study, we propose an unsupervised DL method that models variations in symmetry. An unsupervised approach offers the advantage that it can exploit the abundant amount of unlabeled data generated during routine clinical imaging.
Our work is related to unsupervised brain tumor segmentation methods that can be separated into 2 main groups [10]: a) Local based methods [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], where local features e.g., intensity values, are first calculated and then used for classification by an unsupervised classifier. Commonly used features and unsupervised classifiers include: thresholding [11], region growing [12], [13], K-means [14], Fuzzy C-means [15], [16], [17], [18], Markov Random Field (MRF) methods [19], [20] and methods where multiple methods are combined [21], [22]. Symmetry analysis [5], [6], [45], [46] is a local based method that segments tumors based on their extracted symmetry features. b) Image based discriminative methods [23], [24], [25], [26], [27], where a common approach is unsupervised anomaly detection (UAD). UAD assumes that normal data have constrained variability whereas abnormal data (e.g. tumors) have diverse appearances and can be differentiated from normal. UAD constrains normal variations as a manifold and uses it to detect tumors that cannot be fitted into the ‘learned’ manifold. This parallels the human process of tumor detection which can be considered as searching for anomalies compared to prior learned knowledge of what is normal. The aim of UAD is to find a lower dimensional embedding of the input data where the distance between anomalies and normal data are large [8].
The elements that directy relate to our method are anomaly detection and symmetry analysis. The UAD approach we used is based on the approach used by An and Cho [8]. The current UAD methods used in brain tumor segmentation are auto-encoder (AE) methods [[38], [39], [40], [47]]. Baur et al. [23] investigated deep spatial auto-encoding (SAE) models on 2D whole brain MR images at an image level by comparing the input to the reconstructed image. Chen and Konukoglu [24] argued that the high variability across brain MR images, i.e., different slices of 3D brain volume, can cause a situation where the dissimilarity between two normal images could be larger than the dissimilarity between an abnormal brain and its ‘normal’ version. So, they enhanced the representative ability of an auto-encoder based model by imposing a consistency in the latent space to constrain the encoder to find a latent space where the projections of the input image and the reconstructed image are close to each other. Zimmerer et al. [43] used a variational auto-encoder with the Kullback-Leibler divergence to measure reconstruction errors. The AEs are able to simulate non-linear transformations from the latent space to input data, and then to detect anomalies as a deviation from the transforms by measuring the reconstruction error. The reliance on AEs means that these methods may not have sufficient feature representation to describe the brain regions. AEs also tend to reconstruct blurred images for the hand-engineering measurement of similarity between the reconstructions and the inputs, e.g., element-wise L1 or L2 distance [27]. UAD has also been applied to detect abnormalities with other medical imaging modalities using GANs [27], [9]. Schlegl et al. proposed AnoGAN [25] and f-AnoGAN [26] to model a latent space on normal 2D optical coherence tomography (OCT) samples. Unlike AEs, GANs can produce detailed images as, rather than predefining a similarity measurement, a GAN learns an objective function by itself via the adversarial training process.
With respect to symmetry analysis, human brains are relatively symmetric. The brain is constrained by the bony skull so that any mass lesion in the brain, such as a primary brain tumor, results in asymmetry because it displaces normal brain tissue. Thus, many existing algorithms capture the asymmetry induced by brain tumors. These algorithms combine symmetry analysis with traditional supervised methods such as Support Vector Machine (SVM) [49], [50], AdaBoost [48], Fuzzy [51], decision forest [3], [7]. Anthony et al. [3] estimated the mid-sagittal plane (MSP) by locating symmetric interest points as proposed by Yu et al. [4]. Then, they calculated symmetric texture and symmetric intensity features to measure the difference between the regions reflected across the MSP. Consequently, they incorporated these hand-crafted symmetric features to improve the performance of a traditional decision forest classifier. In unsupervised brain tumor segmentation, investigators segment tumors based on an asymmetry score, or symmetry map calculated from their estimation of the distance between hemispheres. Manu et al. [5] implemented a registration plus a simple intensity extraction between flipped brains to calculate asymmetrical areas as candidate tumor regions for further unsupervised region growing segmentation. Zhao et al., [53] conducted a 3D registration [54] on two hemispheres before detecting asymmetric areas. Saha et al. [45] used the Bhattacharya coefficient, which was calculated based on intensity values, to measure the difference between symmetrical brains to detect tumors. Erihov et al. [46] detected tumor regions as a salient object which were distinctive locally and across its symmetrical region. Hassan et al. [6] estimated the MSP to locate symmetrical areas based on intensity features and then used a deformation model and spatial relations between the tumor and other tissues to segment the brain tumor.
In summary, all the works referred to above first estimated the MSP or registered the brains to mitigate asymmetry introduced by motion. Asymmetrical candidate regions were detected based on hand-crafted symmetry features (e.g., intensity). Then, classical supervised or unsupervised classifiers were further applied to refine the segmentation. However, symmetry property in normal human brain is a high-level semantic concept measuring the visual similarity between two hesmihperes. Therefore, the reliance on using hand-crafted features means that these methods have limited representative capacity of capturing symmetry.
We propose a symmetry driven GAN (SD-GAN) for the unsupervised segmentation of brain tumors. We model symmetry variations as normal brain patterns and then use them to differentiate tumors. We use a conditional GAN (cGAN) [42] to model the transformation between the normal left and right hemispheres, where the normal symmetry variations are embedded in our learning model. Our method, compared to existing methods, contributes the following: a) It tolerates large normal variations e.g., alignment and movement; consequently, it eliminates reliance on annotated training data. b) It provides detailed and realistic brain volumes that resemble the input images; the reconstructed images are conditioned by a latent space of symmetry variations that is learned through the symmetric transformation training. The capacity for generating detailed and realistic brain volumes enables segment brain tumors to show asymmetry. c) We leverage a cGAN to iteratively learn the normal brain appearance in an end-to-end manner by incorporating the symmetry analysis into our voxel-wise classifier. In addition, different from the existing symmetry-based methods [5], [6], [7], [53] which rely on additional registration steps or estimating the MSP, our SD-GAN learns the symmetry only according to the middle vertical line. Thus, our method removes the reliance on applying additional registration steps and estimating the MSP.
Section snippets
Materials
We used the public Brain Tumor Segmentation Challenge Datasets (BraTSs) from 2012 and 2018 that included LLGs and HGGs. The BraTS 2012 dataset had 20 HGGs, 10 LGGs and 50 synthesized brain images that were HGGs or LGGs [37]. The 2018 dataset [35], [36] had 285 patients with LGGs and HGGs from 19 institutions where different protocols were used to acquire the MR data. Each sample, however, had T1 (T1), T1W + contrast (T1c), T2-weighted (T2W) and FLAIR images. These two datasets were used because
Results
In Table 2 we present the evaluation metrics of our method when compared to the state-of-the-art UAD methods and the 3D U-Net. Our approach had the best in Dice at 61.9%, and Sensitivity at 61.3%. Our method was competitive when compared to the supervised 3D U-Net with a Dice of 61.9% versus 64.9%). Our result of AAE using BraTS dataset was 1.5% lower in Dice Score than those reported in the referenced paper [44] (our AAE: 39.5%, AAE [44]: 41%). Our result of VAE using BraTS dataset is 0.9%
Discussion
Our main findings are that our SD-GAN: a) outperformed comparative deep UAD brain tumor segmentation methods; b) perform competitively to supervised DL method and symmetry analysis methods; c) tolerated most normal aymmetry variations thus improving segmentation.
Model variants
We conducted additional experiments on three smaller model variants. The detailed model configurations and the segmentation results are provided in Table 5. The “small” and “medium” models are variants in which fewer channels were adopted in the generator networks (with using the same discriminator network of a size of 2.9 M). The other medium network (“medium-2”) is a variant when we only use a 4-layer U-Net in the generator network. The medium sized network obtained a slightly decrease in
Conclusions
We present our findings using an unsupervised symmetric-driven adversarial network method for segmentation of brain tumors. The novelty was in the modelling of a manifold of symmetry variations on normal brains that can be used to detect tumors based on their fitness to the modelled manifold. Our results on public brain tumor datasets show that our method achieved the best unsupervised segmentation performance and performed competitively to the supervised methods.
CRediT authorship contribution statement
Xinheng Wu: Conceptualization, Methodology, Data curation, Software, Visualization, Formal analysis, Validation, Investigation, Writing - original draft. Lei Bi: Conceptualization, Investigation, Writing - review & editing, Project administration. Michael Fulham: Visualization, Formal analysis, Writing - review & editing. David Dagan Feng: Supervision, Funding acquisition. Luping Zhou: Writing - review & editing. Jinman Kim: Supervision, Writing - review & editing, Project administration,
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Xinheng Wu received her Master of Philosophy (research) from the University of Sydney in 2020. She obtained her master of statistics degree, B.S. degree from the University of Hong Kong and the China University of Mining & Technology, Beijing, respectively. Her research interests include deep learning, computer vision and medical image analysis.
References (57)
- et al.
Multispectral MR images segmentation based on fuzzy knowledge and modified seeded region growing
Magnetic resonance imaging
(2012) - et al.
Adaptive scale fuzzy local Gaussian mixture model for brain MR image segmentation
Neurocomputing
(2014) - et al.
Fuzzy anisotropic diffusion-based segmentation and texture-based ensemble classification of brain tumor
Applied Soft Computing
(2014) - et al.
An improved intuitionistic fuzzy c-means clustering algorithm incorporating local information for brain image segmentation
Applied Soft Computing
(2016) - et al.
Combining stationary wavelet transform and self-organizing maps for brain MR image segmentation
Engineering Applications of Artificial Intelligence
(2011) - et al.
f-AnoGAN: Fast unsupervised anomaly detection with generative adversarial networks
Medical image analysis
(2019) - et al.
Quick detection of brain tumors and edemas: A bounding box method using symmetry
Computerized medical imaging and graphics
(2012) - et al.
3D brain tumor segmentation in MRI using fuzzy classification, symmetry analysis and spatially constrained deformable models
Fuzzy sets and systems
(2009) - et al.
- et al.
State of the art survey on MRI brain tumor segmentation
Magnetic resonance imaging
(2013)
Brain tumor segmentation with symmetric texture and symmetric intensity-based decision forests
IEEE International Symposium on Biomedical Imaging
Reflection symmetry-integrated image segmentation
IEEE transactions on pattern analysis and machine intelligence
Brain tumor segmentation by integrating symmetric property with region growing approach
Annual IEEE India Conference
Image-to-image translation with conditional adversarial networks
Automated brain tumour segmentation techniques—A review
International Journal of Imaging Systems and Technology
Segmentation of brain lesions in diffusion-weighted MRI using thresholding technique
IEEE International Conference on Signal and Image Processing Applications
Modified texture-based region growing segmentation of MR brain images
IEEE Conference on Information & Communication Technologies
Brain tumour segmentation using K-means and fuzzy c-means clustering algorithm
Int J Comput Sci Inf Technol Res Excell
Gamma Knife treatment planning: MR brain tumor segmentation and volume measurement based on unsupervised Fuzzy C-Means clustering
International Journal of Imaging Systems and Technology.
Iterative multilevel MRF leveraging context and voxel information for brain tumour segmentation in MRI
Probabilistic gabor and markov random fields segmentation of brain tumours in mri volumes
Proc MICCAI Brain Tumor Segmentation Challenge
Deep autoencoding models for unsupervised anomaly segmentation in brain mr images
International MICCAI Brainlesion Workshop
Unsupervised Detection of Lesions in Brain MRI using constrained adversarial auto-encoders
In MIDL Conference book
Unsupervised anomaly detection with generative adversarial networks to guide marker discovery
International Conference on Information Processing in Medical Imaging
Cited by (38)
Brain tumor detection based on a novel and high-quality prediction of the tumor pixel distributions
2024, Computers in Biology and MedicineDeep learning automates bidimensional and volumetric tumor burden measurement from MRI in pre- and post-operative glioblastoma patients
2023, Computers in Biology and MedicineSAR-CGAN: Improved generative adversarial network for EIT reconstruction of lung diseases
2023, Biomedical Signal Processing and ControlCitation Excerpt :They have also been used to address the low-spatial problem of EIT imaging [14–17]. As a classical deep learning technology, conditional generative adversarial network (CGAN) is used to reconstruct the medical image [18–21]. It adds extra information to render image generation process controlled, and improves the detailed features of the generated images effectively through adversarial learning between the generator and discriminator [22].
Hybrid transformer UNet for thyroid segmentation from ultrasound scans
2023, Computers in Biology and MedicineCitation Excerpt :Moreover, the gland appearances vary significantly from one patient to another, making the challenge more arduous. Recently, deep learning based methods, including auto encoder decoder neural networks (AED) [4], fully convolutional neural networks (FCN) [5], generative adversarial networks (GAN) [6], UNet [7], and its variants [8–12] have greatly facilitated the medical image segmentation. In the field of thyroid segmentation, many deep learning frameworks [13–18] have been proposed and outperformed conventional methods [19–26] remarkably.
Xinheng Wu received her Master of Philosophy (research) from the University of Sydney in 2020. She obtained her master of statistics degree, B.S. degree from the University of Hong Kong and the China University of Mining & Technology, Beijing, respectively. Her research interests include deep learning, computer vision and medical image analysis.
Lei Bi received his Master of Information Technology, Master of Philosophy (research) and PhD from the University of Sydney, in 2011, 2013 and 2018, respectively. Currently, he is a research fellow with the Australia Research Council Training Centre in Innovative BioEngineering, the University of Sydney. His research interests include in using deep learning technologies for computer-aided diagnosis and medical image analysis.
Dagan Feng received the M. E. degree in Electrical Engineering & Computer Science (EECS) from Shanghai Jiao Tong University, Shanghai, China, in 1982, and the M. S. in Biocybernetics and Ph. D. in Computer Science from the University of California, Los Angeles, CA, USA, in 1985 and 1988 respectively, where he received the Crump Prize for Excellence in Medical Engineering. He is currently a Professor in the School of Information Technologies, Director of Biomedical & Multimedia Information Technology (BMIT) Research Group and Director of the Institute of Biomedical Engineering and Technology at the University of Sydney. He has published over 800 scholarly research papers, pioneered several new research directions, and made a number of landmark contributions in his field. Prof. Feng's research in the areas of biomedical and multimedia information technology seeks to address the major challenges in “big data science” and provide innovative solutions for stochastic data acquisition, compression, storage, management, modelling, fusion, visualization and communication. Prof. Feng is Fellow of ACS, HKIE, IET, IEEE and Australian Academy of Technological Sciences and Engineering.
Michael Fulham is currently the Director of the Department of Molecular Imaging and Senior Neurologist at the Royal Prince Alfred Hospital, Clinical Director of Medical Imaging for Sydney South West Area Health Service, Sydney; and Adjunct Professor in the School of Computer Science, the University of Sydney, Australia. His research interests include neuroimaging, PET-CT imaging, and the application of information technologies for patient management.
Luping Zhou is a Senior Lecturer in School of Electrical and Information Engineering, University of Sydney, Australia. She obtained her PhD, MSc, and BEng from Australian National University, National University of Singapore and Southeast University, China, respectively. Her research interests include medical image analysis, machine learning, computer vision.
Jinman Kim received the B.S. (honours) degree in computer science and PhD degree from the University of Sydney, Australia, in 2001 and 2006, respectively. Since his PhD, he has been a Research Associate at the leading teaching hospital, the Royal Prince Alfred. In 2008 until 2012, he was an ARC postdoc research fellow, one year leave (2009-2010) to join MIRALab research group, Geneva, Switzerland, as a Marie Curie senior research fellow. Since 2013, he has been with the School of Computer Science, The University of Sydney, where he was a Senior Lecturer, and became an Associate Professor in 2016. His research interests include medical image analysis and visualization, computer aided diagnosis, and Telehealth technologies.