Elsevier

Pattern Recognition

Volume 42, Issue 6, June 2009, Pages 1067-1079
Pattern Recognition

Automatic joint classification and segmentation of whole cell 3D images

https://doi.org/10.1016/j.patcog.2008.08.009Get rights and content

Abstract

We present a machine learning tool for automatic texton-based joint classification and segmentation of mitochondria in MNT-1 cells imaged using ion-abrasion scanning electron microscopy (IA-SEM). For diagnosing signatures that may be unique to cellular states such as cancer, automatic tools with minimal user intervention need to be developed for analysis and mining of high-throughput data from these large volume data sets (typically 2GB/cell). Challenges for such a tool in 3D electron microscopy arise due to low contrast and signal-to-noise ratios (SNR) inherent to biological imaging. Our approach is based on block-wise classification of images into a trained list of regions. Given manually labeled images, our goal is to learn models that can localize novel instances of the regions in test datasets. Since datasets obtained using electron microscopes are intrinsically noisy, we improve the SNR of the data for automatic segmentation by implementing a 2D texture-preserving filter on each slice of the 3D dataset. We investigate texton-based region features in this work. Classification is performed by k-nearest neighbor (k-NN) classifier, support vector machines (SVMs), adaptive boosting (AdaBoost) and histogram matching using a NN classifier. In addition, we study the computational complexity vs. segmentation accuracy tradeoff of these classifiers. Segmentation results demonstrate that our approach using minimal training data performs close to semi-automatic methods using the variational level-set method and manual segmentation carried out by an experienced user. Using our method, which we show to have minimal user intervention and high classification accuracy, we investigate quantitative parameters such as volume of the cytoplasm occupied by mitochondria, differences between the surface area of inner and outer membranes and mean mitochondrial width which are quantities potentially relevant to distinguishing cancer cells from normal cells. To test the accuracy of our approach, these quantities are compared against manually computed counterparts. We also demonstrate extension of these methods to segment 3D images obtained using electron tomography.

Introduction

In this paper, we present a machine learning tool for automatic texton-based joint classification and segmentation of mitochondria in MNT-1 cells imaged using ion-abrasion scanning electron microscopy (IA-SEM) [1], [2], [3]. IA-SEM is a novel technique that can be used to image whole mammalian cells at nanometer resolution. Our tool aims at obtaining a quantitative description of features in the image such as mitochondria.

Object recognition can be defined as the localization of object instances within an image. As mentioned in Ref. [4], object localization is to obtain class models that are invariant enough to incorporate naturally occurring intra-class variations and yet discriminative enough to distinguish between different classes. Both object and region localization usually are realized in practice using texton-based models that are developed in the context of texture recognition [5], [6]. “Texton” and “visual word” have been used interchangeably in computer vision literature, which refers to clusters of filter responses in a high-dimensional space.

Particularly, textures are modeled as joint distribution of filter responses, where the distribution is represented as a frequency histogram of filter response cluster centers (textons) and the texture models are learnt from training images [4], [5], [7], [8]. Many texton-based and object category classification approaches have been applied to image databases, for e.g. CUReT, UIUCTex, CalTech101, CalTech6 and PASCAL challenge (Ref. [8] and references therein), scene understanding [9], semantic image retrieval, web search and interactive image editing [4]. The classification algorithm is affected by parameters such as choice of filter bank and rotational invariance [4], [5], [8], the number of training images (Ref. [5] for a survey) and the parameters of the classifiers (for e.g., choice of k in k-NN and kernel width in SVMs, etc.). Texture recognition has been investigated under various factors such as lighting, viewing conditions [5], [6] brightness and color [10]. In Ref. [11], the authors separate texture from untextured regions, where contours are treated in the intervening contour framework and textures are analyzed using textons. They propose a graph theoretic framework of normalized cuts to find partitions of the image. Texture and object category classifications are studied using different key point detectors and descriptors, as well as different kernels and classifiers by Schmid et al. [8]. Classification using support vector machines (SVMs) have also been extensively applied to object categorization [8], texture [12] and medical applications [13], [14]. A segmentation approach for retinal vessels based on textons has been investigated in Ref. [15]. Recently, multi-class boosting techniques have been applied to region-based classification of aerial imagery [9].

Melanin is produced in small organelles known as melanosomes that are transferred by an yet unknown mechanism from melanocytes to the surrounding keratinocytes [16]. Melanogenesis is the process through which the pigmented melanin is synthesized in melanocytes [17]. Numerous diseases leading to abnormal pigmentation in which patients have compromised immune system have been documented [16]. The knowledge of intracellular distribution of melanosomes at different stages of biogenesis remains an unanswered question and is being addressed using IA-SEM [1], [2]. Mitochondria are also prominent features in cellular interior, and their size, shape and spatial distribution may have some relevance to the overall state of the cell. Our goal is to automatically segment out mitochondria in MNT-1 cells and extract quantities such as the internal volume occupied by mitochondria, the difference between surface areas of inner and outer membranes, and the mean mitochondrial width that can be indicative of the state of the cell.

Texton-based segmentation is challenging in this type of work due to the large size of the data, low contrast and SNR, appearance geometry and viewpoint variation. To improve the SNR, we have implemented a preprocessing step using variational filtering. This method uses an adaptive fidelity term that applies different levels of denoising in different regions [18]. This denoising method preserves the textures and smooth regions are better denoised in terms of signal-to-noise ratio (SNR) compared to nonlinear diffusion [19] and total variation methods [20] that preserve sharpness and location of the edges, and in some cases enhance them, but many small details and textured regions of the image are ignored.

For diagnosing signatures that may be unique to cellular states such as cancer, large volume data sets (typically 2GB/cell) need to be analyzed. Various semi-automatic segmentation methods have been developed in the past for biological applications (see chapter 30 of Ref. [21]) that require parameter setting or some sort of a priori information and hence would be a major bottleneck for high-throughput data mining and analysis. Our goal here is to develop a machine learning tool that has minimal user intervention for high-throughput imaging. In our approach, user intervention is limited to choosing the parameters just for the training set (including the parameters for texture preserving filter) which is then retained for subsequent test datasets. We have borrowed texton-based models developed in the context of texture [5] and object [4] classification. Given manually labeled images that contain both segmentation maps and rough annotations of regions, our supervised learning algorithm learns the model such that it can localize novel instances of the regions on test datasets. The segmentation obtained by the proposed methods is compared against the well-studied level-set approach in image segmentation community and manual segmentation by an experienced user.

Binary classification of mitochondria is executed by exploiting region-based texture features. Block-wise classification is performed by k-NN classifier, SVMs, AdaBoost and histogram matching using a NN classifier. Segmentation results demonstrate that the proposed approach using minimal training data perform close to semi-automatic method carried out using variational level-set method and manual segmentation by an experienced user. Our aim is to identify quantitative parameters that describe the mitochondrial structures which can be realized using simple and computationally efficient techniques. Therefore, we study computational complexity in terms of memory and timing requirements versus segmentation accuracy for all the classifiers investigated in this work. Experimental results show that the proposed learning-based algorithm is computationally efficient on large datasets, has minimal user intervention and achieves high classification accuracy. We then make quantitative estimates of measures such as mitochondrial volume and differential surface area between outer and inner membranes with enough numbers to provide statistically meaningful results. To test the accuracy of our approach the quantities are compared against manually computed counterparts. In the end, we test the robustness of automatic texton-based segmentation procedure using a very different type of 3D image dataset obtained using cryo-electron tomography of Doxil, a liposomal formulation of doxorubicin. On tomograms of liposomal doxorubicin formulations (Doxil), an anticancer drug, imaged at cryogenic temperatures, we study the radii and volume distribution which is of great interest in cancer treatment.

The rest of the paper is organized as follows. Data collection techniques are briefly summarized in Section 2. Details of preprocessing, semi-automatic segmentation based on level-sets and automatic segmentation using machine learning approaches are presented in Section 3. Experimental results and quantitative analysis are presented in Section 4. Conclusion and future work are drawn in Section 5.

Section snippets

Data collection techniques

Conventional optical imaging is limited to a resolution of 0.2μm microns in the very best cases. Heymann et al. [1], [2] have recently reported a new strategy for 3D imaging of whole cells at resolutions about an order of magnitude higher than what can be achieved with light microscopy (see Ref. [1] for details). Use of this method, termed as IA-SEM, allows imaging of the subcellular architecture of mammalian cells and reveals the locations and shape of membrane-bound organelles such as

Preprocessing of datasets

Our intention is to automatically classify and segment out the mitochondrial structures in MNT-1 cells (melanoma cells) from the rest of the cytoplasm that have a distinct texture. Datasets obtained using electron microscopes are intrinsically noisy due to various contributing factors during the imaging and reconstruction procedures [22]. Denoising algorithms based on nonlinear diffusion [19] and total variation methods [20] preserve sharpness and location of the edges, and in some cases

Experimental results

In our experiments, we consider two datasets of mitochondria in MNT-1 cells imaged using IA-SEM. Both the datasets were of low contrast and SNR. These datasets were denoised using texture preserving using the following parameters in order to enhance the texture part of the 3D dataset. In our experiments, τ=6, and μ=0.04, such that τμ<0.25. We set σ=1.5 (standard deviation of the Gaussian), filter size to be 10, λ=5, and ν={-2,0,2}. We define initial level-set function such that it points

Conclusion and future work

In this paper, we investigated automatic techniques for joint classification and segmentation and quantitative analysis of mitochondria in mammalian cells imaged using ion-abrasion scanning electron microscope. Given manually labeled images, a machine learning approach was formulated that exploits texture features, and joint image block-wise classification and segmentation of novel instances of the regions on diverse test datasets was performed by histogram matching using a nearest neighbor

About the Author—RAJESH NARASIMHA received the Bachelor's degree in electronics and communication engineering from Mangalore University, Mangalore, India, in 1999, the Master's degree in electrical engineering from the Rochester Institute of Technology (RIT), Rochester, NY, in 2002, and the Ph.D. degree in electrical and computer engineering from the Georgia Institute of Technology, Atlanta, in 2007. From 1999 to 2000, he was a Research Associate with the Electronics and Communication

References (37)

  • J.A. Heymann et al.

    Site-specific 3D imaging of cells and tissues with a dual beam microscope

    Journal of Structual Biology

    (2006)
  • G.R. Rosania

    Mitochondria give cells a tan

    Chem. Bio.

    (2005)
  • L. Rudin et al.

    Nonlinear total variation based noise removal algorithms

    Phys. D: Nonlinear Phenom.

    (1992)
  • S.A. Abraham et al.

    Liposomal formulations of doxorubicin

  • J. Heymann et al.

    3D imaging of melanoma cells using automated ion abrasion scanning electron microscopy

    Microscopy and Microanalysis

    (2007)
  • J.A. Heymann, D. Shi, S. Kim, D. Bliss, L.S. Jacqueline, S. Subramaniam, Nanometer resolution 3D imaging of mammalian...
  • J. Winn, A. Criminisi, T. Minka, Object Categorization by Learned Universal Visual Dictionary, in: Proceedings of the...
  • M. Varma et al.

    A statistical approach to texture classification from single images

    Int. J. Comp. Vision

    (2005)
  • T. Leung et al.

    Representing and recognizing the visual appearance of materials using three-dimensional textons

    Int. J. Comp. Vision

    (2001)
  • O.G. Cula et al.

    Compact representation of bidirectional texture functions

    Proc. Comput. Vision Pattern Recognition

    (2001)
  • J. Zhang et al.

    Local features and kernels for classification of texture and object categories: a comprehensive study

    Int. J. Comp. Vision

    (2007)
  • J. Kaufhold et al.

    Recognition and segmentation of scene content using region-based classification

  • D.R. Martin et al.

    Learning to detect natural image boundaries using brightness color and texture cues

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2002)
  • J. Malik et al.

    Contour and texture analysis for image segmentation

    Int. J. Comp. Vision

    (2001)
  • H. Zhang et al.

    SVM-KNN: discriminative nearest neighbor classification for visual category recognition

  • O. Tuzel, L. Yang, P. Meer, D.J. Foran, Classification of hematologic malignancies using texton signatures, Pattern...
  • E.N. Issam et al.

    A support vector machine approach for detection of microcalcifications

    IEEE Trans. Med. Imaging

    (2002)
  • D.A. Adjeroh et al.

    Texton-based segmentation of retinal vessels

    J. Opt. Soc. Am. A

    (2007)
  • Cited by (42)

    • Machine learning for locating organic matter and pores in scanning electron microscopy images of organic-rich shales

      2019, Fuel
      Citation Excerpt :

      Various segmentation methods have been proposed to process SEM images to derive the information at nanoscale. Narasimha et al. [25] tested kNN, SVMs and Adaboost methods on SEM mammalian cells images in segmentation of mitochondria using texton-based region features. Segmentation results demonstrate the machine learning method can perform close to manual segmentation carried out by an experienced user.

    • On the projection similarity in line grouping

      2015, Pattern Recognition Letters
      Citation Excerpt :

      This fact accounts for the vast range of mitochondrial shapes and textures and challenges a unified approach to localization and segmentation. Presently the segmentation of mitochondria is performed manually [16], though automatic classification has also been recently reported in [14,7]. In many cases transmission electron microscope (TEM) images of mitochondria show characteristic quasi-linear structural elements (Fig. 1(c)–(d)) which can be converted to the proper line segments.

    • Unsupervised mitochondria segmentation using recursive spectral clustering and adaptive similarity models

      2013, Journal of Structural Biology
      Citation Excerpt :

      Due to their central role in cellular bioenergetics, apoptosis and autophagy, identification and classification of mitochondria is an area of increasing biomedical importance. Narasimha et al. (2009), for example, propose an automatic texton-based joint classification and segmentation of mitochondria by utilizing k-nearest neighbor (k-NN) classifier, support vector machines (SVMs), adaptive boosting (AdaBoost) and histogram matching. Another texture-based approach is reported in Vitaladevuni et al. (2008) where authors consider the detection of mitochondria in TEM images of brain tissue and apply the Gentle-Boost classifier.

    View all citing articles on Scopus

    About the Author—RAJESH NARASIMHA received the Bachelor's degree in electronics and communication engineering from Mangalore University, Mangalore, India, in 1999, the Master's degree in electrical engineering from the Rochester Institute of Technology (RIT), Rochester, NY, in 2002, and the Ph.D. degree in electrical and computer engineering from the Georgia Institute of Technology, Atlanta, in 2007. From 1999 to 2000, he was a Research Associate with the Electronics and Communication Engineering Department. Indian Institute of Science (IISc), Bangalore working on speech recognition for Ericsson, Inc. From August to December 2002, he was a Research Associate with RIT, and collaborated with Xerox Corporation, Webster, Rochester, on content-based video summarization and browsing applications. From 2006 to 2007, he was a Predoctoral Fellow at the National Cancer Research Institute, NIH, Bethesda, MD, where he worked on applications of machine learning to biology. He is the recipient of an SPIE scholarship from 2003 to 2004 and the NIH Predoctoral Fellowship from 2006 to 2007. Currently he is a Member of the Technical Staff at the DSPS R&D Center, Texas Instruments, Incorporated. He is a member of SPIE and IEEE. His research interests span broad areas of signal, image, and video processing; machine learning applications to scalable data mining from large and complex datasets; and statistical modeling and data analysis.

    About the Author—HUA OUYANG is currently pursuing his Ph.D. degree at the College of Computing at the Georgia Institute of Technology. His research interests are machine learning applications for biology.

    About the Author—ALEXANDER GRAY After completing Bachelor's degrees in Applied Mathematics (concentration in Computational Statistics) and Computer Science from UC Berkeley, spending Summers at the Santa Fe Institute and Los Alamos National Laboratory, among other places, Alexander Gray worked in the Machine Learning Systems Group of NASA's Jet Propulsion Laboratory, then completed his Ph.D. in Computer Science and a Postdoctoral at Carnegie Mellon University supervised by Prof. Andrew Moore. Since August 2005 he has been an Assistant Professor in the College of Computing at Georgia Tech, within the new Interactive and Intelligent Computing Division and affiliated with the even newer Computational Science and Engineering Division. He is a Member of the Center for the Study of Systems Biology, Center for Robotics and Intelligent Machines, Center for Experimental Research in Computer Systems, and the Graphics, Visualization and Usability Center, and affiliated with the Industrial and Technology Statistics Center.

    About the Author—STEVEN W. MCLAUGHLIN received the B.S. degree from Northwestern University, Evanston, IL, in 1985, the M.S.E. degree from Princeton University, Princeton, NJ, in 1986, and the Ph.D. degree from the University of Michigan, Ann Arbor, in 1992, all in electrical engineering. From 1992 to 1996, he was the Electrical Engineering faculty at the Rochester Institute of Technology, Rochester, NY. He joined the School of Electrical and Computer Engineering at the Georgia Institute of Science and Technology, Atlanta, in 1996, where he is Ken Byers Professor of Electrical and Computer Engineering and Vice Provost for International Initiatives. His research interests are in the general area of communications and information theory. His research group has ongoing projects in the areas of turbo, LDPC , and constrained codes for magnetic and optical recording; forward error correction (FEC) and equalization for wireless and optical networks; quantum key distribution, wireless and radio-frequency identification (RFID) security; and theory of error-control coding. He has published more than 200 papers in journals and conferences and holds 26 US patents. Dr. McLaughlin received the Presidential Early Career Award for Scientists and Engineers (PECASE) where he was cited “for leadership in the development of high-capacity, nonbinary optical recording formats.” He received (with Dr. David Warland at UC-Davis) the Information Storage Industries Consortium Technical Achievement Award in 2002 for “pioneering work in the development of multilevel optical disk storage technology”. In 2005, he was President of the IEEE Information Theory Society. He was an Associate Editor for Coding Techniques for the IEEE TRANSACTIONS ON INFORMATION THEORY and was Publications Editor for that journal from 1995 to 1999. He co-edited (with S.Verdu) Information Theory: 50 Years of Discovery (Wiley/IEEE Press, 1999). He has also served on the IEEE Publications Activities Board (1998–2001) and is a former Secretary of the IEEE Atlanta Section (2000).

    About the Author—SRIRAM SUBRAMANIAM obtained his Ph.D. in Physical Chemistry from Stanford University in 1987 and carried out postdoctoral work in the Departments of Chemistry and Biology at the Massachusetts Institute of Technology. In 1992, he joined the faculty at the Johns Hopkins University School of Medicine as an Assistant Professor, and was promoted to Associate Professor in 1998. He was appointed as the Chief of the Biophysics Section in the Laboratory of Biochemistry in 1998, and subsequently Chief of the Biophysics Section in the Laboratory of Cell Biology in 2003. He continues to maintain a visiting faculty appointment with the Johns Hopkins University School of Medicine.

    View full text