Elsevier

Pattern Recognition

Volume 71, November 2017, Pages 14-25
Pattern Recognition

Feature extraction from histopathological images based on nucleus-guided convolutional neural network for breast lesion classification

https://doi.org/10.1016/j.patcog.2017.05.010Get rights and content

Highlights

  • A feature extraction framework based on convolutional neural network is proposed.

  • A convolutional neural network is designed under the constraint of nuclei locations.

  • Features including the pattern and spatial distribution of the nuclei are extracted.

  • The proposed algorithm achieves a good classification performance with fast speed.

Abstract

Feature extraction is a crucial and challenging aspect in the computer-aided diagnosis of breast cancer with histopathological images. In recent years, many machine learning methods have been introduced to extract features from histopathological images. In this study, a novel nucleus-guided feature extraction framework based on convolutional neural network is proposed for histopathological images. The nuclei are first detected from images, and then used to train a designed convolutional neural network with three hierarchy structures. Through the trained network, image-level features including the pattern and spatial distribution of the nuclei are extracted. The proposed features are evaluated through the classification experiment on a histopathological image database of breast lesions. The experimental results show that the extracted features effectively represent histopathological images, and the proposed framework achieves a better classification performance for breast lesions than the compared state-of-the-art methods.

Introduction

Breast cancer is the second most commonly occurring cancer for females. According to an estimation from the American Cancer Society in 2017 [1], around 30% of the new cancer cases in American women consist of breast cancer. New technologies [2], [3], [4] for breast cancer diagnosis have been developed in recent years. However, final diagnosis still currently relies on biopsies [5]. Owing to the development in digital pathology, the whole slide image (WSI) of a histopathological section can be captured within a few minutes by a micro scanner, and stored in a data server. These cancer images provide valuable data sources for researches in computer science and related areas, such as image processing, pattern recognition, and data mining. As a result, many approaches to histopathological image analysis [6], [7], [8] have been developed and applied to practical clinical diagnoses, relieving the workload of pathologists and assisting pathologists in making more reliable and consistent diagnoses.

Across various applications of histopathological image analysis, image classification and content-based image retrieval (CBIR) are always important challenges. Histopathological image classification aims to directly predict the types of lesions (for example, the classification of benign and malignant tumors). A reliable classification system can provide a verification of the diagnosis of the doctor. Meanwhile, CBIR can search and return cases with similar content to the query image. Using information regarding diagnoses of similar cases for reference, doctors can reach a more reliable diagnosis for the query image. For these applications, feature extraction plays a crucial role. However, histopathological images may contain hundreds of structures [9] and the appearance within the same lesion is varied [10], which makes feature extraction a challenging task.

In this paper, a novel nuclei-guided feature extraction method based on convolutional neural network is proposed for histopathological images stained by hematoxylin and eosin (HE). The nuclei in the image are first detected. Considering the location information of the nuclei, a fine-designed neural network is trained to extract features regarding the pattern and distribution of the nuclei. Through classification experiments on a breast-lesion database, the proposed features are validated to be effective for histopathological images.

The remainder of this paper is organized as follows. Section 2 reviews relevant work regarding histopathological image analysis. Section 3 introduces the proposed feature extraction method. The experiment and discussion are presented in Sections 4 and 5. Finally, Section 6 summarizes the present contributions and suggests directions for future work.

Section snippets

Related work

Many feature extraction methods have been proposed for histopathological images. They can be broadly classified into two categories: statistics based method and learning based method.

Inspired by the success of natural image analysis, some researchers [11], [12], [13], [14] have employed classical feature descriptors, such as color histogram, scale-invariant feature transform (SIFT) [15], histogram of oriented gradient (HOG) [16], and local binary pattern (LBP) [17], to depict histopathological

Method

According to pathologists [5], [43], the appearance of the cell nucleus and its surrounding cytoplasm, as well as the distribution of nuclei, are important indicators for cancer diagnosis. Therefore, both the appearance and distribution of nuclei are considered. The proposed framework consists of an offline training stage and an online encoding stage, where the former can be divided into three steps: nuclei detection, pre-training, and fine-tuning.

Fig. 1 presents a flowchart of the training

Experiment

In this paper, a novel nucleus-guided feature extraction framework based on CNN is proposed for histopathological images. The proposed algorithm is implemented in Matlab 2013a on the PC with a 12 Intel Core Processor (2.10  GHz) and a GPU of Nvidia Tesla k40. The implementation of the whole network is based on the UFLDL tutorial.1

The performance of the proposed method is evaluated using a fine-annotated histopathological image

Discussion

In this paper, the nucleus features including the pattern and spatial distribution are extracted from histopathological images using a designed CNN combined with the guide of nuclei, which has good classification performance for breast lesions. Regarding the proposed method, we have the following discussion:

Conclusion

The appearance and spatial distribution of cell nuclei are significant indicators for cancer diagnosis, according to which a nucleus-guided feature extraction framework based on a convolutional neural network has been proposed for histopathological image representation and classification in this paper. The nuclei are first detected from the histopathological image, and then used to guide the training of the convolutional neural network. With the nucleus-guided strategy, the network paid more

Acknowledgment

This work was supported by the National Natural Science Foundation of China (No. 61371134 and 61471016) and project of Motic-BUAA Image Technology Research and Development Center.

Yushan Zheng received the B.E. and M.S. degree from Beihang University in 2012 and 2015 respectively. He is currently a Ph.D. candidate majored in Pattern Recognition and Intelligent System at Beihang University. His research interests include medical image processing, analysis, classification and retrieval.

References (50)

  • R.L. Siegel et al.

    Cancer statistics, 2017

    CA

    (2017)
  • J. Wu et al.

    Identifying relations between imaging phenotypes and molecular subtypes of breast cancer: model discovery and external validation

    J. Magn. Reson. Imaging

    (2017)
  • J. Wu et al.

    Unsupervised clustering of quantitative image phenotypes reveals breast cancer subtypes with distinct prognoses and molecular pathways

    Clin. Cancer Res.

    (2017)
  • S.L. Robbins et al.

    Robbins and Cotran Pathologic Basis of Disease

    (2010)
  • C. Mosquera-Lopez et al.

    Computer-aided prostate cancer diagnosis from digitized histopathology: a review on texture-based systems

    IEEE Rev. Biomed. Eng.

    (2014)
  • J.S. Duncan et al.

    Medical image analysis: progress over two decades and the challenges ahead

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2000)
  • M.N. Gurcan et al.

    Histopathological image analysis: a review

    IEEE Rev. Biomed. Eng.

    (2009)
  • B.E. Bejnordi et al.

    Automated detection of dcis in whole-slide h&e stained breast histopathology images

    IEEE Trans. Med. Imaging

    (2016)
  • S.R. Lakhani et al.

    WHO Classification of Tumours of the Breast

    (2012)
  • J.C. Caicedo et al.

    Histopathology image classification using bag of features and kernel functions

    Artificial Intelligence in Medicine in Europe

    (2009)
  • E. Mercan et al.

    Localization of diagnostically relevant regions of interest in whole slide images

    International Conference on Pattern Recognition

    (2014)
  • D.G. Lowe

    Distinctive image features from scale-invariant keypoints

    Int. J. Comput. Vision

    (2004)
  • N. Dalal et al.

    Histograms of oriented gradients for human detection

    IEEE Conference on Computer Vision and Pattern Recognition

    (2005)
  • M.M. Dundar et al.

    Computerized classification of intraductal breast lesions using histopathological images

    IEEE Trans. Biomed. Eng.

    (2011)
  • L. Cheng et al.

    Automated analysis and diagnosis of skin melanoma on whole slide histopathological images

    Pattern Recognit.

    (2015)
  • Cited by (106)

    • A review: The detection of cancer cells in histopathology based on machine vision

      2022, Computers in Biology and Medicine
      Citation Excerpt :

      Stanitsas et al. [145] used the AlexNet to classify pathological images, which greatly reduced the requirements for graphics processing unit memory, thus avoiding the need for complex hardware. Zheng et al. [146] proposed a kernel-guided feature extraction framework based on CNN for the representation and classification of breast histopathology images with an average classification accuracy of 96.4%. Han et al. [147] adopted GoogleNet to significantly improve the classification performance through 22 layers deep network and novel initial module.

    • Iterative brain tumor retrieval for MR images based on user's intention model

      2022, Pattern Recognition
      Citation Excerpt :

      This implies a need to design a feature extraction framework to encode/combine both low-level and high-level features. One way to design such a framework is to use the deep learning that can extract powerful feature representations fully describing both the low-level and high-level information and embed the feature extraction phase into self-learning [20]. Nowadays, deep learning has been applied in MR image analysis such as segmentation [21,22], brain tumor classification [23,24], computer aided diagnosis and prediction [25], brain image organization [26].

    View all citing articles on Scopus

    Yushan Zheng received the B.E. and M.S. degree from Beihang University in 2012 and 2015 respectively. He is currently a Ph.D. candidate majored in Pattern Recognition and Intelligent System at Beihang University. His research interests include medical image processing, analysis, classification and retrieval.

    Zhiguo Jiang received the B.E., M.S., and Ph.D. degrees from the Beihang University in 1987, 1990, and 2005, respectively. He is currently a professor in Image Processing Center, School of Astronautics, Beihang University. His research interests include medical image processing and classification, remotely sensed image processing, target detection and recognition.

    Fengying Xie received the Ph.D. Degree in Pattern Recognition and Intelligent System from Beihang University in 2009. She is now a professor in Image Processing Center, School of Astronautics, Beihang University. Her research interests include biomedical image and remote sensing image processing, image quality assessment, image segmentation and classification.

    Haopeng Zhang received his B.S. and Ph.D. degrees from Beihang University in 2008 and 2014, respectively, and is currently with the Image Processing Center, School of Astronautics, Beihang University. His main research interests are multi-view object recognition, 3D object recognition and pose estimation, etc.

    Yibing Ma received the B.E. degree from Beihang University in 2011. He is currently a Ph.D. candidate majored in Pattern Recognition and Intelligent System at Beihang University. His research interests include medical image processing, analysis, classification and retrieval.

    Huaqiang Shi is a deputy chief physician of the General Hospital of the Air Force of PLA. He is also the medical advisor of Motic (Xiamen) Medical Diagnostic Systems Co. Ltd. His research interests are diagnosis of breast cancer and the clinical application of image analysis technology.

    Yu Zhao is the Software R&D Manager of Motic (Xiamen) Medical Diagnostic Systems Co. Ltd. His research interests are clinical application of image analysis technology, telemedicine system and content-based medical image retrieval.

    View full text