Elsevier

Neurocomputing

Volume 406, 17 September 2020, Pages 267-273
Neurocomputing

StainCNNs: An efficient stain feature learning method

https://doi.org/10.1016/j.neucom.2020.04.008Get rights and content

Abstract

Color variation in stained histopathology images prevents the development of computer-assisted diagnosis (CAD) algorithms for whole slide imaging systems. Therefore, stain normalization methods are studied to reduce the influence of color variation combined with digital image processing algorithms. The Structure Preserve Color Normalization (SPCN) method is a promising stain normalization method, utilizing the sparse non-negative matrix factorization to estimate the stain feature appearance matrix. However, the SPCN method suffers from the high computational complexity of dictionary learning, and its official implementation relies on Matlab and CPU. This research proposes the StainCNNs method to simplify the process of stain feature extraction, and imply a GPU-enabled realization to accelerate the learning of stain features in the Tensorflow Framework. What’s more, the StainCNNs method is able to perform the stain normalization quickly in dataset level, more efficient than the SPCN method which is unable to make use of the stain feature distribution in dataset. Stain normalization experiments are conducted on the Camelyon16 dataset and the ICPR2014 dataset, evaluated by the QSSIM score and the FSIM score. Results demonstrate that the proposed StainCNNs method achieves a state-of-the-art performance compared with many conventional stain normalization methods.

Introduction

Whole slide imaging (WSI) brings a paradigm shift on pathology, affecting its reproducibility, dissemination of educational material, pathological workflows and service quality in developing areas. With the help of modern WSI systems, pathologists are able to view virtual tissue slides in the similar way they scroll a Google map. The digitization of WSI leads to new opportunities which are difficult with conventional microscopes. Digital collaboration, integration with health records, and electronic workflows can be easily achieved through WSI. What’s more, machine learning and deep learning techniques can strongly support the data analysis of WSI slides for disease detection, prognosis prediction, segmentation and so on.

Though WSI encompasses the digitization of pathological slides, it is difficult for pathologists and softwares to distinguish the undesirable color variation on a particular stain appearance [1], [2]. Different slide scanners can produce various color responses to the same raw materials, and the presentation lookup tables may exhibit visible color variations [3]. The protocols for preparing samples are usually different between laboratories, including the acquisition, fixation and staining of tissues, which are important sources of perceived color differences. Undesirable color variation can confuse the already existing inter- and intra-expert experience in labeling and diagnosis among pathologists [4].

Color normalization techniques in image preprocessing have been studied to improve the color consistency of medical images and the color accuracy of diagnosis [5]. Stain normalization refers to the color mapping to adjust the color values of the source image in a pixel-to-pixel approach so as to match the color distribution of the template image [1]. In terms of the patterns to extract stain features, the stain normalization methods can be roughly classified into conventional stain normalization methods and deep learning methods to remove the undesirable color variation in WSIs.

Conventional stain normalization methods usually extract stain features with image processing algorithms in pixel-level. There are two types of conventional stain normalization methods: (1) Global color normalization; (2) Color normalization after stain separation. Gonzalez et al. mapped the histogram of the source image with the histogram of the target image in the lαβ color space, then converted the normalized image into the RGB space [6]. The Reinhard method transfers the background color from the source image into the template image in the lαβ space, so as to preserve the structure of source image [7]. Conventional stain normalization methods with stain separation operation usually evaluate the stain feature matrix, and then match the stain features from the source slide image to the target image. Ruifrok et al. measured the relative color proportion for R, G, B channel on every stain, and evaluated the stain color appearance matrix and the stain depth matrix for stain normalization [8]. Macenko et al. utilized the single value decomposition (SVD) to create a plane with the two largest singular values, and projected data to this plane finding the corresponding angles. Finally, stain color matrix was estimated with the maximum and minimum angles for robust stain normalization [9]. Though the SVD operation has a closed solution of the stain color feature matrix, the Macenko method loses some information during the SVD, and its evaluation of the extreme angle values is empirical.

The SPCN method is reported to provide better qualitatively and qualitatively results among conventional stain normalization methods for colorectal and cancer breast histopathology datasets [10]. The SPCN method is comprised of two part: stain separation by the sparse non-negative matrix factorization (SNMF) and the structure preserve color normalization [11], as shown in Fig. 2. The SNMF operation introduces sparseness constraint to the formula of NMF equation, so that the estimated stain color appearance matrix is sparse. Quantitative comparison on color basis and stain density maps showed the SNMF outperformed the NMF significantly [11]. In the normalized image, the structure preserve color normalization preserves the structure information of the source image. The SPCN method achieves stable stain normalization performance, but suffers from high computation complexity in the dictionary learning during SNMF, as Fig. 3(a) shows.

Recently, a novel feature extraction pattern through deep hierarchical convolutional layers is proposed in deep learning area. Deep learning methods are observed to outperform on public competitions of computer vision, like ImageNet [12], MS COCO [13], and promote computer vision areas in facial expression recognition [14], video caption [15], state estimation [16], strip analysis [17] and so on. Improvements in medical image analysis have also been observed with deep learning methods, including registration [18], detection [19], and segmentation [20], [21].

Deep-learning based color transfer methods can be roughly classified into supervised methods and unsupervised ones. Supervised deep learning color transfer methods often collect two large datasets of images, the source dataset and the target dataset, where the color style in the target dataset is restricted to keep blank. Shaban et al. proposed a pure learning based cycleGANs method for image-to-image translation between histological datasets, and preserved the tissue structure through the cycle-consistency constraint [22]. Unsupervised deep learning based color transfer methods usually force the deep feature distribution of certain layer of the source input and template input similar. Li et al. utilized whiten-color-transform (WCT) to process the encoder data, transferred the color feature distribution, and generated the transferred image with decoder network [23]. This WCT universal style transform method is able to learn the style of arbitrary template image, and transfers the source image in an unsupervised way. However, it is not promising to preserve the structure information and has the risk of introducing artifacts, which are undesirable for histopathological diagnosis.

This research aims at accelerating the SPCN method by simplifying the stain color feature extraction, and implies convolutional neural networks (CNNs) method without the strict constraint on datasets. The acceleration is accomplished by two strategies: GPU-enabled realization and deep feature learning by CNNs. Stain normalization experiments are conducted to assess the performance of the StainCNNs method on breast cancer datasets, which are important for developing CAD systems in pathology.

Briefly, the contributions of this article can be summarized as follows:

(1) We propose the StainCNNs method to simplify the evaluation pipeline of stain separation, which evaluating the stain color appearance matrix from the source slide image patches, without the color space transformation and other complex image processing operations.

(2) A GPU-enabled StainCNNs model is implemented to accelerate the stain feature extraction, which is a major computation burden to utilize the SPCN method. A novel joint loss function is designed for stain color appearance matrix, considering the sparseness constraint from the SNMF.

(3) We conduct comprehensive stain normalization experiments on the Camelyon16 dataset and the ICPR2014 dataset, comparing the proposed StainCNNs method with conventional stain normalization methods. Experiments’ result demonstrates that the StainCNNs method performs state-of-the-art stain normalization in an efficient manner.

Section snippets

Color deconvolution method

Color Deconvolution (CD) method is a classical stain normalization method with stain separation, estimating color proportions of stain dyes in every possible location [8]. It designs experiments to empirically evaluate the stain color appearance matrix from the input slides with a single stain per slide. The CD method is calibration-based and separates light absorbing stains on the guidance of Beer-Lambert law [24].

Let I ∈ Rm × n be the matrix of intensities of input slide image, where m is set

Methods

The pipeline of the StainCNNs method is concluded in Fig. 3, collecting the stain color appearance matrix by the SPCN method for training pattern. We first convert the source image Is into the optical density space signed as Vs on the guidance of the Beer-Lambert law, then evaluate the stain color appearance matrix Ws through the trained StainCNNs model. Next, we compute the stain density map Hs in the definition of the relative optical density V=WH, and the target image It is processed in the

Experiments and results

Stain normalization experiments are conducted separately on Camelyon16 dataset and ICPR2014 dataset to evaluate the performance and efficiency of the proposed StainCNNs method. Image patches from multiple WSIs are randomly shuffled and divided into the training dataset and the testing dataset, so that the stain color appearance distributions in both datasets are similar. We design the sensitivity experiments on Camelyon16 dataset and test the robustness of the StainCNNs method on ICPR2014

Discussions and conclusions

This research proposes a novel StainCNNs method to accelerate the stain feature evaluation in the SPCN method, combining the advantages of deep learning methods. Iterative training strategy of dictionary learning in the SPCN method suffers from high computational cost, and its official implementation is based on CPU hardware. Deep learning methods have produced state-of-the-art performance on many computer vision tasks, and are usually implemented efficiently on GPU or TPU hardware. We utilize

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

CRediT authorship contribution statement

Gaoyi Lei: Conceptualization, Software, Investigation. Yuanqing Xia: Visualization, Project administration. Di-Hua Zhai: Funding acquisition, Supervision. Wei Zhang: Writing - original draft. Duanduan Chen: Data curation. Defeng Wang: Writing - review & editing.

Acknowledgements

This work was supported by the Natural Science Foundation of Beijing Municipality under Grant Z170003.

Gaoyi Lei received the Bachelor degree from the School of Automation, Beijing Institute of Technology, Beijing, China, in 2017. He is currently a Master student with the School of Automation, Beijing Institute of Technology. His research interests include deep learning, medical image segmentation, attention mechanism.

References (29)

  • S. Roy et al.

    A study about color normalization methods for histopathology images

    Micron

    (2018)
  • N. Zeng et al.

    Facial expression recognition via learning deep sparse autoencoders

    Neurocomputing

    (2018)
  • J. Li et al.

    Automatic fetal brain extraction from 2d in utero fetal mri slices using deep neural network

    Neurocomputing

    (2020)
  • A.M. Khan et al.

    A nonlinear mapping approach to stain normalization in digital histopathology images using image-specific color deconvolution

    IEEE Trans. Biomed. Eng.

    (2014)
  • L. Peter et al.

    Leveraging random forests for interactive exploration of large histological images

    Medical Image Computing and Computer-Assisted Intervention – MICCAI 2014

    (2014)
  • M.D. Zarella et al.

    A practical guide to whole slide imaging: a white paper from the digital pathology association

    Arch. Pathol. AMP Labor. Med.

    (2019)
  • H. Llewellyn

    Observer variation, dysplasia grading, and hpv typing: a review

    Pathol. Patterns Rev.

    (2000)
  • X. Sun et al.

    Histogram-based normalization technique on human brain magnetic resonance images from different acquisitions

    Biomed. Eng. Online

    (2015)
  • R.C. Gonzalez, R.E. Woods, S.L. Eddins, Digital image processing using...
  • E. Reinhard et al.

    Color transfer between images

    IEEE Comput. Graph. Appl.

    (2001)
  • A.C. Ruifrok et al.

    Quantification of histochemical staining by color deconvolution

    Analyt. Quant. Cytol. Histol.

    (2001)
  • M. Macenko, M. Niethammer, J.S. Marron, D. Borland, J.T. Woosley, X. Guan, C. Schmitt, N.E. Thomas, A method for...
  • A. Vahadane et al.

    Structure-preserving color normalization and sparse stain separation for histological images

    IEEE Trans. Med. Imaging

    (2016)
  • A. Vahadane et al.

    Structure-preserving color normalization and sparse stain separation for histological images

    IEEE Trans. Med. Imaging

    (2016)
  • Cited by (18)

    • Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions

      2022, Information Fusion
      Citation Excerpt :

      Most harmonisation strategies (also named stain/colour normalisation) in pathology aimed to address the stain variance. These studies mainly focused on uterus [108], breast [39,62,102,103,107,110 113], lymphoma [103,138], skin [105], liver [106], renal [59], and stomach [109]. Metabolomics is an omics technology that monitors and discovers metabolic changes in people in relation to illness state or in reaction to a medical or external intervention using current analytical instruments and pattern recognition algorithms.

    View all citing articles on Scopus

    Gaoyi Lei received the Bachelor degree from the School of Automation, Beijing Institute of Technology, Beijing, China, in 2017. He is currently a Master student with the School of Automation, Beijing Institute of Technology. His research interests include deep learning, medical image segmentation, attention mechanism.

    Yuanqing Xia was born in Anhui Province, China in 1971, and graduated from the Department of Mathematics, Chuzhou University, China in 1991. He received a MSc in Fundamental Mathematics from Anhui University, China, in 1998, and a PhD in Control Theory and Control Engineering from Beijing University of Aeronautics and Astronautics, China, in 2001. From 1991 to 1995 he was with Tongcheng Middle-School, China, where he worked as a teacher. From January 2002 to November 2003 he was a postdoctoral research associate at the Institute of Systems Science, Academy of Mathematics and System Sciences, Chinese Academy of Sciences, China, where he worked on navigation, guidance, and control. From November 2003 to February 2004 he was with the National University of Singapore as a Research Fellow, where he worked on variable structure control. From February 2004 to February 2006 he was with the University of Glamorgan, UK, as a Research Fellow, where he studied networked control systems. From February 2007 to June 2008 he was a guest professor with Innsbruck Medical University, Austria, where he worked on biomedical signal processing. Since July 2004 he has been with the School of Automation, Beijing Institute of Technology, Beijing, first as an Associate Professor and then since 2008 as Professor. In 2012 he was appointed Xu Teli Distinguished Professor at the Beijing Institute of Technology, and then in 2016 he was made Chair Professor. In 2012 he obtained the National Science Foundation for Distinguished Young Scholars of China; in 2016 he was honored as the Yangtze River Scholar Distinguished Professor and was supported by the National High Level Talents Special Support Plan (”Million People Plan”) by the Organization Department of the CPC Central Committee. He is now the Dean of the School of Automation, Beijing Institute of Technology. He has published ten monographs in Springer, John Wiley, and CRC, and more than 400 papers in international scientific journals. He is a deputy editor of the Journal of Beijing Institute of Technology and an associate editor of Acta Automatica Sinica; Control Theory and Applications; the International Journal of Innovative Computing, Information, and Control; and the International Journal of Automation and Computing. He obtained the Second Award of the Beijing Municipal Science and Technology (No. 1) in 2010 and 2015, the Second National Award for Science and Technology (No. 2) in 2011, and the Second Natural Science Award of the Ministry of Education (No. 1) in 2012 and 2017. His research interests include cloud control systems, networked control systems, robust control and signal processing, active disturbance rejection control, unmanned system control and flight control.

    Di-Hua Zhai received his B.Eng. degree in Automation from Anhui University, Hefei, China, in 2010, his M.Eng. degree in Control Science and Engineering from the University of Science and Technology of China, Hefei, China, in 2013, and his Dr.Eng. degree in Control Science and Engineering from Beijing Institute of Technology, Beijing, China, in 2017. Since 2017, he has been with the School of Automation, Beijing Institute of Technology, Beijing, China, where he is currently an Assistant Professor. His research interests are in the fields of teleoperation, intelligent robot, human-robot collaboration, switched control, networked control.

    Wei Zhang was born in 1970. He received the Master degree in computer application from QuFu normal university, Qu’fu, China, in 1996. He is currently a Full Professor with Zaozhuang University. His research interests include computer network theory and information security.

    Duanduan Chen was born in Beijing, China, in 1982. She received the DPhil degree in Biomedical Engineering in 2009, at the University of Oxford, UK, and the B.Sc. degree in Mechanical Engineering in 2005, at Fudan University, China. She is now a professor at the School of Life Science in Beijing Institute of Technology, China. Her research focuses on biomechanics and computer simulation. The developed techniques could be applied in surgery planning of various vascular and brain diseases.

    Defeng Wang received the B.Sc. degree from the Department of Computer Science, Jilin University, and the Ph.D. degree from the Department of Computing, The Hong Kong Polytechnic University. He is currently a Professor with the School of Instrumentation Science and Optoelectronics Engineering, Beihang University, and the Beijing Advanced Innovation Center for Big Data-Based Precision Medicine. His research interests include medical imaging, statistical morphometry analysis, fMRI paradigm design and post-processing, quantitative medical image analysis, and computational life science.

    View full text