Abstract
Fine-grained classification and grading of breast cancer (BC) histopathological images are of great value in clinical application. However, automatic classification and grading of BC histopathological images are complicated by (1) small inter-class variance and large intra-class variance exist in BC histopathological images, and (2) features extracted from similar histopathological images with different magnification are quite different. To address these issues, an improved deep convolution neural network model is proposed and the procedure can be divided into three main stages. Firstly, in the representation learning process, multi-class recognition task and verification task of image pair are combined. Secondly, in the feature extraction process, a prior knowledge is built, which is “the variances in feature outputs between different subclasses is relatively large while the variance between the same subclass is small.” Additionally, the prior information that histopathological images with different magnification belong to the same subclass are embedded in the feature extraction process, which contributes to less sensitive with image magnification. The experimental results based on three different histopathological image datasets show that the performance of the proposed method is better than state of the art, with better robustness and generalization ability.
Similar content being viewed by others
References
Akbar S, Peikari M, Salama S, Nofech-Mozes S, Martel A (2017) Transitioning between convolutional and fully connected layers in neural networks. In: Deep learning in medical image analysis and multimodal learning for clinical decision support (DLMIA)
Bayramoglu N, Kannala J, Heikkilä J (2016) Deep learning for magnification independent breast cancer histopathology image classification. In: International conference on pattern recognition (ICPR), pp 2441–2446
Bell S, Bala K (2015) Learning visual similarity for product design with convolutional neural networks. ACM Trans Graph 34(4):98:1–98:10
Chen H, Dou Q, Wang X et al (2016) Mitosis detection in breast cancer histology images via deep cascaded networks. In: Proceedings of the thirtieth AAAI conference on artificial intelligence (AAAI), pp 1160–1166
Chollet F (2016) Xception: deep learning with depthwise separable convolutions. arXiv:1610.02357
Chopra S, Hadsell R, LeCun Y (2005) Learning a similarity metric discriminatively, with application to face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 539–546
Codella N, Moradi M, Matasar M, Sveda-Mahmood T, Smith JR (2016) Lymphoma diagnosis in histopathology using a multi-stage visual learning approach. In: Proceedings of the SPIE, medical imaging 2016: digital pathology
Dimitropoulos K, Barmpoutis P, Zioga C, Kamas A, Patsiaoura K, Grammalidis N (2017) Grading of invasive breast carcinoma through grassmannian VLAD encoding. Plos One 12(9):e0185110
Dozat T (2015) Incorporating nesterov momentum into adam. Technical report, http://cs229.stanford.edu/proj2015/054_report.pdf
Garud H, Karri SPK, Sheet D et al (2017) High-magnification multi-views based classification of breast fine needle aspiration cytology cell samples using fusion of decisions from deep convolutional networks. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 828–833
Gupta V, Bhavsar A (2017) An integrated multi-scale model for breast cancer histopathological image classification with joint colour-texture features. In: 17th international conference on computer analysis of images and patterns (CAIP), pp 354–366
Gupta V, Bhavsar A (2017) Breast cancer histopathological image classification: is magnification important?. In: IEEE conference on computer vision and pattern recognition workshops (CVPRW), pp 769–776
Gupta V, Singh A, Sharma K et al (2017) Automated classification for breast cancer histopathology images: is stain normalization important?. In: Computer assisted and robotic endoscopy and clinical image-based procedures (CLIP) 10550, pp 160–169
Hadsell R, Chopra S, LeCun Y (2006) Dimensionality reduction by learning an invariant mapping. In: Computer vision and pattern recognition (CVPR), pp 1735–1742
Han Z, Wei B, Zheng Y, Yin Y, Li K, Li S (2017) Breast cancer multi-classification from histopathological images with structured deep learning model. Sci Rep 7(1):4172
Janowczyk A, Madabhushi A (2016) Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform 7(1):29
Kowal M, Filipczuk P, Obuchowicz A et al (2013) Computer-aided diagnosis of breast cancer based on fine needle biopsy microscopic images. Comput Biol Med 43 (10):1563–1572
Lan R, Zhou Y (2016) Quaternion-michelson descriptor for color image classification. IEEE Trans Image Process 25(11):5281–5292
Lan R, Zhou Y, Tang Y (2016) Quaternionic local ranking binary pattern: a local descriptor of color images. IEEE Trans Image Process 25(2):566–579
Lan R, Zhou Y, Tang Y (2017) Quaternionic weber local descriptor of color images. IEEE Trans Circuits Syst Video Techn 27(2):261–274
Lan R, Zhong S, Liu Z et al (2018) A simple texture feature for retrieval of medical images. Multimed Tools Appl 77(9):10853–10866
LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521 (7553):436–444
Li P, Wang D, Wang L, Lu H (2018) Deep visual tracking: review and experimental comparison. Pattern Recogn 76:323–338
Li Y, Lu H, Li KC et al (2018) Non-uniform de-scattering and de-blurring of underwater images. Mob Netw Appl 23(4):352–362
Lu H, Zhang L, Serikawa S (2012) Maximum local energy: an effective approach for multisensor image fusion in beyond wavelet transform domain. Comput Math Appl 64(5):996–1003
Lu H, Li B, Zhu J et al (2017) Wound intensity correction and segmentation with convolutional neural networks. Concurr Comput Pract Experience 29(6):3927
Lu H, Li Y, Chen M et al (2018) Brain intelligence: go beyond artificial intelligence. Mob Netw Appl 23(2):1–8
Pan X, Li L, Yang H et al (2017) Accurate segmentation of nuclei in pathological images via sparse reconstruction and deep convolutional networks. Neurocomputing 229:88–99
Pan X, Yang D, Li L et al (2018) Cell detection in pathology and microscopy images with multi-scale fully convolutional neural networks. World Wide Web-internet & Web Information Systems 21(6):1721–1743
Song Y, Zou J, Chang H et al (2017) Adapting fisher vectors for histopathology image classification. In: International symposium on biomedical imaging (ISBI), pp 600–603
Song Y, Chang H, Huang H, Cai W (2017) Supervised intra-embedding of fisher vectors for histopathology image classification. In: Medical image computing and computer-assisted intervention (MICCAI)
Spanhol FA, Cavalin P, Oliveira LS, Petitjean C, Heutte L (2017) Deep features for breast cancer histopathological image classification. In: IEEE international conference on systems, man, and cybernetics (IEEE SMC), pp 1868–1873
Spanhol FA, Oliveira LS, Petitjean C et al (2016) A dataset for breast cancer histopathological image classification. IEEE Trans Biomed Eng 63(7):1455–1462
Spanhol FA, Oliveira LS, Petitjean C, Heutte L (2016) Breast cancer histopathological image classification using convolutional neural networks. In: 2016 international joint conference on neural networks, pp 2560–2567
Stewart B, Wild CP (2014) World cancer report 2014. International Agency for Research on Cancer, World Health Organization
Sun Y, Chen Y, Wang X et al (2014) Deep learning face representation by joint identification-verification. In: International conference on neural information processing systems (ICNIPS), pp 1988–1996
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S et al (2015) Going deeper with convolutions. In: Computer vision and pattern recognition (CVPR)
Taigman Y, Yang M, Ranzato MA, Wolf L (2014) Deepface: closing the gap to human-level performance in face verification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 1701–1708
Wang D, Lu H, Li X (2011) Two dimensional principal components of natural images and its application. Neurocomputing 74(17):2745–2753
Wang P, Hu X, Li Y, Liu Q, Zhu X (2016) Automatic cell nuclei segmentation and classification of breast cancer histopathology images. Signal Process 122:1–13
Wang D, Khosla A, Gargeya R et al (2016) Deep learning for identifying metastatic breast cancer. arXiv:1606.05718
Wei B, Han Z, He X, Yin Y (2017) Deep learning model based breast cancer bistro pathological image classification. In: 2017 IEEE 2nd international conference on cloud computing and big data analysis (ICCCBDA)
Xiao T, Xu Y, Yang K et al (2015) The application of two-level attention models in deep convolutional neural network for fine-grained image classification. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp 842–850
Zhang S, Metaxas D (2016) Large-scale medical image analytics: recent methodologies, applications and future directions. Med Image Anal 33:98–101
Zhang Y, Zhang B, Coenen F et al (2014) One-class kernel subspace ensemble for medical image classification. EURASIP J Adv Signal Process 2014:17. https://doi.org/10.1186/1687-6180-2014-17
Zhang X, Su H, Yang L, Zhang S (2015) Fine-grained histopathological image analysis via robust segmentation and large-scale retrieval. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)
Zhang X, Zhou F, Lin Y, Zhang S (2016) Embedding label structures for fine-grained feature representation. In: 29th IEEE conference on computer vision and pattern recognition (CVPR), pp 1114–1123
Zhi W, Yueng HWF, Chen Z et al (2017) Using transfer learning with convolutional neural networks to diagnose breast cancer from histopathological images. In: 24th international conference on neural information processing (ICNIP)
Zhu X, Li X, Zhang S (2016) Block-row sparse multiview multilabel learning for image classification. IEEE Trans Cybern 46(2):450–461
Zhu X, Suk HI, Lee SW et al (2016) Subspace regularized sparse multitask learning for multiclass neurodegenerative disease identification. IEEE Trans Bio-Med Eng 63(3):607–618
Acknowledgements
The authors would like to thank Spanhol et al. [33], Dimitropoulos et al. [8], and Dr. Andrew Janowczyk et al. [16] for publishing the datasets. We would like to express our gratitude to anonymous reviewers and editor for their helpful comments. This research was supported in part by the National Natural Science Foundation of China (Grant Nos. 21365008, 61462018, 61762026 and 61562013), and Natural Science Foundation of Guangxi Province (No. 2017GXNSFDA198025 and 2017GXNSFAA198278). The authors declare that they have no conflict of interest.
Author information
Authors and Affiliations
Corresponding authors
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Lingqiao Li and Xipeng Pan are contributed equally
Rights and permissions
About this article
Cite this article
Li, L., Pan, X., Yang, H. et al. Multi-task deep learning for fine-grained classification and grading in breast cancer histopathological images. Multimed Tools Appl 79, 14509–14528 (2020). https://doi.org/10.1007/s11042-018-6970-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-018-6970-9