A hybrid network for automatic hepatocellular carcinoma segmentation in H&E-stained whole slide images
Graphical abstract
Introduction
Liver cancer is the fourth leading cause of cancer-related death worldwide, and it is estimated that more than one million patients will die from liver cancer in 2030 (Longo, 2019). As the most common type of primary malignant liver cancer, hepatocellular carcinoma (HCC) occurs most often in people with cirrhosis, which is a leading cause of liver cancer. Although some imaging techniques, such as magnetic resonance imaging, computed tomography, and ultrasound, have made significant improvements in liver cancer detection, pathology imaging that is based on digitized specimen slides is still the gold standard for cancer diagnosis (Lu and Mandal, 2015). Accurate segmentation of hematoxylin and eosin (H&E)-stained whole slide images (WSIs) has also become an indispensable tool for liver cancer surgery evaluation. For instance, pathologists rely on segmentation results to assess tumor load before surgery, and to monitor treatment responses after the surgery (Huang et al., 2019). Traditionally, pathological HCC segmentation is manually performed by skilled experts, which is quite tedious and time-consuming. Moreover, the segmentation accuracy cannot be ensured since the appearance of HCC lesions varies greatly across patients (Schlageter et al., 2014) (cf. Fig. 1). Therefore, a fully automated and reliable HCC segmentation system that can significantly reduce the burden of pathologists is highly desired.
In recent years, deep learning techniques have exhibited their power in representation learning, and they have also been widely applied in pathological image analysis tasks (Madabhushi, Lee, Komura, Ishikawa, 2018, Xing, Xie, Su, Liu, Yang, 2017). Nevertheless, one of the problems is that, as an H&E-stained WSI is usually in the gigapixel range, e.g., 50,00050,000 pixels, it is infeasible to directly input the whole image into a deep learning model. A common solution (Eminaga, Abbas, Kunder, Loening, Shen, Brooks, Langlotz, Rubin, Huang, Chung, Tsai, Chow, Juang, Tsai, Lin, Wang, 2019, Aziz, Kanazawa, Murakami, Kimura, Yamaguchi, Kiyuna, Yamashita, Saito, Ishikawa, Kobayashi, et al., 2015) is to crop the image into small patches, where each patch is assigned with a binary label, representing whether the given patch contains tumorous regions or not. If a patch is classified as tumorous, all the pixels in the whole patch are marked as tumorous. We refer to such a patch-classification based segmentation method as patch-wise segmentation. There are clear drawbacks to such methods. First, since patches at the border of a tumor region usually contain both tumor and nontumor pixels, assigning the same label to the whole patch leads to very coarse and irregular segmentation results. Such coarse segmentation results can also cause large over- or underestimation of the true tumor size depending on how the patch label is determined, especially for small tumors. Tumor size is an important clinical factor for cancer staging. Second, using a single label for a patch containing both tumor and normal subregions may confuse the network training, which makes the network more likely to produce false positives and false negatives. To address these problems, a pixel-wise HCC segmentation method is proposed in this work, in which we use pixel-wise labels to replace traditional patch-wise labels for more accurate supervision. To the best of our knowledge, this is the first study to perform pixel-wise HCC segmentation of H&E-stained WSIs. As one of the key contributions, this improvement helps us achieve the second place in the MICCAI 2019 Pathology AI Platform (PAIP) challenge.
To further improve the recognition capacity of the network, we adopt a multi-task learning strategy, which is capable of extracting more distinctive and more general image features for both normal and tumorous regions. By leveraging the task-specific information contained in the individual task branches, multi-task learning tends to find the most representative features shared by different tasks (different feature spaces), thus also significantly reducing the risk of overfitting. Many studies have adopted multi-task learning (Guo, Liu, Ni, Wang, Su, Guo, Wang, Jiang, Qian, 2019, Tao, Guo, Zhu, Chen, Zhang, Yang, Liu, Takahama, Kurose, Mukuta, Abe, Fukayama, Yoshizawa, Kitagawa, Harada, 2019, Mehta, Mercan, Bartlett, Weaver, Elmore, Shapiro, 2018), in which they usually first use a classifier to identify the tumorous patches and then segment these patches. Although such methods converge quickly, their drawbacks are also obvious. First, the segmentation performance highly depends on the classifier. An inaccurate classification model will severely affect the segmentation performance. The second defect is the class imbalance problem. Since most normal patches are filtered out by the classification model, the segmentation network may fail to learn the representative features of normal regions. Consequently, these normal regions cannot be correctly recognized during the inference process, thus leading to poor performance.
To address these problems, we propose a novel hybrid architecture that comprises a shared encoder and three task-specific branches. The first branch performs pixel-wise segmentation of all the tumorous patches. Different from previous studies, these tumorous patches are cropped from the original WSIs rather than those selected by a classification model. In this way, all the tumorous regions are preserved for training. The second branch is also a pixel-wise segmentation network but takes as input all the image patches, including both tumorous and normal patches. With sufficient training, the segmentation networks can accurately recognize tumorous and normal regions. The third branch implements a classification model, which is just an auxiliary network for extracting general features.
Another key strategy leveraged in this work is model ensembling, which aims to aggregate features from different separately trained base models that are trained separately. This technique, which has also been adopted in medical image analysis, can significantly improve the generalization ability of the networks (Opitz and Maclin, 1999), which has also been adopted in medical image analysis (Kamnitsas, Bai, Ferrante, McDonagh, Sinclair, Pawlowski, Rajchl, Lee, Kainz, Rueckert, et al., 2017, Pimkin, Makarchuk, Kondratenko, Pisov, Krivov, Belyaev, 2018, Qaiser, Tsang, Epstein, Rajpoot, 2017, Tang, Liang, Yan, Zhang, Coppola, Sun, 2019). Different from these methods, we specifically design two complementary networks leveraging selective kernel modules (SKMs) (Li et al., 2019a) and the concurrent spatial and channel squeeze-and-excitation modules (scSEMs) (Roy et al., 2018) in the model ensemble. The SKMs can adaptively learn multi-scale feature representations using different sizes of kernels, while the scSEMs are capable of capturing both spatial features and channel-wise features. This is also the first work that tries to embed SKMs and scSEMs in the same deep learning network. Leveraging model ensembling, our method outperforms the first place model of the 2019 PAIP challenge.
Our contributions are summarized as follows:
- •
This is a pioneering study on pixel-wise HCC segmentation of large H&E-stained liver WSIs, which presents significant advantages over traditional patch-wise segmentation methods.
- •
Based on the idea of multi-task learning, we propose a novel hybrid architecture for pixel-wise HCC segmentation of H&E-stained WSIs, which directly performs segmentation of WSI patches and no longer relies on a classification model.
- •
Based on the idea of model ensembling, two complementary models based on SKMs and scSEMs are designed, which can effectively extract feature representations from different spaces and scales.
- •
Our proposed method achieves state-of-the-art performance on a large H&E-stained liver WSI dataset and two other pathological datasets, which verifies its effectiveness and generalization ability.
The remainder of this paper is organized as follows. Section 2 briefly surveys recent studies on pathological image segmentation. Section 3 explains the details of our proposed method, which has a three-branch network design and an ensemble learning scheme. Experimental results that validate the proposed model are presented and discussed in Section 4. Finally, the paper is concluded in Section 5.
Section snippets
Patch-wise and pixel-wise HCC segmentation
Convolutional neural networks (CNNs), a popular class of deep learning methods, have been widely used in pathological medical image analysis (Campanella, Hanna, Geneslaw, Miraflor, Silva, Busam, Brogi, Reuter, Klimstra, Fuchs, 2019, Kather, Pearson, Halama, Jäger, Krause, Loosen, Marx, Boor, Tacke, Neumann, et al., 2019, Courtiol, Maussion, Moarii, Pronier, Pilcer, Sefta, Manceron, Toldo, Zaslavskiy, Le Stang, et al., 2019, Coudray, Ocampo, Sakellaropoulos, Narula, Snuderl, Fenyö, Moreira,
Methods
In this section, we present the details of our hybrid neural network method for HCC segmentation. We first explain the network architecture and the design of its encoding and decoding blocks. We then present the loss function used in each subnetwork. Finally, we explain our ensemble scheme with two complementary base models to further improve the segmentation accuracy.
Experimental results and discussion
In this section, we first summarize the three publicly available datasets that are used to validate the proposed method. We then show detailed segmentation results comparing our method with alternative strategies and demonstrate the benefits of the proposed improvements.
Conclusion
In this study, we propose a novel multi-task learning architecture with three task-specific branches for pixel-wise HCC segmentation of large H&E-stained WSIs. An ensemble learning strategy combining two complementary base models is leveraged to further boost the segmentation accuracy. Extensive experiments on a large HCC segmentation dataset and two other pathological datasets have demonstrated that our proposed method produces superior segmentation results compared to other state-of-the-art
CRediT authorship contribution statement
Xiyue Wang: Conceptualization, Methodology, Software, Writing - original draft. Yuqi Fang: Conceptualization, Methodology, Software, Writing - original draft. Sen Yang: Conceptualization, Methodology, Software, Writing - original draft. Delong Zhu: Visualization, Validation. Minghui Wang: Visualization, Validation. Jing Zhang: Supervision, Funding acquisition. Kai-yu Tong: Formal analysis, Investigation. Xiao Han: Supervision, Writing - original draft, Project administration.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This research was funded by the National Natural Science Foundation of China under grant number 61571314. We specifically express our gratitude to the PAIP organizer for their released dataset.
References (78)
- et al.
A cascade-learning approach for automated segmentation of tumour epithelium in colorectal cancer
Expert Syst. Appl.
(2019) - et al.
Mild-Net: Minimal information loss dilated network for gland instance segmentation in colon histology images
Med. Image Anal.
(2019) - et al.
Efficient multi-scale 3d CNN with fully connected CRF for accurate brain lesion segmentation
Med. Image Anal.
(2017) - et al.
Machine learning methods for histopathological image analysis
Comput. Struct. Biotechnol. J.
(2018) - et al.
Fully convolutional network ensembles for white matter hyperintensities segmentation in mr images
NeuroImage
(2018) - et al.
Automated analysis and diagnosis of skin melanoma on whole slide histopathological images
Pattern Recognit.
(2015) - et al.
Fast and accurate tumor segmentation of histology images using persistent homology and deep convolutional features
Med. Image Anal.
(2019) - Tao, S., Guo, Y., Zhu, C., Chen, H., Zhang, Y., Yang, J., Liu, J., 2019. Highly efficient follicular segmentation in...
- Yuan, Y., Wang, J., 2018. OCNet: Object context network for scene parsing. arXiv...
- Agarwalla, A., Shaban, M., Rajpoot, N. M., 2017. Representation-aggregation networks for segmentation of...
AggNet: Deep learning from crowds for mitosis detection in breast cancer histology images
IEEE Trans. Med. Imaging
Glandular morphometrics for objective grading of colorectal adenocarcinoma histology images
Sci. Rep.
Enhancing automatic classification of hepatocellular carcinoma images through image masking, tissue changes and trabecular features
J. Pathol. Inf.
SegNet: A deep convolutional encoder-decoder architecture for image segmentation
IEEE Trans. Pattern Anal. Mach.Intell.
Chest pathology detection using deep learning with non-medical training
2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI)
Clinical-grade computational pathology using weakly supervised deep learning on whole slide images
Nat. Med.
Encoder-decoder with atrous separable convolution for semantic image segmentation
Proceedings of the European Conference on Computer Vision (ECCV)
Hippocampus segmentation through multi-view ensemble convnets
2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017)
Classification and mutation prediction from non–small cell lung cancer histopathology images using deep learning
Nat. Med.
Deep learning-based classification of mesothelioma improves prediction of patient outcome
Nat. Med.
High-throughput adaptive sampling for whole-slide histopathology image analysis (HASHI) via convolutional neural networks: application to invasive breast cancer detection
PloS One
Automatic segmentation of histopathological slides of renal tissue using deep learning
Medical Imaging 2018: Digital Pathology
Reinforced auto-zoom net: towards accurate and fast breast cancer segmentation in whole-slide images
Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support
Selective feature aggregation network with area-boundary constraints for polyp segmentation
International Conference on Medical Image Computing and Computer-Assisted Intervention
A fast and refined cancer regions segmentation framework in whole-slide breast pathological images
Sci. Rep.
DME-Net: Diabetic macular edema grading by auxiliary task learning
International Conference on Medical Image Computing and Computer-Assisted Intervention
Squeeze-and-excitation networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Automatic HCC detection using convolutional network with multi-magnification input images
2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)
Semantic segmentation for prostate cancer grading by convolutional neural networks
Medical Imaging 2018: Digital Pathology
Ensembles of multiple models and architectures for robust brain tumour segmentation
International MICCAI Brainlesion Workshop
Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer
Nat. Med.
Context aware lung cancer annotation in whole slide images using fully convolutional neural networks
International Conference on Image Analysis and Recognition
Deep learning for classification of colorectal polyps on whole-slide images
J. Pathol. Inf.
Selective kernel networks
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Path aggregation network for instance segmentation
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
Cited by (37)
Self-supervised multi-magnification feature enhancement for segmentation of hepatocellular carcinoma region in pathological images
2024, Engineering Applications of Artificial IntelligenceA joint Multi-decoder Dual-attention U-Net framework for tumor segmentation in Whole Slide Images
2023, Journal of King Saud University - Computer and Information SciencesDHUnet: Dual-branch hierarchical global–local fusion network for whole slide image segmentation
2023, Biomedical Signal Processing and ControlArtificial Intelligence Applications in Hepatology
2023, Clinical Gastroenterology and Hepatology
- 1
Authors contributed equally to this work.