A hybrid network for automatic hepatocellular carcinoma segmentation in H&E-stained whole slide images

Highlights

• This is one of the pioneer studies on pixel-wise HCC segmentation of large H&E-stained WSIs, which presents significant advantages over the traditional patch-wise segmentation methods.

• Based on the idea of multi-task learning, we propose a novel hybrid architecture for pixel-wise HCC segmentation of H&E-stained WSIs, which directly performs segmentation on WSI patches and no longer relies on a classification model.

• Based on the idea of model ensemble, two complementary models based on SKM and scSEM are designed, which can effectively extract feature representations from different spaces and scales.

• Our proposed method achieves the state-of-the-art performance on a large H&E-stained WSI dataset and other two pathological datasets, which verifies its effectiveness and generalization ability.

Abstract

Hepatocellular carcinoma (HCC), as the most common type of primary malignant liver cancer, has become a leading cause of cancer deaths in recent years. Accurate segmentation of HCC lesions is critical for tumor load assessment, surgery planning, and postoperative examination. As the appearance of HCC lesions varies greatly across patients, traditional manual segmentation is a very tedious and time-consuming process, the accuracy of which is also difficult to ensure. Therefore, a fully automated and reliable HCC segmentation system is in high demand. In this work, we present a novel hybrid neural network based on multi-task learning and ensemble learning techniques for accurate HCC segmentation of hematoxylin and eosin (H&E)-stained whole slide images (WSIs). First, three task-specific branches are integrated to enlarge the feature space, based on which the network is able to learn more general features and thus reduce the risk of overfitting. Second, an ensemble learning scheme is leveraged to perform feature aggregation, in which selective kernel modules (SKMs) and spatial and channel-wise squeeze-and-excitation modules (scSEMs) are adopted for capturing the features from different spaces and scales. Our proposed method achieves state-of-the-art performance on three publicly available datasets, with segmentation accuracies of 0.797, 0.923, and 0.765 in the PAIP, CRAG, and UHCMC&CWRU datasets, respectively, which demonstrates its effectiveness in addressing the HCC segmentation problem. To the best of our knowledge, this is also the first work on the pixel-wise HCC segmentation of H&E-stained WSIs.

Introduction

Liver cancer is the fourth leading cause of cancer-related death worldwide, and it is estimated that more than one million patients will die from liver cancer in 2030 (Longo, 2019). As the most common type of primary malignant liver cancer, hepatocellular carcinoma (HCC) occurs most often in people with cirrhosis, which is a leading cause of liver cancer. Although some imaging techniques, such as magnetic resonance imaging, computed tomography, and ultrasound, have made significant improvements in liver cancer detection, pathology imaging that is based on digitized specimen slides is still the gold standard for cancer diagnosis (Lu and Mandal, 2015). Accurate segmentation of hematoxylin and eosin (H&E)-stained whole slide images (WSIs) has also become an indispensable tool for liver cancer surgery evaluation. For instance, pathologists rely on segmentation results to assess tumor load before surgery, and to monitor treatment responses after the surgery (Huang et al., 2019). Traditionally, pathological HCC segmentation is manually performed by skilled experts, which is quite tedious and time-consuming. Moreover, the segmentation accuracy cannot be ensured since the appearance of HCC lesions varies greatly across patients (Schlageter et al., 2014) (cf. Fig. 1). Therefore, a fully automated and reliable HCC segmentation system that can significantly reduce the burden of pathologists is highly desired.

In recent years, deep learning techniques have exhibited their power in representation learning, and they have also been widely applied in pathological image analysis tasks (Madabhushi, Lee, Komura, Ishikawa, 2018, Xing, Xie, Su, Liu, Yang, 2017). Nevertheless, one of the problems is that, as an H&E-stained WSI is usually in the gigapixel range, e.g., 50,000$\times$50,000 pixels, it is infeasible to directly input the whole image into a deep learning model. A common solution (Eminaga, Abbas, Kunder, Loening, Shen, Brooks, Langlotz, Rubin, Huang, Chung, Tsai, Chow, Juang, Tsai, Lin, Wang, 2019, Aziz, Kanazawa, Murakami, Kimura, Yamaguchi, Kiyuna, Yamashita, Saito, Ishikawa, Kobayashi, et al., 2015) is to crop the image into small patches, where each patch is assigned with a binary label, representing whether the given patch contains tumorous regions or not. If a patch is classified as tumorous, all the pixels in the whole patch are marked as tumorous. We refer to such a patch-classification based segmentation method as patch-wise segmentation. There are clear drawbacks to such methods. First, since patches at the border of a tumor region usually contain both tumor and nontumor pixels, assigning the same label to the whole patch leads to very coarse and irregular segmentation results. Such coarse segmentation results can also cause large over- or underestimation of the true tumor size depending on how the patch label is determined, especially for small tumors. Tumor size is an important clinical factor for cancer staging. Second, using a single label for a patch containing both tumor and normal subregions may confuse the network training, which makes the network more likely to produce false positives and false negatives. To address these problems, a pixel-wise HCC segmentation method is proposed in this work, in which we use pixel-wise labels to replace traditional patch-wise labels for more accurate supervision. To the best of our knowledge, this is the first study to perform pixel-wise HCC segmentation of H&E-stained WSIs. As one of the key contributions, this improvement helps us achieve the second place in the MICCAI 2019 Pathology AI Platform (PAIP) challenge.

To further improve the recognition capacity of the network, we adopt a multi-task learning strategy, which is capable of extracting more distinctive and more general image features for both normal and tumorous regions. By leveraging the task-specific information contained in the individual task branches, multi-task learning tends to find the most representative features shared by different tasks (different feature spaces), thus also significantly reducing the risk of overfitting. Many studies have adopted multi-task learning (Guo, Liu, Ni, Wang, Su, Guo, Wang, Jiang, Qian, 2019, Tao, Guo, Zhu, Chen, Zhang, Yang, Liu, Takahama, Kurose, Mukuta, Abe, Fukayama, Yoshizawa, Kitagawa, Harada, 2019, Mehta, Mercan, Bartlett, Weaver, Elmore, Shapiro, 2018), in which they usually first use a classifier to identify the tumorous patches and then segment these patches. Although such methods converge quickly, their drawbacks are also obvious. First, the segmentation performance highly depends on the classifier. An inaccurate classification model will severely affect the segmentation performance. The second defect is the class imbalance problem. Since most normal patches are filtered out by the classification model, the segmentation network may fail to learn the representative features of normal regions. Consequently, these normal regions cannot be correctly recognized during the inference process, thus leading to poor performance.

To address these problems, we propose a novel hybrid architecture that comprises a shared encoder and three task-specific branches. The first branch performs pixel-wise segmentation of all the tumorous patches. Different from previous studies, these tumorous patches are cropped from the original WSIs rather than those selected by a classification model. In this way, all the tumorous regions are preserved for training. The second branch is also a pixel-wise segmentation network but takes as input all the image patches, including both tumorous and normal patches. With sufficient training, the segmentation networks can accurately recognize tumorous and normal regions. The third branch implements a classification model, which is just an auxiliary network for extracting general features.

Another key strategy leveraged in this work is model ensembling, which aims to aggregate features from different separately trained base models that are trained separately. This technique, which has also been adopted in medical image analysis, can significantly improve the generalization ability of the networks (Opitz and Maclin, 1999), which has also been adopted in medical image analysis (Kamnitsas, Bai, Ferrante, McDonagh, Sinclair, Pawlowski, Rajchl, Lee, Kainz, Rueckert, et al., 2017, Pimkin, Makarchuk, Kondratenko, Pisov, Krivov, Belyaev, 2018, Qaiser, Tsang, Epstein, Rajpoot, 2017, Tang, Liang, Yan, Zhang, Coppola, Sun, 2019). Different from these methods, we specifically design two complementary networks leveraging selective kernel modules (SKMs) (Li et al., 2019a) and the concurrent spatial and channel squeeze-and-excitation modules (scSEMs) (Roy et al., 2018) in the model ensemble. The SKMs can adaptively learn multi-scale feature representations using different sizes of kernels, while the scSEMs are capable of capturing both spatial features and channel-wise features. This is also the first work that tries to embed SKMs and scSEMs in the same deep learning network. Leveraging model ensembling, our method outperforms the first place model of the 2019 PAIP challenge.

Our contributions are summarized as follows:

• This is a pioneering study on pixel-wise HCC segmentation of large H&E-stained liver WSIs, which presents significant advantages over traditional patch-wise segmentation methods.

• Based on the idea of multi-task learning, we propose a novel hybrid architecture for pixel-wise HCC segmentation of H&E-stained WSIs, which directly performs segmentation of WSI patches and no longer relies on a classification model.

• Based on the idea of model ensembling, two complementary models based on SKMs and scSEMs are designed, which can effectively extract feature representations from different spaces and scales.

• Our proposed method achieves state-of-the-art performance on a large H&E-stained liver WSI dataset and two other pathological datasets, which verifies its effectiveness and generalization ability.

The remainder of this paper is organized as follows. Section 2 briefly surveys recent studies on pathological image segmentation. Section 3 explains the details of our proposed method, which has a three-branch network design and an ensemble learning scheme. Experimental results that validate the proposed model are presented and discussed in Section 4. Finally, the paper is concluded in Section 5.