Meta multi-task nuclei segmentation with fewer training samples

Highlights

• We proposed a meta multi-task learning scheme (Meta-MTL) for nuclei segmentation with better domain adaptation and less data dependency.

• In the contour-aware multi-task learning model, we proposed a feature fusion and interaction block (FFIB) for features communication between two parallel tasks.

• Our proposed model achieves comparable performances with state-of-the-art models with fewer training samples. And the proposed model performs well on domain adaptation and domain generalization.

Abstract

Cells/nuclei deliver massive information of microenvironment. An automatic nuclei segmentation approach can reduce pathologists’ workload and allow precise of the microenvironment for biological and clinical researches. Existing deep learning models have achieved outstanding performance under the supervision of a large amount of labeled data. However, when data from the unseen domain comes, we still have to prepare a certain degree of manual annotations for training for each domain. Unfortunately, obtaining histopathological annotations is extremely difficult. It is high expertise-dependent and time-consuming. In this paper, we attempt to build a generalized nuclei segmentation model with less data dependency and more generalizability. To this end, we propose a meta multi-task learning (Meta-MTL) model for nuclei segmentation which requires fewer training samples. A model-agnostic meta-learning is applied as the outer optimization algorithm for the segmentation model. We introduce a contour-aware multi-task learning model as the inner model. A feature fusion and interaction block (FFIB) is proposed to allow feature communication across both tasks. Extensive experiments prove that our proposed Meta-MTL model can improve the model generalization and obtain a comparable performance with state-of-the-art models with fewer training samples. Our model can also perform fast adaptation on the unseen domain with only a few manual annotations. Code is available at https://github.com/ChuHan89/Meta-MTL4NucleiSegmentation

Introduction

Computational pathology has been one of the hottest concepts in biological and clinical research in recent years. As the basic unit of the life system, cells/nuclei microenvironment delivers massive information. Many research works attempt to explore both explicit and implicit patterns of nuclei for disease diagnosis (Yamashita et al., 2020), prognostic prediction (Skrede, De Raedt, Kleppe, Hveem, Liestøl, Maddison, Askautrud, Pradhan, Nesheim, Albregtsen, et al., 2020, Lu, Koyuncu, Corredor, Prasanna, Leo, Wang, Janowczyk, Bera, Lewis, Velcheti, et al., 2020), tissue phenotyping (Javed et al., 2020) and etc. However, it is impractical to segment millions of nuclei manually due to the huge resolution of whole slide images. Therefore, it is valuable to construct a fully automatic model for nuclei segmentation.

Convolutional neural network has demonstrated the superiority in medical image segmentation (Moeskops, Wolterink, van der Velden, Gilhuijs, Leiner, Viergever, Išgum, 2016, Zhou, Siddiquee, Tajbakhsh, Liang, 2018, Chan, Hosseini, Rowsell, Plataniotis, Damaskinos, 2019). Meanwhile, Various CNN-based models (Graham, Vu, Raza, Azam, Tsang, Kwak, Rajpoot, 2019, Liu, Zhang, Song, Zhang, Zhang, O’Donnell, Cai, 2019) have been developed for nuclei segmentation and achieved outstanding performance under full supervision. However, most of the existing models are data-dependent. They have to optimize the neural networks for each individual dataset. When data from the different domains come, we still have to prepare a new dataset for the new training. Unfortunately, manual annotations are expensive, especially for histopathology images. It requires high expertise and is extremely tedious and time-consuming. Moreover, the neural network is often trained on top of a massive amount of training data with manual labels. Otherwise, it may drop into the overfitting problem. It is impractical to collect a large amount of annotated data for every single task. Therefore, it will be convenient and effective if fewer training samples can be used for model training while also greatly relieving the tremendous annotation workload of pathologists.

There have been a few attempts to develop neural network models with fewer training samples. Transfer learning (Pan and Yang, 2009) was proposed to address the lack of labeled data problem by transferring the data distribution from the source domain to the target domain. This kind of model fine-tuning strategy also relies on a certain degree of training data of the target domain. Recently, multi-task learning (Chen, Qi, Yu, Heng, 2016, Zhao, Chen, Li, Yu, Yao, Yan, Wang, Liu, Liang, Han, 2020) has been widely used for medical image segmentation by setting up auxiliary tasks from the same domain to improve the target task. This strategy can effectively increase model generalization. Meta-learning (Naik, Mammone, 1992, Hospedales, Antoniou, Micaelli, Storkey) aims to optimize the model itself instead of optimizing the final goal of the model. The learning to learn spirit is much closer to how humans learn the world.

In this paper, we aim to build a generalized model for nuclei segmentation with fewer training samples. Consequently, we absorb the spirits of meta-learning as well as multi-task learning and associate them together as a Meta Multi-Task Learning (Meta-MTL) scheme. To improve the model generalizability, we apply a model-agnostic meta-learning approach (Finn et al., 2017) as the outer optimization algorithm for the complete model. Then we construct a contour-aware multi-task learning model as the inner model for nuclei segmentation. There are two tasks in the inner model, a segmentation task and a contour detection task. These two parallel tasks can complement each other and let the neural network focus on more generalized features instead of task-specific ones. We construct an intermediate component with several feature fusion and interaction blocks (FFIB) to support feature interactions and feature sharing across two tasks. A feature attention strategy is introduced in the decoding phase to further amplify contour information in the segmentation task.

Extensive experiments are conducted to evaluate the effectiveness of our proposed model in three aspects, data dependency, domain adaptation and domain generalization. (1) For each dataset, we progressively deduct the among of training samples randomly with a 20% descending proportion. (2) Given a well-trained model on a specific dataset, we transfer this model to an unseen dataset with fewer training samples. (3) Given a model train on a specific dataset, we directly apply this model to an unseen dataset. Experimental results suggest that our model achieves comparable performances with state-of-the-art nuclei segmentation models with fewer training data. And our model demonstrates a better potential for domain adaptation and domain generalization. With such properties, our proposed model could be an initial segmentation model for the pathologist-in-the-loop labeling strategy with only limited start-up pixel-level labels, to save the annotation efforts. The contributions of this paper are three-fold:

• We propose a novel meta multi-task learning (Meta-MTL) scheme for nuclei segmentation with better domain adaptation and domain generalization abilities as well as less data dependency.

• In the contour-aware multi-task learning model, we propose a feature fusion and interaction block (FFIB) for features communication between two parallel tasks.

• Our proposed model achieves comparable performances with state-of-the-art models with fewer training samples.