Abstract
Recently, weakly supervised histology image segmentation has received increasingly more attentions. Most solutions utilize a convolutional neural network (CNN) as a classifier and treat the generated class activation map (CAM) as a pseudo annotation, based on which a segmentation network is trained in a supervised manner. This pipeline suffers from two disadvantages. First, the CNN classifier may fail to generate the high-quality CAM that highlights the exact and integral target, resulting in incomplete activation and blurred boundaries. Second, it splits the original problem into two, leading to a sub-optimal solution and low efficiency. To address both issues, we propose a Transformer-based weakly supervised segmentation (TransWS) method for histology images. TransWS is composed of a classification branch and a segmentation branch. The former learns semantic information from image-level annotations and uses CAM to generate pseudo pixel-level annotations. The latter performs the class-agnostic segmentation (CAS), i.e., binary segmentation, under the supervision of pseudo annotations. The semantic information and foreground region are combined to generate the final segmentation result. Comparing to CNN, Transformer is superior in modeling long-term dependencies and can generate more integral and accurate CAMs. More important, both branches in our TransWS can be jointly optimized in an end-to-end manner. We evaluated TransWS on the benchmark GlaS and Camelyon16-P512 datasets. Our results suggest that TransWS outperforms other weakly supervised segmentation competitors, setting a new state of the art.
S. Zhang and J. Zhang—Co-first authors.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bejnordi, B.E., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph nodemetastases in women with breast cancer. Jama 318(22), 2199–2210 (2017)
Belharbi, S., Rony, J., Dolz, J., Ayed, I.B., McCaffrey, L., Granger, E.: Deep interpretable classification and weakly-supervised segmentation of histology images via max-min uncertainty. IEEE Trans. Med. Imaging 41 (2021)
Chen, H., Qi, X., Yu, L., Heng, P.A.: Dcan: deep contour-aware networks for accurate gland segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2487–2496 (2016)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Durand, T., Mordan, T., Thome, N., Cord, M.: Wildcat: weakly supervised learning of deep convnets for image classification, pointwise localization and segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 642–651 (2017)
Gu, W., Wang, S., Zhao, S., Wan, L., Zhu, Z.: Histosegrest: a weakly supervised learning method for histopathology image segmentation. In: 2022 the 5th International Conference on Image and Graphics Processing (ICIGP), pp. 189–195 (2022)
Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning, pp. 2127–2136. PMLR (2018)
Kervadec, H., Dolz, J., Tang, M., Granger, E., Boykov, Y., Ayed, I.B.: Constrained-CNN losses for weakly supervised segmentation. Med. Image Anal. 54, 88–99 (2019)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)
Oquab, M., Bottou, L., Laptev, I., Sivic, J.: Is object localization for free?-weakly-supervised learning with convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 685–694 (2015)
Pinheiro, P.O., Collobert, R.: From image-level to pixel-level labeling with convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1713–1721 (2015)
Qaiser, T., et al.: Fast and accurate tumor segmentation of histology images using persistent homology and deep convolutional features. Med. Image Anal. 55, 1–14 (2019)
Qian, Z., et al.: Transformer based multiple instance learning for weakly supervised histopathology image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (2022)
Rony, J., Belharbi, S., Dolz, J., Ayed, I.B., McCaffrey, L., Granger, E.: Deep weakly-supervised learning methods for classification and localization in histology images: a survey. arXiv preprint arXiv:1909.03354 (2019)
Ru, L., Zhan, Y., Yu, B., Du, B.: Learning affinity from attention: end-to-end weakly-supervised semantic segmentation with transformers. arXiv preprint arXiv:2203.02664 (2022)
Selvaraju, R.R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., Batra, D.: Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 618–626 (2017)
Sirinukunwattana, K., Snead, D.R., Rajpoot, N.M.: A stochastic polygons model for glandular structures in colon histology images. IEEE Trans. Med. Imaging 34(11), 2366–2378 (2015)
Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: transformer for semantic segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7262–7272 (2021)
Sun, C., Paluri, M., Collobert, R., Nevatia, R., Bourdev, L.: Pronet: learning to propose object-specific boxes for cascaded neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3485–3493 (2016)
Vaswani, A., et al.: Attention is all you need. Adv. Neural Inform. Process. Syst. 30 (2017)
Wei, Y., Feng, J., Liang, X., Cheng, M.M., Zhao, Y., Yan, S.: Object region mining with adversarial erasing: a simple classification to semantic segmentation approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1568–1576 (2017)
Zhao, B., et al.: Triple u-net: hematoxylin-aware nuclei segmentation with progressive dense feature aggregation. Med. Image Anal. 65, 101786 (2020)
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)
Acknowledgement
This work was supported in part by the Natural Science Foundation of Ningbo City, China, under Grant 2021J052, in part by the Ningbo Clinical Research Center for Medical Imaging under Grant 2021L003, and in part by the Open Project of Ningbo Clinical Research Center for Medical Imaging under Grant 2022LYKFZD06.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, S., Zhang, J., Xia, Y. (2022). TransWS: Transformer-Based Weakly Supervised Histology Image Segmentation. In: Lian, C., Cao, X., Rekik, I., Xu, X., Cui, Z. (eds) Machine Learning in Medical Imaging. MLMI 2022. Lecture Notes in Computer Science, vol 13583. Springer, Cham. https://doi.org/10.1007/978-3-031-21014-3_38
Download citation
DOI: https://doi.org/10.1007/978-3-031-21014-3_38
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21013-6
Online ISBN: 978-3-031-21014-3
eBook Packages: Computer ScienceComputer Science (R0)