Abstract
Microsatellite instability (MSI) is a crucial biomarker to clinical immunotherapy in gastrointestinal cancer, while additional immunohistochemical or genetic tests for MSI are generally missing due to lack of medical resources. Deep learning has achieved promising performance in detecting MSI from hematoxylin and eosin (H &E) stained histopathology slides. However, these methods are primarily based on patch-supervised slide-label models and then aggregate patch-level results into the slides-level result, resulting unstable prediction due to noisy patches and aggregation ways.
In this paper, we propose a joint region-attention and multi-scale transformer (RAMST) network for microsatellite instability detection from whole slide images in gastrointestinal cancer. Specifically, we present a region-attention mechanism and a feature weight uniform sampling (FWUS) method to learn a representative subset of image patches from whole slide images. Moreover, we introduce the transformer architecture to fuse the multi-scale histopathology features consisting of patch-level features with region-level features to characterize the whole slide images for slide-level MSI detection. Compared to the existing MSI detection methods, the proposed RAMST shows the best performances on the colorectal and stomach cancer dataset from The Cancer Genome Atlas (TCGA) and provides an effective features representation learning method for WSI-label tasks.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bhargava, R., Madabhushi, A.: Emerging themes in image informatics and molecular analysis for digital pathology. Annu. Rev. Biomed. Eng. 18, 387 (2016)
Bilal, M., et al.: Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study. Lancet Digital Health 3(12), e763–e772 (2021)
Boland, C.R., Goel, A.: Microsatellite instability in colorectal cancer. Gastroenterology 138(6), 2073–2087 (2010)
Cao, R., et al.: Development and interpretation of a pathomics-based model for the prediction of microsatellite instability in colorectal cancer. Theranostics 10(24), 11080 (2020)
Dosovitskiy, A., et al.: An image is worth 16 \(\times \) 16 words: transformers for image recognition at scale. In: International Conference on Learning Representations, ICLR (2021)
Echle, A., et al.: Clinical-grade detection of microsatellite instability in colorectal tumors by deep learning. Gastroenterology 159(4), 1406–1416 (2020)
Echle, A., et al.: Deep learning for the detection of microsatellite instability from histology images in colorectal cancer: a systematic literature review. ImmunoInformatics 3–4, 100008 (2021)
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000–16009 (2022)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Kather, J.N., et al.: Deep learning can predict microsatellite instability directly from histology in gastrointestinal cancer. Nat. Med. 25(7), 1054–1056 (2019)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 25 (2012)
Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Schirris, Y., Gavves, E., Nederlof, I., Horlings, H.M., Teuwen, J.: DeepSMILE: self-supervised heterogeneity-aware multiple instance learning for DNA damage response defect classification directly from H &E whole-slide images. arXiv preprint arXiv:2107.09405 (2021)
Schmauch, B., et al.: A deep learning model to predict RNA-Seq expression of tumours from whole slide images. Nat. Commun. 11(1), 1–15 (2020)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)
Wang, W., et al.: Pyramid vision transformer: a versatile backbone for dense prediction without convolutions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568–578 (2021)
Yamashita, R., Long, J., Banda, S., Shen, J., Rubin, D.L.: Learning domain-agnostic visual representation for computational pathology using medically-irrelevant style transfer augmentation. IEEE Trans. Med. Imaging 40(12), 3945–3954 (2021)
Yamashita, R., et al.: Deep learning model for the prediction of microsatellite instability in colorectal cancer: a diagnostic study. Lancet Oncol. 22(1), 132–141 (2021)
Acknoledgements
The research is supported by the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDA16021400), and the NSFC Projects Grants (61932018, 62072441 and 62072280).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lv, Z., Yan, R., Lin, Y., Wang, Y., Zhang, F. (2022). Joint Region-Attention and Multi-scale Transformer for Microsatellite Instability Detection from Whole Slide Images in Gastrointestinal Cancer. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. Lecture Notes in Computer Science, vol 13432. Springer, Cham. https://doi.org/10.1007/978-3-031-16434-7_29
Download citation
DOI: https://doi.org/10.1007/978-3-031-16434-7_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-16433-0
Online ISBN: 978-3-031-16434-7
eBook Packages: Computer ScienceComputer Science (R0)
