Abstract
Cancer survival prediction requires considering not only the biological morphology but also the contextual interactions of tumor and surrounding tissues. The major limitation of previous learning frameworks for whole slide image (WSI) based survival prediction is that the contextual interactions of pathological components (e.g., tumor, stroma, lymphocyte, etc.) lack sufficient representation and quantification. In this paper, we proposed a multi-scope analysis driven Hierarchical Graph Transformer (HGT) to overcome this limitation. Specifically, we first utilize a multi-scope analysis strategy, which leverages an in-slide superpixel and a cross-slide clustering, to mine the spatial and semantic priors of WSIs. Furthermore, based on the extracted spatial prior, a hierarchical graph convolutional network is proposed to progressively learn the topological features of the variant microenvironments ranging from patch-level to tissue-level. In addition, guided by the identified semantic prior, tissue-level features are further aggregated to represent the meaningful pathological components, whose contextual interactions are established and quantified by the designed Transformer-based prediction head. We evaluated the proposed framework on our collected Colorectal Cancer (CRC) cohort and two public cancer cohorts from the TCGA project, i.e., Liver Hepatocellular Carcinoma (LIHC) and Kidney Clear Cell Carcinoma (KIRC). Experimental results demonstrate that our proposed method yields superior performance and richer interpretability compared to the state-of-the-art approaches.
W. Hou and Y. He—Contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
AbdulJabbar, K., et al.: Geospatial immune variability illuminates differential evolution of lung adenocarcinoma. Nat. Med. 26(7), 1054–1062 (2020)
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Süsstrunk, S.: SLIC superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)
Bilal, M., et al.: Development and validation of a weakly supervised deep learning framework to predict the status of molecular pathways and key mutations in colorectal cancer from routine histology images: a retrospective study. Lancet Digit. Health 3(12), e763–e772 (2021)
Chen, R.J., et al.: Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16144–16155 (2022)
Chen, R.J., et al.: Whole slide images are 2D point clouds: context-aware survival prediction using patch-based graph convolutional networks. In: de Bruijne, M., et al. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 339–349. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_33
Diao, J.A., et al.: Human-interpretable image features derived from densely mapped cancer pathology slides predict diverse molecular phenotypes. Nat. Commun. 12(1), 1613 (2021)
Fey, M., Lenssen, J.E.: Fast graph representation learning with PyTorch geometric. In: ICLR Workshop on Representation Learning on Graphs and Manifolds (2019)
Foersch, S., et al.: Multistain deep learning for prediction of prognosis and therapy response in colorectal cancer. Nat. Med. 29, 1–10 (2023)
Guan, Y., et al.: Node-aligned graph convolutional network for whole-slide image representation and classification. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18813–18823 (2022)
Hamilton, W., Ying, Z., Leskovec, J.: Inductive representation learning on large graphs. In: Advances in Neural Information Processing Systems, pp. 1024–1034 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hou, W., et al.: \(\text{H}^2\)-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 933–941 (2022)
Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning, pp. 2127–2136. PMLR (2018)
Kather, J.N., et al.: Predicting survival from colorectal cancer histology slides using deep learning: a retrospective multicenter study. PLoS Med. 16(1), e1002730 (2019)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations (2017). https://openreview.net/forum?id=SJU4ayYgl
Li, B., Li, Y., Eliceiri, K.W.: Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In: Proceedings of the IEEE/CVF Conference On Computer Vision and Pattern Recognition, pp. 14318–14328 (2021)
Liu, P., Fu, B., Ye, F., Yang, R., Ji, L.: DSCA: a dual-stream network with cross-attention on whole-slide image pyramids for cancer prognosis. Expert Syst. Appl. 227, 120280 (2023)
Lu, M.Y., Williamson, D.F., Chen, T.Y., Chen, R.J., Barbieri, M., Mahmood, F.: Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomed. Eng. 5(6), 555–570 (2021)
Shao, Z., Bian, H., Chen, Y., Wang, Y., Zhang, J., Ji, X., et al.: TransMIL: Transformer based correlated multiple instance learning for whole slide image classification. Adv. Neural. Inf. Process. Syst. 34, 2136–2147 (2021)
Sterlacci, W., Vieth, M.: Early colorectal cancer. In: Baatrup, G. (ed.) Multidisciplinary Treatment of Colorectal Cancer, pp. 263–277. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-58846-5_28
Sung, H., et al.: Global cancer statistics 2020: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA Cancer J. Clin. 71(3), 209–249 (2021). https://doi.org/10.3322/caac.21660
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems 30 (2017)
Wang, P., Li, Y., Reddy, C.K.: Machine learning for survival analysis: a survey. ACM Comput. Surv. 51(6), 3214306 (2019). https://doi.org/10.1145/3214306
Wang, X., et al.: RetCCL: clustering-guided contrastive learning for whole-slide image retrieval. Med. Image Anal. 83, 102645 (2023)
Wang, X., et al.: SCL-WC: cross-slide contrastive learning for weakly-supervised whole-slide image classification. In: Thirty-Sixth Conference on Neural Information Processing Systems (2022)
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R.R., Smola, A.J.: Deep sets. In: Advances in Neural Information Processing Systems 30 (2017)
Acknowledgement
This work was supported by Ministry of Science and Technology of the People’s Republic of China (2021ZD0201900)(2021ZD0201903).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hou, W. et al. (2023). Multi-scope Analysis Driven Hierarchical Graph Transformer for Whole Slide Image Based Cancer Survival Prediction. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14225. Springer, Cham. https://doi.org/10.1007/978-3-031-43987-2_72
Download citation
DOI: https://doi.org/10.1007/978-3-031-43987-2_72
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43986-5
Online ISBN: 978-3-031-43987-2
eBook Packages: Computer ScienceComputer Science (R0)
