Abstract
The tumor microenvironment is a complex ecosystem consisting of various immune and stromal cells in addition to neoplastic cells. The spatial interaction and organization of these cells play a critical role in tumor progression. Single-cell analysis of histopathology images offers an intrinsic advantage over traditional patch-based approach by providing fine-grained cellular information. However, existing studies do not perform explicit cell classification, and therefore still suffer from limited interpretability and lack biological relevance, which may negatively affect the performance for clinical outcome prediction. To address these challenges, we propose a cell-level contextual learning approach to explicitly capture the major cell types and their spatial interaction in the tumor microenvironment. To do this, we first segmented and classified each cell into tumor cells, lymphocytes, fibroblasts, macrophages, neutrophils, and other nonmalignant cells on histopathology images. Given this single-cell map, we constructed a graph and trained a graph attention network to learn the cell-level contextual features for survival prediction. Extensive experiments demonstrate that our model consistently outperform existing patch-based and cell graph-based approaches in two independent datasets. Further, we used the feature attribution method to discover distinct spatial patterns that are associated with prognosis, leading to biologically meaningful and interpretable results.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Li, Z., et al.: Development and validation of a machine learning model for detection and classification of tertiary lymphoid structures in gastrointestinal cancers. JAMA Netw. Open 6, e2252553–e2252553 (2023)
Javed, S., et al.: Cellular community detection for tissue phenotyping in colorectal cancer histology images. Med. Image Anal. 63, 101696 (2020)
Chang, J.-R., Lee, C.-Y., Chen, C.-C., Reischl, J., Qaiser, T., Yeh, C.-Y.: Hybrid aggregation network for survival analysis from whole slide histopathological images. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 731–740. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_70
Agarwal, S., Eltigani Osman Abaker, M., Daescu, O.: Survival prediction based on histopathology imaging and clinical data: A novel, whole slide cnn approach. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 762–771. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_73
Braman, N., Gordon, J.W.H., Goossens, E.T., Willis, C., Stumpe, M.C., Venkataraman, J.: Deep orthogonal fusion: multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) MICCAI 2021. LNCS, vol. 12905, pp. 667–677. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87240-3_64
Li, H., et al.: DT-MIL: Deformable Transformer for Multi-instance Learning on Histopathological Image. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 206–216. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_20
Chen, R.J., et al.: Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878. e866 (2022)
Bag, S., et al.: Computational analysis of p63+ nuclei distribution pattern by graph theoretic approach in an oral pre-cancer (sub-mucous fibrosis). J. Pathol. Inform. 4, 35 (2013)
Chen, R.J., et al.: Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 339–349. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_33
Pati, P., et al.: Hact-net: A hierarchical cell-to-tissue graph neural network for histopathological image classification. In: Sudre, C.H., Fehri, H., Arbel, T., Baumgartner, C.F., Dalca, A., Tanno, R., Van Leemput, K., Wells, W.M., Sotiras, A., Papiez, B., Ferrante, E., Parisot, S. (eds.) UNSURE/GRAIL -2020. LNCS, vol. 12443, pp. 208–219. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60365-6_20
Zhou, Y., et al.: CGC-Net: cell graph convolutional network for grading of colorectal cancer histology images. In: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (2019)
Chen, R.J., et al.: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging (2020)
Kang, B., et al.: Prognostic value of tumor-infiltrating lymphocytes in Epstein-Barr virus-associated gastric cancer. Ann. Oncol. 27, 494–501 (2016)
Jiang, Y., et al.: Predicting peritoneal recurrence and disease-free survival from CT images in gastric cancer with multitask deep learning: a retrospective study. The Lancet Digit. Health 4, e340–e350 (2022)
Stringer, C., Wang, T., Michaelos, M., Pachitariu, M.: Cellpose: a generalist algorithm for cellular segmentation. Nat. Methods 18, 100–106 (2021)
Graham, S., et al.: Hover-net: Simultaneous segmentation and classification of nuclei in multi-tissue histology images. Med. Image Anal. 58, 101563 (2019)
Greenwald, N.F., et al.: Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat. Biotechnol. 40, 555–565 (2022)
Gamper, J., et al.: Pannuke dataset extension, insights and baselines. arXiv preprint arXiv:2003.10778 (2020)
Muja, M., Lowe, D.G.: Fast approximate nearest neighbors with automatic algorithm configuration. In: VISAPP 2009 - Proceedings of the Fourth International Conference on Computer Vision Theory and Applications (1) 2, 2 (2009)
Lee, Y., et al.: Derivation of prognostic contextual histopathological features from whole-slide images of tumours via graph deep learning. Nat. Biomed. Eng., 1–15 (2022)
Lin, D.Y., Wei, L.-J., Ying, Z.: Checking the Cox model with cumulative sums of martingale-based residuals. Biometrika 80, 557–572 (1993)
Liu, J., et al.: An integrated TCGA pan-cancer clinical data resource to drive high-quality survival outcome analytics. Cell 173, 400–416. e411 (2018)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Gao, Z., et al.: Instance-based vision transformer for subtyping of papillary renal cell carcinoma in histopathological image. In: de Bruijne, M., Cattin, P.C., Cotin, S., Padoy, N., Speidel, S., Zheng, Y., Essert, C. (eds.) MICCAI 2021. LNCS, vol. 12908, pp. 299–308. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-87237-3_29
Hou, W., Huang, H., Peng, Q., Yu, R., Yu, L., Wang, L.: Spatial-hierarchical graph neural network with dynamic structure learning for histological image classification. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part II, pp. 181–191. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16434-7_18
Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant 2022YFC2009903 / 2022YFC2009900, in part by the National Natural Science Foundation of China under Grants 62171377, in part by the Ningbo Clinical Research Center for Medical Imaging under Grant 2021L003 (Open Project 2022LYKFZD06), and in part by the Natural Science Foundation of Ningbo City, China, under Grant 2021J052.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, Z., Jiang, Y., Liu, L., Xia, Y., Li, R. (2024). Single-Cell Spatial Analysis of Histopathology Images for Survival Prediction via Graph Attention Network. In: Wu, S., Shabestari, B., Xing, L. (eds) Applications of Medical Artificial Intelligence. AMAI 2023. Lecture Notes in Computer Science, vol 14313. Springer, Cham. https://doi.org/10.1007/978-3-031-47076-9_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-47076-9_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47075-2
Online ISBN: 978-3-031-47076-9
eBook Packages: Computer ScienceComputer Science (R0)
