Abstract
Current methods for whole slide image (WSI) histopathology subregion classification and survival prediction rely on phenotype clustering from randomly sampled image tiles or on analyzing key tiles selected by experts from the much larger in size WSIs. These approaches do not capture the whole tissue region present in a histopathology image, also missing the spatial distribution of features that could be critical for good survival predictors. We propose a novel method that extracts a whole slide feature map (WSFM) in the first step and then uses it to train the survival prediction model. Specifically, we partition the WSI into tiles, and for each tile extract InceptionV3 features followed by PCA dimension reduction. The low dimension features of each tile are stored as the channel information in the WSFM. The resulting WSFM preserves the tile adjacency information and captures the entire tissue in the WSI. To overcome the small-size data set concern inherent to previous methods, we design a siamese survival convolutional neural network (SSCNN) that takes the WSFM and multivariate clinical features as input and predicts the survival score. We train the SSCNN using a novel loss function that combines a modified pairwise ranking loss and a bounded inverse term. The key advantages of the proposed method are that it does not require pixel-level annotations, a notorious bottleneck, and it can be easily adapted for any type of tumor without performance dependence on other parameters like the number of clusters. Experimental results demonstrate the effectiveness of the proposed SSCNN over other state-of-the-art survival analysis approaches.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Carpenter, A.E., et al.: CellProfiler: image analysis software for identifying and quantifying cell phenotypes. Genome Biol. 7(10), 1–11 (2006)
Harrell, F.E., Califf, R.M., Pryor, D.B., Lee, K.L., Rosati, R.A.: Evaluating the yield of medical tests. JAMA 247(18), 2543–2546 (1982)
Ishwaran, H., Gerds, T.A., Kogalur, U.B., Moore, R.D., Gange, S.J., Lau, B.M.: Random survival forests for competing risks. Biostatistics 15(4), 757–773 (2014)
Jaber, M.I., et al.: A deep learning image-based intrinsic molecular subtype classifier of breast tumors reveals tumor heterogeneity that may affect survival. Breast Cancer Res. 22(1), 12 (2020)
Jing, B., et al.: A deep survival analysis method based on ranking. Artif. Intell. Med. 98, 1–9 (2019)
Katzman, J.L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., Kluger, Y.: DeepSurv: personalized treatment recommender system using a cox proportional hazards deep neural network. BMC Med. Res. Methodol. 18(1), 1–12 (2018)
Koch, G., Zemel, R., Salakhutdinov, R.: Siamese neural networks for one-shot image recognition. In: ICML Deep Learning Workshop, Lille, vol. 2 (2015)
Lee, E.T., Wang, J.: Statistical Methods for Survival Data Analysis, vol. 476. Wiley, Hoboken (2003)
Li, R., Yao, J., Zhu, X., Li, Y., Huang, J.: Graph CNN for survival analysis on whole slide pathological images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_20
Li, Y., Wang, J., Ye, J., Reddy, C.K.: A multi-task learning formulation for survival analysis. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1715–1724 (2016)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818–2826 (2016)
Tibshirani, R.: The Lasso method for variable selection in the Cox model. Stat. Med. 16(4), 385–395 (1997)
Tomczak, K., Czerwińska, P., Wiznerowicz, M.: The cancer genome atlas (TCGA): an immeasurable source of knowledge. Contemp. Oncol. 19(1A), A68 (2015)
Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLOS ONE 15(6), e0233678 (2020)
Yang, Y., Zou, H.: A cocktail algorithm for solving the elastic net penalized Cox’s regression in high dimensions. Stat. Interface 6(2), 167–173 (2013)
Yao, J., Zhu, X., Huang, J.: Deep multi-instance learning for survival prediction from whole slide images. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 496–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_55
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)
Yogananda, C.G.B., et al.: Fully automated brain tumor segmentation and survival prediction of gliomas using deep learning and MRI. In: Crimi, A., Bakas, S. (eds.) BrainLes 2019. LNCS, vol. 11993, pp. 99–112. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-46643-5_10
Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 544–547. IEEE (2016)
Zhu, X., Yao, J., Zhu, F., Huang, J.: WSISA: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7234–7242 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Agarwal, S., Eltigani Osman Abaker, M., Daescu, O. (2021). Survival Prediction Based on Histopathology Imaging and Clinical Data: A Novel, Whole Slide CNN Approach. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12905. Springer, Cham. https://doi.org/10.1007/978-3-030-87240-3_73
Download citation
DOI: https://doi.org/10.1007/978-3-030-87240-3_73
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87239-7
Online ISBN: 978-3-030-87240-3
eBook Packages: Computer ScienceComputer Science (R0)