Abstract
Understanding of prognosis and mortality is crucial for evaluating the treatment plans for patients. Recent developments of digital pathology and deep learning bring the possibility of predicting survival time using histopathology whole slide images (WSIs). However, most prevalent methods usually rely on a small set of patches sampled from a WSI and are unable to directly learn from an entire WSI. We argue that a small patch set cannot fully represent patients’ survival risks due to the heterogeneity of tumors; moreover, multiple WSIs from one patient need to be evaluated together. In this paper, we propose a Hybrid Aggregation Network (HANet) to adaptively aggregate information from multiple WSIs of one patient for survival analysis. Specifically, we first extract features from WSIs using a convolutional neural network trained in a self-supervised manner, and further aggregate feature maps using two proposed aggregation modules. The self-aggregation module propagates informative features to the entire WSI, and further abstract features to region representations. The WSI-aggregation module fuses all the region representations from different WSIs of one patient to predict patient-level survival risk. We conduct experiments on two WSI datasets that have accompanying survival data, i.e., NLST and TCGA-LUSC. The proposed method achieves state-of-the-art performances with concordance indices of 0.734 for NLST and 0.668 for TCGA-LUSC, outperforming existing approaches.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bejnordi, B.E., et al.: Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318(22), 2199–2210 (2017)
Courtiol, P., et al.: Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nat. Med. 25(10), 1519–1525 (2019)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Dietterich, T.G., Lathrop, R.H., Lozano-Pérez, T.: Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89(1–2), 31–71 (1997)
Ganeshan, B., Miles, K.A., Young, R.C., Chatwin, C.R.: Hepatic enhancement in colorectal cancer: texture analysis correlates with hepatic hemodynamics and patient survival. Acad. Radiol. 14(12), 1520–1530 (2007)
Gao, Y., Beijbom, O., Zhang, N., Darrell, T.: Compact bilinear pooling. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 317–326 (2016)
Hao, J., Kim, Y., Mallavarapu, T., Oh, J.H., Kang, M.: Interpretable deep neural network for cancer survival analysis by integrating genomic and clinical data. BMC Med. Genomics 12(10), 1–13 (2019)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 (2019)
He, K., Zhang, X., Ren, S., Sun, J.: Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Trans. Pattern Anal. Mach. Intell. 37(9), 1904–1916 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Kandoth, C., et al.: Mutational landscape and significance across 12 major cancer types. Nature 502(7471), 333–339 (2013)
Kim, J.H., et al.: Breast cancer heterogeneity: Mr imaging texture analysis and survival outcomes. Radiology 282(3), 665–675 (2017)
Li, R., Yao, J., Zhu, X., Li, Y., Huang, J.: Graph CNN for survival analysis on whole slide pathological images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_20
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, pp. 8024–8035 (2019)
Steck, H., Krishnapuram, B., Dehing-Oberije, C., Lambin, P., Raykar, V.C.: On ranking in survival analysis: Bounds on the concordance index. In: Advances in Neural InformationProcessing Systems, pp. 1209–1216 (2008)
Team, N.L.S.T.R.: Reduced lung-cancer mortality with low-dose computed tomographic screening. New England J. Med. 365(5), 395–409 (2011)
Tibshirani, R.: The lasso method for variable selection in the cox model. Stat. Med. 16(4), 385–395 (1997)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7794–7803 (2018)
Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One 15(6), e0233678 (2020)
Yao, J., Wang, S., Zhu, X., Huang, J.: Imaging biomarker discovery for lung cancer survival prediction. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 649–657. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_75
Yao, J., Zhu, X., Huang, J.: Deep multi-instance learning for survival prediction from whole slide images. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., Khan, A. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 496–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_55
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis p. 101789, July 2020. https://doi.org/10.1016/j.media.2020.101789. https://linkinghub.elsevier.com/retrieve/pii/S1361841520301535
Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: 2016 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pp. 544–547. IEEE (2016)
Zhu, X., Yao, J., Zhu, F., Huang, J.: Wsisa: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7234–7242 (2017)
Acknowledgement
We are grateful to Taiwan’s National Center for High-performance Computing for providing computing resources. We also thank Szu-Hua Chen, M.D. (aetherAI) for valuable advice.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Chang, JR., Lee, CY., Chen, CC., Reischl, J., Qaiser, T., Yeh, CY. (2021). Hybrid Aggregation Network for Survival Analysis from Whole Slide Histopathological Images. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12905. Springer, Cham. https://doi.org/10.1007/978-3-030-87240-3_70
Download citation
DOI: https://doi.org/10.1007/978-3-030-87240-3_70
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87239-7
Online ISBN: 978-3-030-87240-3
eBook Packages: Computer ScienceComputer Science (R0)
