Abstract
Whole slide pathological images (WSIs) are the gold standard for lung cancer prognosis. However, due to their high resolution and limited annotations, lung cancer survival analysis based on WSIs becomes a challenging task. Some recent methods that fuse WSI and other modalities have achieved certain results. However, these methods tend to focus on integrating gene related information while overlooking relatively easily obtainable clinical variables and often rely on labor-intensive ROI annotations. In this work, we propose a novel framework for lung cancer survival analysis, which obviates the need for ROI annotations and fully exploits WSIs and clinical information by introducing a multi-modality fusion module and multi-task learning. We also utilizes self-supervised learning to eliminate the heterogeneity between WSIs and natural images. Experimental results via 5-fold cross-validation on 1,225 WSIs from 444 patients from NLST validate the state-of-the-art performance of our proposed method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. Proc. Nat. Acad. Sci. 115(13), E2970–E2979 (2018)
Subramanian, V., Chidester, B., Ma, J., Do, M.N.: Correlating cellular features with gene expression using CCA. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 805–808. IEEE (2018)
Chen, R.J., et al.: Pathomic fusion: an integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Trans. Med. Imaging 41(4), 757–770 (2020)
Zhu, X., Yao, J., Zhu, F., Huang, J.: WSISA: making survival prediction from whole slide histopathological images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7234–7242 (2017)
Yao, J., Zhu, X., Huang, J.: Deep multi-instance learning for survival prediction from whole slide images. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 496–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_55
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255. IEEE (2009)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Ilse, M., Tomczak, J., Welling, M.: Attention-based deep multiple instance learning. In: International Conference on Machine Learning, pp. 2127–2136. PMLR (2018)
Kaplan, E.L., Meier, P.: Nonparametric estimation from incomplete observations. J. Am. Stat. Assoc. 53(282), 457–481 (1958)
Ishwaran, H., Kogalur, U.B., Blackstone, E.H., Lauer, M.S.: Random survival forests (2008)
Antol, S., et al.: VQA: visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2425–2433 (2015)
Yu, Z., Yu, J., Fan, J., Tao, D.: Multi-modal factorized bilinear pooling with co-attention learning for visual question answering. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1821–1830 (2017)
de Torres, J.P., et al.: Assessing the relationship between lung cancer risk and emphysema detected on low-dose CT of the chest. Chest 132(6), 1932–1938 (2007)
Kravchenko, J., Berry, M., Arbeev, K., Lyerly, H.K., Yashin, A., Akushevich, I.: Cardiovascular comorbidities and survival of lung cancer patients: medicare data based analysis. Lung Cancer 88(1), 85–93 (2015)
Team, N.L.S.T.R.: The national lung screening trial: overview and study design. Radiology 258(1), 243–253 (2011)
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)
Zubiolo, A.: Extraction de caractéristiques et apprentissage statistique pour l’imagerie biomédicale cellulaire et tissulaire. Ph.D. thesis, Université Nice Sophia Antipolis (2015)
Harrell, F.E., Jr., Lee, K.L., Mark, D.B.: Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat. Med. 15(4), 361–387 (1996)
Tibshirani, R.: The lasso method for variable selection in the cox model. Stat. Med. 16(4), 385–395 (1997)
Pölsterl, S., Navab, N., Katouzian, A.: Fast training of support vector machines for survival analysis. In: Appice, A., Rodrigues, P.P., Santos Costa, V., Gama, J., Jorge, A., Soares, C. (eds.) ECML PKDD 2015. LNCS (LNAI), vol. 9285, pp. 243–259. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-23525-7_15
Yao, J., Zhu, X., Zhu, F., Huang, J.: Deep correlational learning for survival prediction from multi-modality data. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 406–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_46
Acknowledgments
This work was partially supported by the General Program of National Natural Science Foundation of China (NSFC) under Grant 62276189, and the Fundamental Research Funds for the Central Universities No. 22120220583.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wang, Y., Luo, Y., Li, B., Shen, X. (2024). Multi-modality Fusion Based Lung Cancer Survival Analysis with Self-supervised Whole Slide Image Representation Learning. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14437. Springer, Singapore. https://doi.org/10.1007/978-981-99-8558-6_28
Download citation
DOI: https://doi.org/10.1007/978-981-99-8558-6_28
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8557-9
Online ISBN: 978-981-99-8558-6
eBook Packages: Computer ScienceComputer Science (R0)