Abstract
Recent deep learning techniques have shown promising performance on survival prediction from Whole Slide Images (WSIs). These methods are often based on multiple-step frameworks including patch sampling, feature extraction, and feature aggregation. However, feature extraction typically relies on handcrafted features or Convolutional Neural Networks (CNNs) pretrained on ImageNet without fine-tuning, thus leading to suboptimal performance. Besides, to aggregate features, previous studies focus on WSI-level survival prediction but ignore the heterogeneous information that is present in multiple WSIs acquired for the same patient. To address the above challenges, we propose a survival prediction model that exploits heterogeneous features at the patient-level. Specifically, we introduce colorization as the pretext task to train the CNNs which are tailored for extracting features from patches of WSIs. In addition, we develop a patient-level framework integrating multiple WSIs for survival prediction with consistency and ranking losses. Extensive experiments show that our model achieves state-of-the-art performance on two large-scale public datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
We gratefully acknowledge the help and suggestions from the authors of RankSurv.
References
Abbet, C., Zlobec, I., Bozorgtabar, B., Thiran, J.-P.: Divide-and-rule: self-supervised learning for survival analysis in colorectal cancer. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 480–489. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_46
Davidson-Pilon, C.: lifelines: survival analysis in python. J. Open Sour. Softw. 4(40), 1317 (2019)
Di, D., Li, S., Zhang, J., Gao, Y.: Ranking-based survival prediction on histopathological whole-slide images. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 428–438. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_41
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Heagerty, P.J., Zheng, Y.: Survival model predictive accuracy and ROC curves. Biometrics 61(1), 92–105 (2005). https://doi.org/10.1111/j.0006-341X.2005.030814.x
Kandoth, C., et al.: Mutational landscape and significance across 12 major cancer types. Nature 502(7471), 333–339 (2013)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Kolesnikov, A., Zhai, X., Beyer, L.: Revisiting self-supervised visual representation learning. In: CVPR, pp. 1920–1929 (2019)
Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 577–593. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_35
Larsson, G., Maire, M., Shakhnarovich, G.: Colorization as a proxy task for visual understanding. In: CVPR, pp. 6874–6883 (2017)
Li, R., Yao, J., Zhu, X., Li, Y., Huang, J.: Graph CNN for survival analysis on whole slide pathological images. In: Frangi, A.F., Schnabel, J.A., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds.) MICCAI 2018. LNCS, vol. 11071, pp. 174–182. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-00934-2_20
Mobadersany, P., et al.: Predicting cancer outcomes from histology and genomics using convolutional networks. PNAS 115(13), E2970–E2979 (2018). https://doi.org/10.1073/pnas.1717139115
Muhammad, H., et al.: Unsupervised subtyping of cholangiocarcinoma using a deep clustering convolutional autoencoder. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 604–612. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_67
Otsu, N.: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)
Paszke, A., et al.: PyTorch: an imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d’Alché-Buc, F., Fox, E., Garnett, R. (eds.) NeurIPS, pp. 8024–8035. Curran Associates, Inc. (2019)
Pech-Pacheco, J.L., Cristobal, G., Chamorro-Martinez, J., Fernandez-Valdivia, J.: Diatom autofocusing in brightfield microscopy: a comparative study. In: ICPR 2000, vol. 3, pp. 314–317 (2000). https://doi.org/10.1109/ICPR.2000.903548
Shao, W., et al.: Diagnosis-guided multi-modal feature selection for prognosis prediction of lung squamous cell carcinoma. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 113–121. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_13
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2015)
Wulczyn, E., et al.: Deep learning-based survival prediction for multiple cancer types using histopathology images. PLoS One 15(6), e0233678 (2020)
Xiao, L., et al.: Censoring-aware deep ordinal regression for survival prediction from pathological images. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 449–458. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_43
Yang, H., Tianyi Zhou, J., Cai, J., Soon Ong, Y.: MIML-FCN+: multi-instance multi-label learning via fully convolutional networks with privileged information. In: CVPR (2017)
Yao, J., Zhu, X., Huang, J.: Deep multi-instance learning for survival prediction from whole slide images. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 496–504. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_55
Yao, J., Zhu, X., Jonnagaddala, J., Hawkins, N., Huang, J.: Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Med. Image Anal. 65, 101789 (2020)
Yao, J., Zhu, X., Zhu, F., Huang, J.: Deep correlational learning for survival prediction from multi-modality data. In: Descoteaux, M., Maier-Hein, L., Franz, A., Jannin, P., Collins, D.L., Duchesne, S. (eds.) MICCAI 2017. LNCS, vol. 10434, pp. 406–414. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66185-8_46
Yuan, Y., et al.: Assessing the clinical utility of cancer genomic and proteomic data across tumor types. Nat. Biotechnol. 32(7), 644–652 (2014)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR (2016)
Zhu, X., Yao, J., Huang, J.: Deep convolutional neural network for survival analysis with pathological images. In: BIBM, pp. 544–547. IEEE, December 2016. https://doi.org/10.1109/BIBM.2016.7822579
Zhu, X., Yao, J., Zhu, F., Huang, J.: WSISA: making survival prediction from whole slide histopathological images. In: CVPR (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Fan, L., Sowmya, A., Meijering, E., Song, Y. (2021). Learning Visual Features by Colorization for Slide-Consistent Survival Prediction from Whole Slide Images. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12908. Springer, Cham. https://doi.org/10.1007/978-3-030-87237-3_57
Download citation
DOI: https://doi.org/10.1007/978-3-030-87237-3_57
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87236-6
Online ISBN: 978-3-030-87237-3
eBook Packages: Computer ScienceComputer Science (R0)
