Abstract
Early and accurate prediction of overall survival (OS) time can help to obtain better treatment planning for brain tumor patients. Although many OS time prediction methods have been developed and obtain promising results, there are still several issues. First, conventional prediction methods rely on radiomic features at the local lesion area of a magnetic resonance (MR) volume, which may not represent the full image or model complex tumor patterns. Second, different types of scanners (i.e., multi-modal data) are sensitive to different brain regions, which makes it challenging to effectively exploit the complementary information across multiple modalities and also preserve the modality-specific properties. Third, existing methods focus on prediction models, ignoring complex data-to-label relationships. To address the above issues, we propose an end-to-end OS time prediction model; namely, Multi-modal Multi-channel Network (\(\text {M}^2\text {Net}\)). Specifically, we first project the 3D MR volume onto 2D images in different directions, which reduces computational costs, while preserving important information and enabling pre-trained models to be transferred from other tasks. Then, we use a modality-specific network to extract implicit and high-level features from different MR scans. A multi-modal shared network is built to fuse these features using a bilinear pooling model, exploiting their correlations to provide complementary information. Finally, we integrate the outputs from each modality-specific network and the multi-modal shared network to generate the final prediction result. Experimental results demonstrate the superiority of our \(\text {M}^2\text {Net}\) model over other methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chartsias, A., Joyce, T., Giuffrida, M.V., Tsaftaris, S.A.: Multimodal MR synthesis via modality-invariant latent representation. IEEE TMI 37(3), 803–814 (2017)
Fan, J., Cao, X., Wang, Q., Yap, P.T., Shen, D.: Adversarial learning for mono-or multi-modal registration. Med. Image Anal. 58, 101545 (2019)
Fan, J., Cao, X., et al.: BIRNet: brain image registration using dual-supervised fully convolutional networks. Med. Image Anal. 54, 193–206 (2019)
Feng, X., Tustison, N., Meyer, C.: Brain tumor segmentation using an ensemble of 3 D U-Nets and overall survival prediction using radiomic features. In: International MICCAI Brainlesion Workshop, pp. 279–288. Springer (2018)
Gao, Y., Beijbom, O., et al.: Compact bilinear pooling. In: CVPR, pp. 317–326. IEEE (2016)
Gevertz, J.L., Torquato, S.: Modeling the effects of vasculature evolution on early brain tumor growth. J. Theor. Biol. 243(4), 517–531 (2006)
Havaei, M., et al.: Brain tumor segmentation with deep neural networks. Med. Image Anal. 35, 18–31 (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778. IEEE (2016)
Isensee, F., Kickingereder, P., Wick, W., Bendszus, M., Maier-Hein, K.H.: Brain tumor segmentation and radiomics survival prediction: contribution to the BRATS 2017 challenge. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 287–297. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75238-9_25
Kao, P.Y., Ngo, T., Zhang, A., Chen, J.W., Manjunath, B.S.: Brain tumor segmentation and tractographic feature extraction from structural MR images for overall survival prediction. In: Crimi, A., Bakas, S., Kuijf, H., Keyvan, F., Reyes, M., van Walsum, T. (eds.) BrainLes 2018. LNCS, vol. 11384, pp. 128–141. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11726-9_12
Khvostikov, A., Aderghal, K., Benois-Pineau, J., Krylov, A., Catheline, G.: 3D CNN-based classification using sMRI and MD-DTI images for alzheimer disease studies. arXiv preprint arXiv:1801.05968 (2018)
Li, W., Zhao, Y., Chen, X., Xiao, Y., Qin, Y.: Detecting alzheimer’s disease on small dataset: a knowledge transfer perspective. IEEE JBHI 23(3), 1234–1242 (2018)
Lin, T.Y., RoyChowdhury, A., Maji, S.: Bilinear cnn models for fine-grained visual recognition. In: ICCV, pp. 1449–1457. IEEE (2015)
Lipkova, J., et al.: Personalized radiotherapy design for glioblastoma: integrating mathematical tumor models, multimodal scans, and bayesian inference. IEEE TMI 38(8), 1875–1884 (2019)
Liu, J., Ji, S., Ye, J.: SLEP: sparse learning with efficient projections. Ariz. State Univ. 6(491), 7 (2009)
Liu, L., et al.: Overall survival time prediction for high-grade glioma patients based on large-scale brain functional networks. Brain Imaging Behav. 13(5), 1333–1351 (2019). https://doi.org/10.1007/s11682-018-9949-2
Liu, Y., Xu, X., Yin, L., Zhang, X., Li, L., Lu, H.: Relationship between glioblastoma heterogeneity and survival time: an MR imaging texture analysis. Am. J. Neuroradiol. 38(9), 1695–1701 (2017)
Menze, B.H., et al.: The multimodal brain tumor image segmentation benchmark ( BRATS). IEEE TMI 34(10), 1993–2024 (2015)
Nie, D., Lu, J., Zhang, H., Adeli, E., Wang, J., et al.: Multi-channel 3 D deep feature learning for survival time prediction of brain tumor patients using multi-modal neuroimages. Sci. Rep. 9(1), 1103 (2019)
Pope, W.B., Sayre, J., Perlina, A., Villablanca, J.P., Mischel, P.S., Cloughesy, T.F.: MR imaging correlates of survival in patients with high-grade gliomas. Am. J. Neuroradiol. 26(10), 2466–2474 (2005)
Provost, F., Fawcett, T.: Robust classification for imprecise environments. Mach. Learn. 42(3), 203–231 (2001)
Razek, A., El-Serougy, L., Abdelsalam, M., Gaballa, G., Talaat, M.: Differentiation of residual/recurrent gliomas from postradiation necrosis with arterial spin labeling and diffusion tensor magnetic resonance imaging-derived metrics. Neuroradiology 60(2), 169–177 (2018). https://doi.org/10.1007/s00234-017-1955-3
Ricard, D., Idbaih, A., Ducray, F., Lahutte, M., Hoang-Xuan, K., Delattre, J.Y.: Primary brain tumours in adults. Lancet 379(9830), 1984–1996 (2012)
Tang, Z., et al.: Pre-operative overall survival time prediction for glioblastoma patients using deep learning on both imaging phenotype and genotype. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11764, pp. 415–422. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32239-7_46
Wang, Y.X., Hebert, M.: Learning to learn: model regression networks for easy small sample learning. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 616–634. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_37
Zacharaki, E.I., Morita, N., Bhatt, P., Orourke, D., Melhem, E.R., Davatzikos, C.: Survival analysis of patients with high-grade gliomas based on data mining of imaging variables. Am. J. Neuroradiol 33(6), 1065–1071 (2012)
Zhang, C., Liu, Y., Fu, H.: Ae2-nets: autoencoder in autoencoder networks. In: VPR, pp. 2577–2585. IEEE (2019)
Zhang, T., et al.: SkrGAN: sketching-rendering unconditional generative adversarial networks for medical image synthesis. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 777–785. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_85
Zhou, F., Li, T., Li, H., Zhu, H.: TPCNN: two-phase patch-based convolutional neural network for automatic brain tumor segmentation and survival prediction. In: Crimi, A., Bakas, S., Kuijf, H., Menze, B., Reyes, M. (eds.) BrainLes 2017. LNCS, vol. 10670, pp. 274–286. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75238-9_24
Zhou, T., Fu, H., Chen, G., Shen, J., Shao, L.: Hi-net: hybrid-fusion network for multi-modal MR image synthesis. In: TMI. IEEE (2020)
Zhou, T., Thung, K.H., et al.: Effective feature learning and fusion of multimodality data using stage-wise deep neural network for dementia diagnosis. Hum. Brain Mapp. 40(3), 1001–1016 (2019)
Zhou, T., Zhang, C., Peng, X., Bhaskar, H., Yang, J.: Dual shared-specific multiview subspace clustering. IEEE Transactions on Cybernetics (2019)
Zhou, T., Fu, H., Gong, C., Shen, J., Shao, L., Porikli, F.: Multi-mutual consistency induced transfer subspace learning for human motion segmentation. In: CVPR, pp. 10277–10286. IEEE (2020)
Zhou, T., et al.: Deep multi-modal latent representation learning for automated dementia diagnosis. In: Shen, D., et al. (eds.) MICCAI 2019. LNCS, vol. 11767, pp. 629–638. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32251-9_69
Zhu, H., Yuan, M., Qiu, C., et al.: Multivariate classification of earthquake survivors with post-traumatic stress disorder based on large-scale brain networks. Acta Psychiatr. Scand. 141(3), 285–298 (2020)
Acknowledgement
This research was supported in part by NSF of China (No: 61973090) and NSF of Tianjin (No: 19JCYBJC15200).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhou, T. et al. (2020). \(\text {M}^2\text {Net}\): Multi-modal Multi-channel Network for Overall Survival Time Prediction of Brain Tumor Patients. In: Martel, A.L., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2020. MICCAI 2020. Lecture Notes in Computer Science(), vol 12262. Springer, Cham. https://doi.org/10.1007/978-3-030-59713-9_22
Download citation
DOI: https://doi.org/10.1007/978-3-030-59713-9_22
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59712-2
Online ISBN: 978-3-030-59713-9
eBook Packages: Computer ScienceComputer Science (R0)
