Abstract
Radiotherapy plays a vital role in treating patients with esophageal cancer (EC), whereas potential complications such as esophageal fistula (EF) can be devastating and even life-threatening. Therefore, predicting EF risks prior to radiotherapies for EC patients is crucial for their clinical treatment and quality of life. We propose a novel method of combining thoracic Computerized Tomography (CT) scans and clinical tabular data to improve the prediction of EF risks in EC patients. The multimodal network includes encoders to extract salient features from images and clinical data, respectively. In addition, we devise a self-attention module, named VisText, to uncover the complex relationships and correlations among different features. The associated multimodal features are integrated with clinical features by aggregation to further enhance prediction accuracy. Experimental results indicate that our method classifies EF status for EC patients with an accuracy of 0.8366, F1 score of 0.7337, specificity of 0.9312 and AUC of 0.9119, outperforming other methods in comparison.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Hirano, H., Boku, N.: The current status of multimodality treatment for unresectable locally advanced esophageal squamous cell carcinoma. Asia Pac J Clin Oncol 14, 291–299 (2018)
Borggreve, A.S., et al.: Surgical treatment of esophageal cancer in the era of multimodality management. Ann. New York Acad. Sci. (2018)
Tsushima, T., et al.: Risk factors for esophageal fistula associated with chemoradiotherapy for locally advanced unresectable esophageal cancer: a supplementary analysis of JCOG0303. Medicine 95 (2016)
Zhang, Y., Li, Z., Zhang, W., Chen, W., Song, Y.: Risk factors for esophageal fistula in patients with locally advanced esophageal carcinoma receiving chemoradiotherapy. Onco Targets Ther 11, 2311–2317 (2018)
Xu, Y., et al.: Development and validation of a risk prediction model for radiotherapy-related esophageal fistula in esophageal cancer. Radiat. Oncol. 14, 181 (2019)
Zhu, C., Wang, S., You, Y., Nie, K., Ji, Y.: Risk factors for esophageal fistula in esophageal cancer patients treated with radiotherapy: a systematic review and meta-analysis. Oncol Res Treat 43, 34–41 (2020)
Cui, H., Xu, Y., Li, W., Wang, L., Duh, H.: Collaborative learning of cross-channel clinical attention for radiotherapy-related esophageal fistula prediction from CT. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12261, pp. 212–220. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59710-8_21
Starke, S., et al.: 2D and 3D convolutional neural networks for outcome modelling of locally advanced head and neck squamous cell carcinoma. Sci. Rep. 10, 1–13 (2020)
Wu, Z., Shen, C., Van Den Hengel, A.: Wider or deeper: revisiting the resnet model for visual recognition. Pattern Recogn. 90, 119–133 (2019)
Jin, Q., Meng, Z., Sun, C., Cui, H., Su, R.: RA-UNet: a hybrid deep attention-aware network to extract liver and tumor in CT scans. Front. Bioengi. Biotechnol. 8, 1–15 (2020)
Jin, Q., Cui, H., Sun, C., Meng, Z., Su, R.: Free-form tumor synthesis in computed tomography images via richer generative adversarial network. Knowl.-Based Syst. 218, 106753 (2021)
Li, R., et al.: Referring image segmentation via recurrent refinement networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5745–5753 (2018)
Shi, H., Li, H., Meng, F., Wu, Q.: Key-word-aware network for referring expression image segmentation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11210, pp. 38–54. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01231-1_3
Ye, L., Rochan, M., Liu, Z., Wang, Y.: Cross-modal self-attention network for referring image segmentation. In: IEEE conference on computer vision and pattern recognition (CVPR), pp. 10502–10511 (2019)
Sharma, A., Vans, E., Shigemizu, D., Boroevich, K.A., Tsunoda, T.: DeepInsight: a methodology to transform a non-image data to an image for convolution neural network architecture. Sci. Rep. 9, 1–7 (2019)
Silva, L.A.V., Rohr, K.: Pan-cancer prognosis prediction using multimodal deep learning. In: International Symposium on Biomedical Imaging (ISBI), pp. 568–571. IEEE (2020)
Chauhan, G., et al.: Joint modeling of chest radiographs and radiology reports for pulmonary edema assessment. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 529–539. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_51
Yap, J., Yolland, W., Tschandl, P.: Multimodal skin lesion classification using deep learning. Exp. Dermatol. 27, 1261–1267 (2018)
Xu, T., Zhang, H., Huang, X., Zhang, S., Metaxas, D.N.: Multimodal deep learning for cervical dysplasia diagnosis. In: Ourselin, S., Joskowicz, L., Sabuncu, M.R., Unal, G., Wells, W. (eds.) MICCAI 2016. LNCS, vol. 9901, pp. 115–123. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46723-8_14
Vaswani, A., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)
Wang, X., Girshick, R., Gupta, A., He, K.: Non-local neural networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7794–7803 (2018)
Schlemper, J., et al.: Attention gated networks: learning to leverage salient regions in medical images. Med. Image Anal. 53, 197–207 (2019)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Guan, Y. et al. (2021). Predicting Esophageal Fistula Risks Using a Multimodal Self-attention Network. In: de Bruijne, M., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. MICCAI 2021. Lecture Notes in Computer Science(), vol 12905. Springer, Cham. https://doi.org/10.1007/978-3-030-87240-3_69
Download citation
DOI: https://doi.org/10.1007/978-3-030-87240-3_69
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-87239-7
Online ISBN: 978-3-030-87240-3
eBook Packages: Computer ScienceComputer Science (R0)
