Abstract
Intracerebral hemorrhage (ICH) is a cerebrovascular disease with high mortality and morbidity rates. Early-stage ICH patients often lack clear surgical indications, which is quite challenging for neurosurgeons to make treatment decisions. Currently, early treatment decisions for ICH primarily rely on the clinical experience of neurosurgeons. Although there have been attempts to combine local CT imaging with clinical data for decision-making, these approaches fail to provide deep semantic analysis and do not fully leverage the synergistic effects between different modalities. To address this issue, this paper introduces a novel multi-modality predictive model that combines CT images and clinical data to provide reliable treatment decisions for ICH patients. Specifically, this model employs a combination of 3D CNN and Transformer to analyze patients’ brain CT scans, effectively capturing the 3D spatial information of intracranial hematomas and surrounding brain tissue. In addition, it utilizes a contrastive language-image pre-training (CLIP) module to extract demographic features and important clinical data and integrates with CT imaging data through a cross-attention mechanism. Furthermore, a novel CNN-based multilayer perceptron (MLP) layer is designed to enhance the understanding of the 3D spatial features. Extensive experiments conducted on real clinical datasets demonstrate that the proposed method significantly improves the accuracy of treatment decisions compared to existing state-of-the-art methods. Code is available at https://github.com/Henry-Xiong/3DCT-ICH.
Z. Xiong and K. Zhao—Equal Contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Adeoye, O., Broderick, J.P.: Advances in the management of intracerebral hemorrhage. Nature Reviews Neurology 6(11), 593–601 (2010)
Borsos, B., Allaart, C.G., van Halteren, A.: Predicting stroke outcome: A case for multimodal deep learning methods with tabular and ct perfusion data. Artificial Intelligence in Medicine 147, 102719 (2024)
Boutet, A., Madhavan, R., Elias, G.J., Joel, S.E., Gramer, R., Ranjan, M., Paramanandam, V., Xu, D., Germann, J., Loh, A., et al.: Predicting optimal deep brain stimulation parameters for parkinson’s disease using functional mri and machine learning. Nature communications 12(1), 3043 (2021)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European conference on computer vision. pp. 213–229. Springer (2020)
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
He, S., Grant, P.E., Ou, Y.: Global-local transformer for brain age estimation. IEEE transactions on medical imaging 41(1), 213–224 (2021)
Heit, J.J., Iv, M., Wintermark, M.: Imaging of intracranial hemorrhage. Journal of stroke 19(1), 11 (2017)
Ji, R., Shen, H., Pan, Y., Wang, P., Liu, G., Wang, Y., Li, H., Zhao, X., Wang, Y.: A novel risk score to predict 1-year functional outcome after intracerebral hemorrhage and comparison with existing scores. Critical Care 17, 1–10 (2013)
Keep, R.F., Hua, Y., Xi, G.: Intracerebral haemorrhage: mechanisms of injury and therapeutic targets. The Lancet Neurology 11(8), 720–731 (2012)
Li, L., Poon, M.T., Samarasekera, N.E., Perry, L.A., Moullaali, T.J., Rodrigues, M.A., Loan, J.J., Stephen, J., Lerpiniere, C., Tuna, M.A., et al.: Risks of recurrent stroke and all serious vascular events after spontaneous intracerebral haemorrhage: pooled analyses of two population-based studies. The Lancet Neurology 20(6), 437–447 (2021)
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 10012–10022 (2021)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Lu, Z., Xie, H., Liu, C., Zhang, Y.: Bridging the gap between vision transformers and convolutional neural networks on small datasets. Advances in Neural Information Processing Systems 35, 14663–14677 (2022)
Ma, W., Chen, C., Abrigo, J., Mak, C.H.K., Gong, Y., Chan, N.Y., Han, C., Liu, Z., Dou, Q.: Treatment outcome prediction for intracerebral hemorrhage via generative prognostic model with imaging and tabular data. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 715–725. Springer (2023)
Magid-Bernstein, J., Girard, R., Polster, S., Srinath, A., Romanos, S., Awad, I.A., Sansing, L.H.: Cerebral hemorrhage: pathophysiology, treatment, and future directions. Circulation research 130(8), 1204–1229 (2022)
Nguyen, H.H., Saarakkala, S., Blaschko, M.B., Tiulpin, A.: Climat: Clinically-inspired multi-agent transformers for knee osteoarthritis trajectory forecasting. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). pp. 1–5. IEEE (2022)
de Oliveira Manoel, A.L.: Surgery for spontaneous intracerebral hemorrhage. Critical Care 24(1), 45 (2020)
Pölsterl, S., Wolf, T.N., Wachinger, C.: Combining 3d image and tabular data via the dynamic affine feature map transform. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part V 24. pp. 688–698. Springer (2021)
Puy, L., Parry-Jones, A.R., Sandset, E.C., Dowlatshahi, D., Ziai, W., Cordonnier, C.: Intracerebral haemorrhage. Nature Reviews Disease Primers 9(1), 14 (2023)
Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
Shan, X., Li, X., Ge, R., Wu, S., Elazab, A., Zhu, J., Zhang, L., Jia, G., Xiao, Q., Wan, X., et al.: Gcs-ichnet: Assessment of intracerebral hemorrhage prognosis using self-attention with domain knowledge integration. In: 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 2217–2222. IEEE (2023)
Wang, K., Liu, Q., Mo, S., Zheng, K., Li, X., Li, J., Chen, S., Tong, X., Cao, Y., Li, Z., et al.: A decision tree model to help treatment decision-making for severe spontaneous intracerebral hemorrhage. International Journal of Surgery pp. 10–1097 (2023)
Wang, W., Chen, C., Ding, M., Yu, H., Zha, S., Li, J.: Transbts: Multimodal brain tumor segmentation using transformer. In: Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part I 24. pp. 109–119. Springer (2021)
Zhou, H.Y., Yu, Y., Wang, C., Zhang, S., Gao, Y., Pan, J., Shao, J., Lu, G., Zhang, K., Li, W.: A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nature Biomedical Engineering pp. 1–13 (2023)
Acknowledgement
This work was supported by the National Natural Science Foundation of China (NSFC) under Grant No. 62102133, the High-level and Urgently Needed Overseas Talent Programs of Jiangxi Province under Grant No. 20232BCJ25026, the Natural Science Foundation of Henan Province under Grant No. 242300421402, the Kaifeng Major Science and Technology Project under Grant No. 21ZD011, the Ji’an Finance and Science Foundation under Grants No. 20211-085454, 20222-151746, 20222-151704, Ji’an key core common technology “reveal the list” Project under Grant No. 2022-1, Science and Technology Fund Project of Guizhou Provincial Health Commission No. gzwkj2021-198, Quanzhou Science and Technology Plan Project No. 2024NY040, the Seed Fund Project of the Second Affiliated Hospital of Fujian Medical University No. 2021MP01 and the Henan University Graduate Student Excellence Program No. SYLYC2023135.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Ethics declarations
Disclosure of Interests
The authors have no competing interests to declare that are relevant to the content of this article.
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xiong, Z. et al. (2024). Multi-modality 3D CNN Transformer for Assisting Clinical Decision in Intracerebral Hemorrhage. In: Linguraru, M.G., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. MICCAI 2024. Lecture Notes in Computer Science, vol 15005. Springer, Cham. https://doi.org/10.1007/978-3-031-72086-4_49
Download citation
DOI: https://doi.org/10.1007/978-3-031-72086-4_49
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-72085-7
Online ISBN: 978-3-031-72086-4
eBook Packages: Computer ScienceComputer Science (R0)
