Abstract
COVID-19 is a viral disease that causes severe acute respiratory inflammation. Although with less death rate, its increasing infectivity rate, together with its acute symptoms and high number of infections, is still attracting growing interests in the image analysis of COVID-19 pneumonia. Current accurate diagnosis by radiologists requires two modalities of X-Ray and Computed Tomography (CT) images from one patient. However, one modality might miss in clinical practice. In this study, we propose a novel multi-modality model to integrate X-Ray and CT data to further increase the versatility and robustness of the AI-assisted COVID-19 pneumonia diagnosis that can tackle incomplete modalities. We develop a Convolutional Neural Networks (CNN) and Transformers hybrid architecture, which extracts extensive features from the distinct data modalities. This classifier is designed to be able to predict COVID-19 images with X-Ray image, or CT image, or both, while at the same time preserving the robustness when missing modalities are found. Conjointly, a new method is proposed to fuse three-dimensional and two-dimensional images, which further increase the feature extraction and feature correlation of the input data. Thus, verified with a real-world public dataset of BIMCV-COVID19, the model outperform state-of-the-arts with the AUC score of 79.93%. Clinically, the model has important medical significance for COVID-19 examination when some image modalities are missing, offering relevant flexibility to medical teams. Besides, the structure may be extended to other chest abnormalities to be detected by X-ray or CT examinations. Code is available at https://github.com/edurbi/MICCAI2023.
E. L. Carbonell and Y. Shen—Equal contributions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Percentage of Visits for COVID-19-Like Illness: Covid data page. https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covidview/index.html
Griffin, D.O., et al.: The importance of understanding the stages of covid-19 in treatment and trials. AIDS Rev. 23(1), 40–47 (2021). https://doi.org/10.24875/aidsrev.200001261
Luo, N., et al.: Utility of chest CT in diagnosis of covid-19 pneumonia. Diagn. Interv. Radiol. 26(5), 437–442 (2020). https://doi.org/10.5152/dir.2020.20144. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7490028/. pMID: 32490829; PMCID: PMC7490028
Alyasseri, Z.A.A., et al.: Review on covid-19 diagnosis models based on machine learning and deep learning approaches. Exp. Syst. 39(3), e12759 (2022). https://doi.org/10.1111/exsy.12759. https://onlinelibrary.wiley.com/doi/abs/10.1111/exsy.12759
Li, B., et al.: Diagnostic value and key features of computed tomography in coronavirus disease 2019. Emerg. Microbes Infect. 9(1), 787–793 (2020). https://doi.org/10.1080/22221751.2020.1750307. pMID: 32241244
Abdelaziz, M., Wang, T., Elazab, A.: Alzheimer’s disease diagnosis framework from incomplete multimodal data using convolutional neural networks. J. Biomed. Informatics 121, 103863 (2021). https://doi.org/10.1016/j.jbi.2021.103863. https://www.sciencedirect.com/science/article/pii/S1532046421001921
Azad, R., Khosravi, N., Dehghanmanshadi, M., Cohen-Adad, J., Merhof, D.: Medical image segmentation on MRI images with missing modalities: a review (2022). https://doi.org/10.48550/ARXIV.2203.06217. https://arxiv.org/abs/2203.06217
Ma, M., Ren, J., Zhao, L., Tulyakov, S., Wu, C., Peng, X.: SMIL: multimodal learning with severely missing modality. Proc. AAAI Conf. Artif. Intell. 35(3), 2302–2310 (2021). https://doi.org/10.1609/aaai.v35i3.16330. https://ojs.aaai.org/index.php/AAAI/article/view/16330
Jin, L., Zhao, K., Zhao, Y., Che, T., Li, S.: A hybrid deep learning method for early and late mild cognitive impairment diagnosis with incomplete multimodal data. Frontiers Neuroinf. (2022). https://doi.org/10.3389/fninf.2022.843566. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8965366/
Gao, X., Shi, F., Shen, D., Liu, M.: Task-induced pyramid and attention Gan for multimodal brain image imputation and classification in Alzheimer’s disease. IEEE J. Biomed. Health Inform. 26(1), 36–43 (2022). https://doi.org/10.1109/JBHI.2021.3097721
Zhang, Y., et al.: mmFormer: multimodal medical transformer for incomplete multimodal learning of brain tumor segmentation (2022). https://doi.org/10.48550/ARXIV.2206.02425. https://arxiv.org/abs/2206.02425
Altman, D.G., Bland, J.M.: Missing data. BMJ 334(7590), 424 (2007). https://doi.org/10.1136/bmj.38977.682025.2C. https://www.bmj.com/content/334/7590/424
Gadzicki, K., Khamsehashari, R., Zetzsche, C.: Early vs late fusion in multimodal convolutional neural networks. In: 2020 IEEE 23rd International Conference on Information Fusion (FUSION), pp. 1–6 (2020). https://doi.org/10.23919/FUSION45008.2020.9190246
Choi, J.H., Lee, J.S.: EmbraceNet: a robust deep learning architecture for multimodal classification. Inf. Fusion 51, 259–270 (2019). https://doi.org/10.1016/j.inffus.2019.02.010. https://www.sciencedirect.com/science/article/pii/S1566253517308242
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale (2020). https://doi.org/10.48550/ARXIV.2010.11929. https://arxiv.org/abs/2010.11929
de la Iglesia Vayá, M., et al.: BIMCV covid-19+: a large annotated dataset of RX and CT images from covid-19 patients with extension Part II (2023). https://doi.org/10.21227/mpqg-j236
Acknowledgments
This work was supported by National Natural Science Foundation of China (Grant No. 62102247) and Natural Science Foundation of Shanghai (No. 23ZR1430700).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Lloret Carbonell, E., Shen, Y., Yang, X., Ke, J. (2023). COVID-19 Pneumonia Classification with Transformer from Incomplete Modalities. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14224. Springer, Cham. https://doi.org/10.1007/978-3-031-43904-9_37
Download citation
DOI: https://doi.org/10.1007/978-3-031-43904-9_37
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43903-2
Online ISBN: 978-3-031-43904-9
eBook Packages: Computer ScienceComputer Science (R0)
