Assessing robustness of a deep-learning model for COVID-19 classification on chest radiographs

Mena Shenouda; Aditi Kaveti; Isabella Flerlage; Jayashree Kalpathy-Cramer; Maryellen L. Giger; Samuel G. Armato III

doi:10.1117/12.2652106

7 April 2023 Assessing robustness of a deep-learning model for COVID-19 classification on chest radiographs

Mena Shenouda, Aditi Kaveti, Isabella Flerlage, Jayashree Kalpathy-Cramer, Maryellen L. Giger, Samuel G. Armato III

Proceedings Volume 12465, Medical Imaging 2023: Computer-Aided Diagnosis; 124650F (2023) https://doi.org/10.1117/12.2652106
Event: SPIE Medical Imaging, 2023, San Diego, California, United States

Abstract

The COVID-19 pandemic has made a dramatic impact on human life, medical systems, and financial resources. Due to the disease’s pervasive nature, many different and interdisciplinary fields of research pivoted to study the disease. For example, deep learning (DL) techniques were employed early to assess patient diagnosis and prognosis from chest radiographs (CXRs) and computed tomography (CT) scans. While the use of artificial intelligence (AI) in the medical sector has displayed promising results, DL may suffer from lack of reproducibility and generalizability. In this study, the robustness of a pre-trained DL model utilizing the DenseNet-121 architecture was evaluated by using a larger collection of CXRs from the same institution that provided the original model with its test and training datasets. The current test set contained a larger span of dates, incorporated different strains of the virus, and included different immunization statuses. Considering differences in these factors, model performance between the original and current test sets was evaluated using area under the receiver operating characteristic curve (ROC AUC) [95% CI]. Statistical comparisons were performed using the Delong, KolmogorovSmirnov, and Wilcoxon rank-sum tests. Uniform manifold approximation and projection (UMAP) was used to help visualize whether underlying causes were responsible for differences in performance between test sets. In the task of classifying between COVID-positive and COVID-negative patients, the DL model achieved an AUC of 0.67 [0.65, 0.70], compared with the original performance of 0.76 [0.73, 0.79]. The results of this study suggest that underlying biases or overfitting may hinder performance when generalizing the model.

Conference Presentation

Citation Download Citation

Mena Shenouda, Aditi Kaveti, Isabella Flerlage, Jayashree Kalpathy-Cramer, Maryellen L. Giger, and Samuel G. Armato III "Assessing robustness of a deep-learning model for COVID-19 classification on chest radiographs", Proc. SPIE 12465, Medical Imaging 2023: Computer-Aided Diagnosis, 124650F (7 April 2023); https://doi.org/10.1117/12.2652106

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available