Authors:
Raquel Simão
1
;
2
;
Marília Barandas
2
;
1
;
David Belo
2
and
Hugo Gamboa
1
;
2
Affiliations:
1
LIBPhys (Laboratory for Instrumentation, Biomedical Engineering and Radiation Physics), NOVA School of Science and Technology, Campus da Caparica, 2829-516, Portugal
;
2
Associação Fraunhofer Portugal Research, Rua Alfredo Allen 455/461, 4200-135 Porto, Portugal
Keyword(s):
Uncertainty Quantification, Monte Carlo Dropout, Deep Ensemble, Dataset Shift, Active Learning.
Abstract:
Machine Learning (ML) models can predict diseases with noteworthy results. However, when implemented, their generalization are compromised, resulting in lower performances and render healthcare professionals more susceptible into delivering erroneous diagnostics. This study focuses on the use of uncertainty measures to abstain from classifying samples and use the rejected samples as a selection criterion for active learning. For the multi-label classification of cardiac arrhythmias different methods for uncertainty quantification were compared using three Deep Learning (DL) models: a single model and two pseudoensemble models using Monte-Carlo (MC) Dropout and Deep Ensemble (DE) techniques. When tested with an external dataset, the models’ performances dropped from a F1-Score of 96% to 70%, indicating the possibility of dataset shift. The uncertainty measures for classification with rejection resulted in an increase of the rejection rate from 10% in the training set to a range betwee
n 30% to 50% on the external dataset. For the active learning approach, 10% of the highest uncertainty samples were used to retrain the models and their performance increased by almost 5%. Although there are still challenges to the implementation of ML models, the results show that uncertainty quantification is a valuable method to employ in safety mechanisms under dataset shift conditions.
(More)