Rethinking an ROC partial area index for evaluating the classification performance at a high specificity range

Abstract

The area under a receiver operating characteristic (ROC) curve is valuable for evaluating the classification performance described by the entire ROC curve in many fields including decision making and medical diagnosis. However, this can be misleading when clinical tasks demand a restricted specificity range. The partial area under a portion of the ROC curve (${ pAUC}$) has more practical relevance in such situations, but it is usually transformed to overcome some drawbacks and improve its interpretation. The standardized ${ pAUC}$ (${ SpAUC}$) index is considered as a meaningful relative measure of predictive accuracy. Nevertheless, this ${ SpAUC}$ index might still show some limitations due to ROC curves crossing the diagonal line, and to the problem when comparing two tests with crossing ROC curves in the same restricted specificity range. This paper provides an alternative ${ pAUC}$ index which overcomes these limitations. Tighter bounds for the ${ pAUC}$ of an ROC curve are derived, and then a modified ${ pAUC}$ index for any restricted specificity range is established. In addition, the proposed tighter partial area index (${ TpAUC}$) is also shown for classifier when high specificity must be clinically maintained. The variance of the ${ TpAUC}$ is also studied analytically and by simulation studies in a theoretical framework based on the most typical assumption of a binormal model, and estimated by using nonparametric bootstrap resampling in the empirical examples. Simulated and real datasets illustrate the practical utility of the ${ TpAUC}$.