Semi-parametric estimation of the area under the precision-recall curve

Berkman Sahiner; Weijie Chen; Aria Pezeshk; Nicholas Petrick

doi:10.1117/12.2216434

24 March 2016 Semi-parametric estimation of the area under the precision-recall curve

Berkman Sahiner, Weijie Chen, Aria Pezeshk, Nicholas Petrick

Proceedings Volume 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment; 97870D (2016) https://doi.org/10.1117/12.2216434
Event: SPIE Medical Imaging, 2016, San Diego, California, United States

Abstract

Precision and recall are two common metrics used in the evaluation of information retrieval systems. By changing the number of retrieved documents, one can obtain a precision-recall curve. The area under the precision-recall curve (AUCPR) has been suggested as a performance measure for information retrieval systems, in a manner similar to the use of the area under the receiver operating characteristic curve in binary classification. Limited work has been performed in the literature to investigate the bias and variance of AUCPR estimators. The goal of our study was to investigate the bias and variability of a semi-parametric binormal method for estimating the AUCPR, and to compare it to other techniques, such as average precision (AP) and lower trapezoid (LT) approximation. We show how AUCPR can be obtained given the binormal model parameters, and how its variance can be estimated using the delta method. We performed simulation experiments with normal and non-normal data, and investigated the effect of sample size and prevalence. Our results indicated that the semi-parametric binormal approach provided AUCPR estimates with small bias and confidence intervals with acceptable coverage when the sample size was large, and the performance of the binormal model was comparable to or better than alternative methods evaluated in this study when the sample size was small. We conclude that the semi-parametric binormal model can be used to accurately estimate the AUCPR, and that the confidence intervals derived from the model can be at least as accurate as from other alternatives, even for non-normal decision variable distributions.

Citation Download Citation

Berkman Sahiner, Weijie Chen, Aria Pezeshk, and Nicholas Petrick "Semi-parametric estimation of the area under the precision-recall curve", Proc. SPIE 9787, Medical Imaging 2016: Image Perception, Observer Performance, and Technology Assessment, 97870D (24 March 2016); https://doi.org/10.1117/12.2216434

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available