Abstract
The problem of comparing the accuracy of diagnostic tests is usually carried out through the comparison of the corresponding receiver operating characteristic (ROC) curves. This matter has been approached from different perspectives. Usually, ROC curves are compared through their respective areas under the curve, but in cases where there is no uniform dominance between the involved curves other procedures are preferred. Although the asymptotic distributions of the statistics behind these methods are, in general, known, resampling plans are considered. With the purpose of comparing the performance of different approaches, with different ways of calibrating the distribution of the tests, a simulation study is carried out in order to investigate the statistical power and the nominal level of each methodology.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Antoch J, Prchal L, Sarda P (2010) Nonparametric comparison of ROC curves: testing equivalence. In: Nonparametrics and robustness in modern statistical inference and time series analysis: a Festschrift in honor of Professor Jana Jurečková, Institute of Mathematics and Statistics Collection, vol 7. Institute of Mathematics and Statistics, Beachwood, pp 12–24
Bamber D (1975) The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psychol 12(4):387–415
Bandos AI, Rockette HE, Gur D (2005) A permutation test sensitive to differences in areas for comparing ROC curves from a paired design. Stat Med 24(18):2873–2893
Braun TM, Alonzo TA (2008) A modified sign test for comparing paired roc curves. Biostatistics 9(2):364–372
DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44(3):837–845
Gonçalves L, Subtil A, Oliveira MR, Bermudez P (2014) Roc curve estimation: an overview. REVSTAT 12(1):1–20
Good P (2013) Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer, Berlin
Green D, Swets J (1966) Signal detection theory and psychophysics. Peninsula, Los Altos
Hanley JA, McNeil BJ (1983) A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148(3):839–843
Hsieh F, Turnbull BW (1996) Nonparametric and semiparametric estimation of the receiver operating characteristic curve. Ann Stat 24(1):25–40
Jokiel-Rokita A, Pulit M (2013) Nonparametric estimation of the ROC curve based on smoothed empirical distribution functions. Stat Comput 23(6):703–712
Krzanowski WJ, Hand DJ (2009) ROC curves for continuous data. CRC Press, Boca Raton
Lloyd CJ (1998) Using smoothed receiver operating characteristic curves to summarize and compare diagnostic systems. J Am Stat Assoc 93(444):1356–1364
Lloyd CJ, Yong Z (1999) Kernel estimators of the ROC curve are better than empirical. Stat Probab Lett 44(3):221–228
Martínez-Camblor P (2007) Comparación de pruebas diagnósticas desde la curva ROC. Rev Colomb Estad 30(2):163–176
Martínez-Camblor P, Corral N (2012) A general bootstrap algorithm for hypothesis testing. J Stat Plan Inference 142(2):589–600
Martínez-Camblor P, Carleos C, Corral N (2011) Powerful nonparametric statistics to compare \(k\) independent roc curves. J Appl Stat 38(7):1317–1332
Martínez-Camblor P, Carleos C, Corral N (2013) General nonparametric ROC curve comparison. J Korean Stat Soc 42(1):71–81
Pepe MS (2003) The statistical evaluation of medical tests for classification and prediction. Oxford University Press, Cary
Pérez-Fernández S (2017) nsROC: non-standard ROC curve analysis. R package version 1.0. https://CRAN.R-project.org/package=nsROC
R Core Team (2015) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna. https://www.R-project.org/
Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, Mller M (2011) pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics 12:77
Sheather SJ, Jones MC (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B Methodol 53(3):683–690
Venkatraman E (2000) A permutation test to compare receiver operating characteristic curves. Biometrics 56(4):1134–1138
Venkatraman E, Begg CB (1996) A distribution-free procedure for comparing receiver operating characteristic curves from a paired experiment. Biometrika 83(4):835–848
Wieand S, Gail MH, James BR, James KL (1989) A family of nonparametric statistics for comparing diagnostic markers with paired or unpaired data. Biometrika 76(3):585–592
Acknowledgements
The authors would like to thank associate editor, the Co-Editor and the reviewers for their constructive comments and suggestions on an earlier version of this manuscript. The research of A. Fanjul-Hevia is supported by the Spanish Ministry of Education, Culture and Sport “Beca de Formación de Profesorado Universitario”; fellowship (FPU14/05316). Both authors acknowledged the support from the Spanish Ministry of Economy and Competitiveness, through grant numbers MTM2013-41383P and MTM2016-76969-P, which includes support from the European Regional Development Fund (ERDF). The research of W. González-Manteiga is also supported by the the IAP network P7/06 StUDyS of the Belgian Government (Belgian Science Policy).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Fanjul-Hevia, A., González-Manteiga, W. A comparative study of methods for testing the equality of two or more ROC curves. Comput Stat 33, 357–377 (2018). https://doi.org/10.1007/s00180-017-0783-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-017-0783-6