Abstract
In many areas of our daily lives (e.g., healthcare), the performance of a binary diagnostic test or classification model is often represented as a curve in a Receiver Operating Characteristic (ROC) plot and a quantity known as the area under the ROC curve (AUC or AUROC). In ROC plots, the main diagonal is often referred to as “chance” or the “random line”. In general, however, this does not correspond to the layperson’s concept of chance or randomness for binary outcomes. Rather, this represents a special case of layperson’s chance, or the ROC curve for a classifier that has the same distribution of scores for the positive class and negative class. Where the ROC curve of a model deviates from the main diagonal, there is information. However, not all information is “useful information” compared to chance, including some areas and points above the diagonal. We define the binary chance baseline to identify areas and points in a ROC plot that are more useful than chance. In this paper, we explain this novel contribution about the state-of-art and provide examples that classify benchmark data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Two probability density functions (PDF) are the same “almost everywhere” if they disagree on, at most, a set of isolated points (more formally, on a set of measure zero). This qualification is, admittedly, somewhat pedantic but necessary because any two such PDFs are effectively the same (and share the same cumulative distribution function). Changing a PDF at only individual points has no actual effect on the corresponding random variable it describes.
References
Althouse, A.D.: Statistical graphics in action: making better sense of the ROC curve. Int. J. Cardiol. 100(215), 9–10 (2016)
Beran, R.: Minimum Hellinger distance estimates for parametric models. Ann. Stat. 5(3), 445–463 (1977)
Carrington, A.M., et al.: A new concordant partial AUC and partial C statistic for imbalanced data in the evaluation of machine learning algorithms. BMC Med. Inform. Decis. Making 20(1), 1–12 (2020)
Chance Noun: In the Cambridge Dictionary. Cambridge University Press. https://dictionary.cambridge.org/dictionary/english/chance
Cohen, J.: A coefficient of agreement for nominal scales. Educ. Psychol. Measur. 20(1), 37–46 (1960)
Cohen, J.: Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol. Bull. 70(4), 213 (1968)
Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley, Hoboken (2012)
Dua, D., Graff, C.: UCI machine learning repository (2017). http://archive.ics.uci.edu/ml
Fawcett, T.: An introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006)
Flach, P.A.: The geometry of ROC space: understanding machine learning metrics through ROC isometrics. In: Proceedings of the Twentieth International Conference on Machine Learning (2003)
Halligan, S., Altman, D.G., Mallett, S.: Disadvantages of using the area under the receiver operating characteristic curve to assess imaging tests: a discussion and proposal for an alternative approach. Eur. Radiol. 25(4), 932–939 (2015). https://doi.org/10.1007/s00330-014-3487-0
Hand, D.J.: Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach. Learn. 77(1), 103–123 (2009). https://doi.org/10.1007/s10994-009-5119-5
Inácio, V., Rodríguez-Álvarez, M.X., Gayoso-Diz, P.: Statistical evaluation of medical tests. Ann. Rev. Stat. Appl. 8, 41–67 (2021)
Lin, J.: Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 37(1), 145–151 (1991). https://doi.org/10.1109/18.61115. http://ieeexplore.ieee.org/document/61115/
Menéndez, M., Pardo, J., Pardo, L., Pardo, M.: The Jensen-Shannon divergence. J. Franklin Inst. 334(2), 307–318 (1997). Publisher: Elsevier
Metz, C.E.: Basic principles of ROC analysis. In: Seminars in Nuclear Medicine, vol. 8, pp. 283–298. Elsevier (1978)
Nielsen, F.: On a variational definition for the Jensen-Shannon symmetrization of distances based on the information radius. Entropy 23(4), 464 (2021)
Obuchowski, N.A.: Receiver operating characteristic curves and their use in radiology. Radiology 229(1), 3–8 (2003). https://doi.org/10.1148/radiol.2291010898
Obuchowski, N.A., Bullen, J.A.: Receiver operating characteristic (ROC) curves: review of methods with applications in diagnostic medicine. Phys. Med. Biol. 63(7), 07TR01 (2018)
Powers, D.M.W.: Evaluation: from precision, recall and F-factor to ROC, informedness, markedness & correlation. Technical report, Flinders University, December 2007
Provost, F., Fawcett, T.: Robust classification for imprecise environments. Mach. Learn. 42, 203–231 (2001). https://doi.org/10.1023/A:1007601015854
Sackett, D.L., Straus, S.: On some clinically useful measures of the accuracy of diagnostic tests. BMJ Evid.-Based Med. 3(3), 68 (1998)
Streiner, D.L., Cairney, J.: What’s under the ROC? An introduction to receiver operating characteristics curves. Can. J. Psychiatry 52(2), 121–128 (2007)
Subtil, F., Rabilloud, M.: An enhancement of ROC curves made them clinically relevant for diagnostic-test comparison and optimal-threshold determination. J. Clin. Epidemiol. 68(7), 752–759 (2015)
Van den Hout, W.B.: The area under an ROC curve with limited information. Med. Decis. Making 23(2), 160–166 (2003). https://doi.org/10.1177/0272989X03251246
Wolberg, W.H., Mangasarian, O.L.: Multisurface method of pattern separation for medical diagnosis applied to breast cytology. Proc. Natl. Acad. Sci. 87(23), 9193–9196 (1990)
Zhou, X.H., McClish, D.K., Obuchowski, N.A.: Statistical Methods in Diagnostic Medicine, vol. 569. Wiley, Hoboken (2002)
Acknowledgements
Parts of this work has received funding by the Austrian Science Fund (FWF), Project: P-32554 “A reference model for explainable Artificial Intelligence in the medical domain”.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Contributions
All authors contributed in writing this article. AC conceived the main ideas initially. In consultation with PF, JP, FM, and NJ various ideas were further developed and refined, with AH and RA providing guidance. Experiments were conducted and coded by AC and FM. All authors reviewed and provided edits to the article.
Rights and permissions
Copyright information
© 2022 IFIP International Federation for Information Processing
About this paper
Cite this paper
Carrington, A.M. et al. (2022). The ROC Diagonal is Not Layperson’s Chance: A New Baseline Shows the Useful Area. In: Holzinger, A., Kieseberg, P., Tjoa, A.M., Weippl, E. (eds) Machine Learning and Knowledge Extraction. CD-MAKE 2022. Lecture Notes in Computer Science, vol 13480. Springer, Cham. https://doi.org/10.1007/978-3-031-14463-9_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-14463-9_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14462-2
Online ISBN: 978-3-031-14463-9
eBook Packages: Computer ScienceComputer Science (R0)
