Abstract
The receiver-operating characteristics (stROC) analysis depicts the performance of a population-wise bimodal-distributed, quantitative continuous random variable for distinguishing dichotomous outcomes using a single threshold. However, test results that have three-modal distributions show no-better-than-chance discriminative performance. This study proposes a parameter-free ROC plot analysis with an application to random variables with a population-wise three-modal distribution. A double-threshold ROC plot (dtROC) is constructed by replacing the single threshold by a double threshold. The sensitivity–specificity coordinates are selected for maximizing the sensitivity for a given specificity value. The generalizability of the method is investigated using computational simulations of a mixture of Gaussian distributions. The clinical application is studied by secondary data analysis of a palpation test to locate the C7 spinous process using the modified thorax-rib static method. The simulation shows a poor discrimination performance of the stROC plot (area under the ROC plot [AUROC] < 0.7 in all simulations); better discrimination performance is observed for the dtROC plot (AUROC > 0.9 in 51% of the simulated samples). The accuracy of the palpation test using dtROC (AUROC = 0.652 95%CI = [0.597; 0.775], thresholds = 24.2 to 26.8 cm) was higher as compared to the ROC (AUROC = 0.517 95%CI = [0.385; 0.659]; threshold = 25.45 cm). The dtROC plot analysis outperforms the stROC plot when applied to test results with three-modal distributions.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data availability
The complete dataset of this study is available as Online Resources.
References
Adhia DB, Tumilty S, Mani R et al (2016) Can hip abduction and external rotation discriminate sacroiliac joint pain? Man Ther 21:191–197. https://doi.org/10.1016/j.math.2015.08.002
Altman DG, Bland JM (1994a) Diagnostic tests 3: receiver operating characteristic plots. Br Med J 309:188. https://doi.org/10.1136/bmj.309.6948.188
Altman DG, Bland JM (1994b) Diagnostic tests 1: sensitivity and specificity. Br Med J 308:1552–1552. https://doi.org/10.1136/bmj.308.6943.1552
Altman DG, Bland JM (1994c) Diagnostic tests 2: predictive values. Br Med J 309:102. https://doi.org/10.1136/bmj.309.6947.102
Ashman KA, Bird CM, Zepf SE (1994) Detecting bimodality in astronomical datasets. Astron J 108:2348. https://doi.org/10.1086/117248
Aune D, Sen A, Prasad M et al (2016) BMI and all cause mortality: systematic review and non-linear dose-response meta-analysis of 230 cohort studies with 3.74 million deaths among 30.3 million participants. BMJ 94:2156. https://doi.org/10.1136/bmj.i2156
Behrens JT (1997) Principles and procedures of exploratory data analysis. Psychol Methods 2:131–160. https://doi.org/10.1037/1082-989X.2.2.131
Beurskens AJ, de Vet HC, Köke AJ (1996) Responsiveness of functional status in low back pain: a comparison of different instruments. Pain 65:71–76
Chila AG (2017) Foundations of osteopathic medicine-Google Books
de Wit LM, van Straten A, van Herten M et al (2009) Depression and body mass index, a u-shaped association. BMC Public Health 9:14. https://doi.org/10.1186/1471-2458-9-14
Efron B, Tibshirani RJ (1998) An introduction to the bootstrap, 1st edn. CRC Press LLC, Boca Raton
Fawcett T (2006) An introduction to ROC analysis. Pattern Recognit Lett 27:861–874. https://doi.org/10.1016/j.patrec.2005.10.010
Ferreira APA, Póvoa LC, Zanier JFC, Ferreira AS (2017a) Locating the seventh cervical spinous process: accuracy of the thorax-rib static method and the effects of clinical data on its performance. J Manipulative Physiol Ther 40:98–105. https://doi.org/10.1016/j.jmpt.2016.10.011
Ferreira APA, Póvoa LC, Zanier JFC, Ferreira AS (2017b) Locating the seventh cervical spinous process: development and validation of a multivariate model using palpation and personal information. J Manipulative Physiol Ther 40:89–97. https://doi.org/10.1016/j.jmpt.2016.10.012
Gabel CP, Melloh M, Yelland M et al (2011) Predictive ability of a modified ÖRebro Musculoskeletal Pain Questionnaire in an acute/subacute low back pain working population. Eur Spine J 20:449–457. https://doi.org/10.1007/s00586-010-1509-4
George SZ, Fritz JM, Childs JD (2008) Investigation of elevated fear-avoidance beliefs for patients with low back pain: a secondary analysis involving patients enrolled in physical therapy clinical trials. J Orthop Sport Phys Ther 38:50–58. https://doi.org/10.2519/jospt.2008.2647
Han SS, Ahn SY, Ryu J et al (2014) U-shape relationship of white blood cells with acute kidney injury and mortality in critically Ill patients. Tohoku J Exp Med 232:177–185. https://doi.org/10.1620/tjem.232.177
Hand DJ (2009) Measuring classifier performance: a coherent alternative to the area under the ROC curve. Mach Learn 77:103–123. https://doi.org/10.1007/s10994-009-5119-5
Hanley JA (1988) The robustness of the “Binormal” assumptions used in fitting ROC curves. Med Decis Mak 8:197–203. https://doi.org/10.1177/0272989X8800800308
Hanley JA (1996) The use of the “binormal” model for parametric ROC analysis of quantitative diagnostic tests. Stat Med 15:1575–1585. https://doi.org/10.1002/(SICI)1097-0258(19960730)15:14%3c1575::AID-SIM283%3e3.0.CO;2-2
Heuch I, Heuch I, Hagen K, Zwart JA (2016) Is there a U-shaped relationship between physical activity in leisure time and risk of chronic low back pain? a follow-up in the HUNT Study. BMC Public Health 16:306. https://doi.org/10.1186/s12889-016-2970-8
Hilden J (1991) The area under the ROC curve and its competitors. Med Decis Mak 11:95–101. https://doi.org/10.1177/0272989X9101100204
Ho KM (2017) Effect of non-linearity of a predictor on the shape and magnitude of its receiver-operating-characteristic curve in predicting a binary outcome. Sci Rep 7:1–7. https://doi.org/10.1038/s41598-017-10408-9
Kikuya M, Staessen JA, Ohkubo T et al (2007) Ambulatory arterial stiffness index and 24-hour ambulatory pulse pressure as predictors of mortality in Ohasama, Japan. Stroke 38:1161–1166. https://doi.org/10.1161/01.STR.0000259604.67283.69
Kusnierczyk W (2015) Package ‘rbenchmark.’ 7
Lee W-C, Hsiao CK (1996) Alternative summary indices for the receiver operating characteristic curve. Epidemiology 7:605–611. https://doi.org/10.1097/00001648-199611000-00007
MacDonald P (2018) Package ‘ mixdist.’ 34
Mossman D (1999) Three-way ROCs. Med Decis Mak 19:78–89. https://doi.org/10.1177/0272989X9901900110
Nakas CT (2014) Developments in roc surface analysis and assessment of diagnostic markers in three-class classification problems. Revstat Stat J 12:43–65
Nakas CT, Reiser B (2018) Editorial for the special issue of “Statistical Methods in Medical Research” on “Advanced ROC analysis.” Stat Methods Med Res 27:649–650. https://doi.org/10.1177/0962280217742536
Nakas CT, Yiannoutsos CT (2004) Ordered multiple-class ROC analysis with continuous measurements. Stat Med 23:3437–3449. https://doi.org/10.1002/sim.1917
Parodi S, Pistoia V, Muselli M (2008) Not proper ROC curves as new tool for the analysis of differentially expressed genes in microarray experiments. BMC Bioinf 9:410. https://doi.org/10.1186/1471-2105-9-410
Pesce LL, Metz CE, Berbaum KS (2010) On the Convexity of ROC Curves Estimated from Radiological Test Results. Acad Radiol 17:960-968.e4. https://doi.org/10.1016/j.acra.2010.04.001
R Core Team (2019) R: A language and environment for statistical computing
Rücker G, Steinhauser S, Schwarzer G (2019) Meta-analysis of diagnostic accuracy studies with several cutpoints
Saito T, Rehmsmeier M (2015) The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PLoS ONE 10:1–21. https://doi.org/10.1371/journal.pone.0118432
Scurfield BK (1996) Multiple-event forced-choice tasks in the theory of signal detectability. J Math Psychol 40:253–269. https://doi.org/10.1006/jmps.1996.0024
Smits N (2010) A note on Youden’s Jand its cost ratio. BMC Med Res Methodol 10:89. https://doi.org/10.1186/1471-2288-10-89
Stratford PW, Binkley J, Solomon P et al (1994) Assessing change over time in patients with low back pain. Phys Ther 74:528–533. https://doi.org/10.1093/ptj/74.6.528
Tukey JW (1980) We need both exploratory and confirmatory. Am Stat 34:23–25. https://doi.org/10.1080/00031305.1980.10482706
Yarnold PR (2014) UniODA versus ROC analysis: computing the “optimal” cut -point. Optim Data Anal 3:117–120
Yin J, Nakas CT, Tian L, Reiser B (2018) Confidence intervals for differences between volumes under receiver operating characteristic surfaces (VUS) and generalized Youden indices (GYIs). Stat Methods Med Res 27:675–688. https://doi.org/10.1177/0962280217740787
Youden WJ (1950) Index for rating diagnostic tests. Cancer 3:32–35. https://doi.org/10.1002/1097-0142(1950)3:1%3c32::AID-CNCR2820030106%3e3.0.CO;2-3
Zhang C, Mapes BE, Soden BJ (2003) Bimodality in tropical water vapour. Q J R Meteorol Soc 129:2847–2866. https://doi.org/10.1256/qj.02.166
Acknowledgements
The authors would like to thank the reviewers for the valuable discussion and contributions to the revised version of this report.
Funding
This study was supported by the Fundação Carlos Chagas Filho de Amparo à Pesquisa do Estado do Rio de Janeiro (FAPERJ) [Grant Number E-26/202.769/2015] and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior-Brasil (CAPES)-Finance Code 001.
Author information
Authors and Affiliations
Contributions
ASF performed a review of the topic, implemented the computational routines, wrote and revised the manuscript. NMF tested the computational routines, wrote and revised the manuscript. APF tabulated the data, wrote and revised the manuscript. ASF, NMF, and APA contributed to the writing of the manuscript and provided critical revision of the article. All the authors approved the final version of the manuscript.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Code availability
The complete set of routines of this study is available as Online Resources.
Consent to participate
Not applicable (secondary data analysis of public data set).
Consent for publication
Not applicable (secondary data analysis of public data set).
Ethics approval
Not applicable (secondary data analysis of public data set).
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Ferreira, A.D.S., Meziat-Filho, N. & Ferreira, A.P.A. Double threshold receiver operating characteristic plot for three-modal continuous predictors. Comput Stat 36, 2231–2245 (2021). https://doi.org/10.1007/s00180-021-01080-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-021-01080-9