High against Low Quantile Comparison for Biomarker and Classifier Evaluation

Abstract

We propose a method to check whether a – usually high – quantile from a sample is smaller than another – usually low – quantile from another sample. It is only required that the two samples come from (independent) continuous, but not necessarily the same distributions. The proposed method is based on confidence intervals (CI) and can be used for the evaluation of classifiers for two class problems as an alternative to ROC analysis and in the context of biomarkers. The quantiles correspond to the acceptable false positive and false negative rates. We also introduce a visualisation that shows how the probabilities for the overlap of CIs we compute change when the chosen quantiles are varied. This can help to select a suitable combination of quantiles, i.e. the acceptable false positive and false negative rate.