SCHEP — A Geometric Quality Measure for Regression Rule Sets, Gauging Ranking Consistency Throughout the Real-Valued Target Space

Abstract

As is well known since Fürnkranz and Flach’s 2005 ROC ‘n’ Rule Learning paper [6], rule learning can benefit from result evaluation based on ROC analysis. More specifically, given a (set of) rule(s), the Area Under the ROC Curve (AUC) can be interpreted as the probability that the (best) rule(s) will rank a positive example before a negative example. This interpretation is well-defined (and stimulates the intuition!) for the situation where the rule (set) concerns a classification problem. For a regression problem, however, the concepts of “positive example” and “negative example” become ill-defined, hindering both ROC analysis and AUC interpretation. We argue that for a regression problem, an interesting property to gauge is the probability that the (best) rule(s) will rank an example with a high target value before an example with a low target value. Moreover, it will do so consistently for all possible thresholds separating the target values into the high and the low. For each such threshold, one can retrieve an old-fashioned binary-target ROC curve for a given rule set. Aggregating all such ROC curves, we introduce SCHEP: the Surface of the Convex-Hull-Enclosing Polygon. This is a geometric quality measure, gauging how consistently a given rule (set) performs the aforementioned separation when the threshold is varied through the target space.