Standardized Partition Spaces

Abstract

We propose a standardized partition space (SPS) that offers a unifying framework for the comparison of a wide variety of classification rules. Using SPS, one can define measures for the performance of classifiers w.r.t. goodness concepts beyond the expected rate of correct classifications of the objects of interest. These measures are comparable for rules from so different techniques as support vector machines, neural networks, discriminant analysis, and many more. In particular, we are interested in assessing the reliability of classification rules when used for interpretation of the relationship between the values of predictors and the membership in classes.

We will demonstrate the high potential of SPS for the comparison of classification methods in a simulation study to analyse the following problem:

Given a medium number of predictors, (10–20), and a potentially complex relation between classes and predictors, one would expect flexible classification methods like support vector machines or neural networks to do better than simple methods like e.g. the linear discriminant analysis or classification and regression trees. Nevertheless, one often observes on real data sets, that the simple procedures do pretty well. Our assumption is, that simple methods are more robust against instability, and that the effect of instability superposes the effect of complexity of the relation. By instability we mean the deviation from the assumption that the collected data is some independent and identically distributed sample from some joint distribution of predictors and classes.

We analyse this problem with a simulation study using experimental design.

This work has been supported by the Deutsche Forschungsgemeinschaft,Sonderforschungsbereich 475.