New Measures of Classifier Competence - Heuristics and Application to the Design of Multiple Classifier Systems

Abstract

In the paper three new methods based on different heuristics for calculating competence of a classifier are proposed. In the common two-step procedure, first the so-called source competence at validation points are determined and next these competence values are extended to the entire feature space. The first proposition of the source competence reflects both the uncertainty of classifier’s decision and its correctness. In the second method the source competence states the difference of membership degrees to the fuzzy sets of competent and incompetent classifiers. The third method is based on the normalized entropy of supports which classifier gives for particular classes. The dynamic selection (DCS) and dynamic ensemble selection (DES) systems were developed using proposed measures of competence. The performance of multiclassifiers was evaluated using six benchmark databases from the UCI Machine Learning Repository. Classification results obtained for five multiclassifier system with selection and fusion strategy were used for a comparison. The experimental results showed that, regardless of the strategy used by the multiclassifier system, the classification accuracy for homogeneous base classifiers has increased when the measure of competence was employed.