A novel Error-Correcting Output Codes algorithm based on genetic programming

Highlights

• A novel ECOC algorithm is proposed based on Genetic Programming.

• Each tree-structure individual represents a solution for a multiclass problem.

• A new mutation operator is designed to maintain the legality of the individual.

• A local optimization algorithm tackles tough classes by adding proper columns.

• A new decoding method is proposed to reduce errors in the decoding process.

Abstract

Error-Correcting Output Codes (ECOC) is widely used in the field of multiclass classification. As an optimal codematrix is key to the performance of an ECOC algorithm, this paper proposes a genetic programming (GP) based ECOC algorithm (GP-ECOC). In the design of individual of our GP, each terminal node represents a class, and nonterminal nodes combine the classes in their child nodes. In this way, an individual is a class combination tree, and represents an ECOC codematrix. A legality checking process is embedded in our algorithm to check each codematrix, so as to ensure each codematrix satisfying ECOC constraints. Those violating the constraints will be corrected by a proposed Guided Mutation operator. Before fitness evaluation, a local optimization algorithm is proposed to append new columns for tough classes, so as to improve the generalization ability of each individual and accelerate the evolutionary speed. In this way, our GP can evolve optimal codematrices through the evolutionary process. Experiments show that compared with other ensemble algorithms, our algorithm can achieve stable and high performances with relatively small ensemble scales on various UCI data sets. To the best of our knowledge, it is the first time that GP has been applied to implement the ECOC encoding algorithm. Our Python code is available at https://github.com/samuellees/gpecoc.

Introduction

In the pattern recognition research field, the multi-class classification problem refers to the task of classifying instances into one of N classes, where N > 2. Due to the effect of class overlapping and class imbalance, the multiclass classification problem still remains to be a tough task nowadays [1].

Instead of tackling multiple classes directly, an alternative solution is the divide and conquer method. That is, a multiclass problem is split into several binary class problems firstly, and then the solutions for these problems are combined to produce a final decision for the original problem. The most widely used framework for this strategy is Error-Correcting Output Codes (ECOC) [2], which originates from the error correcting code scheme in digital communication field [3]. With a proper coding scheme, an ECOC algorithm promises good performances in the multiclass classification tasks, so it has been applied to various fields successfully, such as traffic sign recognition [4], face recognition [5], and action recognition [6].

The key to an ECOC algorithm is the generation of discriminant codematrix in the encoding process. However, it was proved in Ref. [7] that the design of an optimum codematrix is a NP problem. So many researchers had been attracted to this researcher field, and tried to develop various encoding strategies to produce optimal codematrices. As a famous optimization algorithm, Genetic Algorithm (GA) had been successfully applied to optimize codematrices by treating each individual as a codematrix, aiming to minimize the size of codematrices without losing generalization ability [8].

In contrast to the wide use of GA in ECOC framework, no Genetic Programming (GP) based ECOC algorithm has been proposed till now. Compared with the linear structure of GA, the tree structure offers GP higher flexibility, especially in the pattern recognition field [9,10]. Nowadays, GP had been applied to solve diverse problems successfully, including the design of classifier for both semi-supervised learning [11] and supervised learning [12,13], the fusion of base learners [14], feature selection [15,16] and feature extraction [12,17]. Inspired by its significant power, this paper proposes a novel GP based ECOC algorithm (GP-ECOC for short).

In our GP framework, the terminal set is the original class set, and the nonterminal set contains two units: the combination unit and the feature selection unit. With this setting, a nonterminal node combines the classes included in its child nodes, and assigns a proper feature subspace for the associated classifier (dichotomizers). So each individual consists of a set of class partition schemes, and represents an ECOC codematrix. To ensure there is no codematrices breaking ECOC codematrix constraints, a legality check process is carried out right after the generation of new individual. Individuals violating the ECOC constraints are illegal, and will be corrected by the proposed guided mutation operator. With F-score index used as the fitness function, individuals’ fitness values are calculated through an ECOC decoding process. Furthermore, to accelerate the evolutionary speed, an effective local optimization algorithm is designed to add proper columns to a codematrix, aiming to tackle tough classes and to enhance the generalization ability of each individual. In the GP, individuals undergo the crossover and mutation operations to produce new generation, the population evolves towards optimal codematrices. To verify our algorithm’s performance, experiments are carried out on twelve UCI data sets, and results reveal that compared with ten ECOC algorithms and five state-of-art ensemble learning algorithms, our GP-ECOC can achieve satisfactory performance with compact ensemble size.

The rest of this paper is organized as follows: Section 2 introduces the background of ECOC and GP. Section 3 describes the framework of our GP, and gives the details about the design of genetic operators and the local optimization algorithm. In section 4, experiments are carried out to compare our algorithm with some widely deployed ensemble algorithms on UCI data sets. Experimental results are analyzed along with discussions. Finally, section 5 concludes this paper and points out the future work direction.