Designing a classifier by a layered multi-population genetic programming approach

Abstract

This paper proposes a method called layered genetic programming (LAGEP) to construct a classifier based on multi-population genetic programming (MGP). LAGEP employs layer architecture to arrange multiple populations. A layer is composed of a number of populations. The results of populations are discriminant functions. These functions transform the training set to construct a new training set. The successive layer uses the new training set to obtain better discriminant functions. Moreover, because the functions generated by each layer will be composed to a long discriminant function, which is the result of LAGEP, every layer can evolve with short individuals. For each population, we propose an adaptive mutation rate tuning method to increase the mutation rate based on fitness values and remaining generations. Several experiments are conducted with different settings of LAGEP and several real-world medical problems. Experiment results show that LAGEP achieves comparable accuracy to single population GP in much less time.

Introduction

Genetic programming (GP) [1], an important evolutionary computation (EC) technique, has developed rapidly in recent years. Researchers have proposed creative ideas to improve the effectiveness and efficiency of GP, such as new fitness functions, new architectures, and new individual expressions.

Traditionally, GP works with a single population. Multi-population GP (MGP) [3], [18], which employs several populations to discover optimal solutions, has been proposed and developed. Many different topologies of MGP have been proposed, such as the circle topology and the random topology. Fig. 1 shows the circle topology where circles stand for populations [3]. An important characteristic of MGP is migration. This means that individuals can be transmitted from one population to another. The arrows in Fig. 1 indicate the migration direction. Fernández et al. [18] performed several experiments with parallel and distributed GP (PADGP), isolated multi-population GP (IMGP), where “isolated” means that there is no migration between populations, and traditional single population GP. Their experiments show that PADGP and IMGP usually obtain better performance than traditional single population GP.

Many classifiers have been developed based on GP in recent years [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [19], [21]. To generate classification rules, Freitas [6] proposed the tuple-set-descriptor (TSD), a logical formula to represent an individual. Kotani and Sherrah [9], [13] used GP to perform feature selection before using other classification methods. Multi-category classification problems are more difficult than two-class classification problems. Kishore et al. [7] and the present authors [4] have considered such a problem as multiple two-class classification problems and then generated corresponding expressions or discriminant functions. These methods need k runs for a k-class classification problem. Muni et al. [12] proposed a novel method to solve k-class classification problems in a single run. Each individual in their work is represented by a multi-tree. Evolving one individual is equivalent to evolving k trees simultaneously. Loveard and Ciesielski [11] proposed five methods for solving multi-category classification problems including binary decomposition, static range selection, dynamic range selection, class enumeration, and evidence enumeration. Brameier and Banzhaf [3] used linear GP and MGP techniques. Individuals are represented as strings and can be transmitted between demes, i.e. subpopulations, according to their fitness value. Tsakonas [21] compares four different structures evolved by GP in several different classification problems.

Using functional expressions to represent individuals is effective in GP [4], [7], [10]. The tree structure is a common data structure for functional expressions. However, two problems occur when GP is employed to generate functional expressions. First, it is difficult to choose appropriate operations for a given problem because characteristics of the problem are completely unknown. If the operator set contains many operations, there is a greater possibility of discovering optimal solutions, but the searching space becomes larger and therefore may become impracticable. Fortunately, as shown in Ref. [7], GP with an operation set comprising only basic arithmetic operations, i.e. $\{+,-,\times ,÷\}$, generates results comparable to that with an operation set comprising additional operations. Second, it is difficult to know the proper length of an individual because there is no prior knowledge about optimal solutions. The predefined individual length, as the length of a string-expression individual or the number of available nodes of a tree-expression individual, is usually chosen according to heuristic or empirical assumptions. The following is an example of a classification problem containing 64 dimensional data, i.e. a training instance x is represented by $x=(a_1,a_2,\dots ,a_{64})$. Suppose that an optimal solution F is known as $F={\prod }_{i=1}^{64}{a}_i$. F can be represented as a skew binary tree with a height of 64 or a balanced binary tree with a height of seven, as shown in Fig. 2. An individual can contain at most $2^{64}-1$ nodes if the predefined maximum depth is 64. A population containing so many large trees is highly complex and is thereby impracticable. On the other hand, if the predefined maximum depth is fixed at seven, it is very difficult to generate the ideal balanced tree. Moreover, the function $F$ will never be obtained if the maximum depth is less than seven.

Using an acceptable and practicable individual size is a simple but dangerous way to avoid this problem. This problem has motivated us to develop this work. Since a long function can be viewed as a composition of a number of small functions, it is possible to combine a number of small GP solutions into a large one. Therefore, it is desirable to generate those small solutions with a practicable size of individuals and then use them to compose a larger solution. For example, consider the above function F and two functions $B={\prod }_{i=1}^{32}{a}_i$ and $C={\prod }_{i=33}^{64}{a}_i$. Clearly, F can be represented as ($B\times C)$, as shown in Fig. 3, where the tree representations of B and C have at most a height of 32 rather than 64. Functions B and C can be generated by two separate GPs and then are combined together to form F. Here we attempt to develop a method by which we can determine a proper node to combine small functions, for example, the shaded $\times$ operation in Fig. 3.

The method proposed in this paper is called layered genetic programming (LAGEP). It is a method based on MGP. LAGEP arranges populations in a layered architecture. Populations in the same layer evolve with identical training set and store the results of their best individuals into a dataset; this dataset becomes a new training set for the successive layer. After all layers have finished the evolution process, the output of the final layer is used as the result of LAGEP.

The rest of this paper is organized as follows. Section 2 describes the details of LAGEP. Section 3 presents and discusses the experimental results on selected classification problems. Conclusions are drawn in Section 4.