Improving multiclass pattern recognition with a co-evolutionary RBFNN
Introduction
Multiclass pattern recognition or multiclass classification tasks are ubiquitous in real-world, and there have been growing interests on the multiclass classification problems in the community of machine learning. A multiclass classification task attempts to learn a concept by the training instances with known labels in order to correctly label unknown instances.
There are basically two approaches to solve multiclass classification tasks. One is to create a group of pairwise classifications by matching one class with the others, and the total concept for the multiclass classification task consists of the pairwise classification concepts. The multiclass prediction is done based on the predictions of all two-class concepts. The other is to adapt directly learning algorithms to deal with multiclass problems. The first approach is feasible by restricting each instance taking only one label, and there are some well-known methods such as the error-correcting output codes (Dietterich and Bakiri, 1995), and round robin classification (Furnkranz, 2002). However, this approach fails to consider the correlations between the different labels of each instance in prediction (McCallum, 1999, Elisseeff and Weston, 2002). Nowadays, a lot of methods mainly belonging to the second approach have been proposed specially for multiclass learning tasks (Schapire and Singer, 2000, Kazawa et al., 2005).
The artificial neural network (ANN) has been verified to be a good method to learn multiclass classification concepts, and usually can yield the most predictive concepts for complicated problems. The mostly adopted network topology is radial basis function neural network (RBFNN) (Mitchell, 2003) due to a number of advantages compared with other types of ANNs, such as better prediction capabilities, simpler network structures, and faster learning process. Different variants of RBFNNs were invented to solve multiclass classification problems recently (Asim et al., 1995, Gao and Yang, 2002, Fu and Wang, 2003).
In this article, co-evolutionary RBFNN (CO-RBFNN) is proposed. It attempts to construct the RBFNN models for the multiclass classification problems by using a specially designed co-operative co-evolutionary algorithm (Co-CEA). The Co-CEA algorithm utilizes a divide-and-co-operative mechanism to evolve subpopulations with evolutionary algorithms in parallel (Zhao and Higuchi, 1996). After the initial hidden nodes are obtained by a decaying radius selection clustering (DRSC) method (Berthold and Diamond, 1995), a modified K-means method is employed to divide them further into modules. Then the hidden node modules are used to generate subpopulations for the Co-CEA to carry on the co-operative co-evolutionary searching. Collaborations among the modules are required to obtain complete solutions. The algorithm adopts a matrix-form mixed encoding which includes two determinant parameters of RBFNN’s topology (the network centers and the radius widths) and a control vector. The optimal hidden layer structure is obtained by co-evolving all of the parameters. The CO-RBFNN is applied to 14 real-world classification problems from the UCI machine learning repository, and the CO-RBFNN achieves higher accuracies of prediction with a much simpler network structure in fewer evolutionary trials on nearly all of them.
The rest of the paper is structured as follows: Section 2 presents the fundamentals of our models such as the RBFNN architecture, the Co-CEA as well as the preliminaries of the multiclass learning. In Section 3, the proposed algorithm is described in details. Section 4 illustrates the new algorithm’s performance on 14 UCI datasets in comparison with other learning techniques. Finally, Section 5 summarizes the key points of the paper and presents the concluding remarks.
Section snippets
Multiclass learning
A typical classification problem of n classes based on N patterns or examples with known class membership is defined as follows. Let S = {(x1, y1), (x2, y2), … , (xN, yN)} be a set of N training samples, where xi ∈ Rm, and yi ∈ Y, Y = {1, … , n}. The multiclass learning is to output a multi-label classifier h:X → 2∣Y∣ which optimizes some specific evaluation metric.
Researches on multiclass learning were initially motivated by the difficulty encountered in text categorization due to concept ambiguity, where each
Configuration of RBFNN with Co-CEA
The Co-CEA is particularly well-suited for the configuration of the RBFNN on complicated classification problems. The idea is that the multiclass classification tasks may benefit from partitioning potential solutions into smaller components that can be obtained much easier. Thus, a clustering process is introduced into the standard RBFNN configuration process. After the initial hidden nodes of RBFNN are generated by the DRSC method, the K-means approach is utilized to further group them into
Experimental studies
In order to evaluate the performance of the proposed method on various learning problems, the CO-RBFNN and standard algorithms are tested on 14 datasets from UCI Repository. These datasets are briefly characterized in Table 1. They are real-world classification problems that are different with respect to the number of available patterns (from 101 to 5000), attributes (from 9 to 60), and classes (from 2 to 21).
All datasets except the Sonar and the Vowel were divided into three subsets: 50% of
Conclusions
A hybrid framework for training RBFNN models with the Co-CEA has been presented in this paper. The Co-CEA was introduced to realize the co-evolution of the subpopulations in parallel. After the initial hidden nodes are obtained by the DRSC method, a modified K-means method is employed to divide them further into modules that are used to generate subpopulations for the Co-CEA. Collaborations among the modules are formed to obtain complete solutions. The algorithm adopts a matrix-form mixed
Acknowledgements
The work was supported by the National Science Foundation of China (Grant No.70171002, No. 70571057) and by the Program for New Century Excellent Talents in Universities of China (NCET-05-0253).
References (55)
- et al.
GOLS-Genetic orthogonal least squares algorithm for training RBF networks
Neurocomputing
(2006) - et al.
A real-coded genetic algorithm for training recurrent neural networks
Neural Networks
(2001) - et al.
Radial basis function neural networks for nonlinear fisher discrimination and Neyman–Pearson classification
Neural Networks
(2003) The constraint based decomposition (CBD) training architecture
Neural Networks
(2001)- et al.
A decision-theoretic generalization of on-line learning and an application to boosting
J. Comput. Syst. Sci.
(1997) - et al.
An Efficient MDL Based Construction of RBF Networks
Neural Networks
(1998) - et al.
A fast kernel-based nonlinear discriminant analysis for multi-class problems
Pattern Recognition
(2006) - et al.
DistAI: An inter-pattern distance-based constructive learning algorithm
Intell. Data Anal.
(1999) - et al.
A global learning algorithm for a RBF Network
Neural Networks
(1999) - Annalisa, A., Michelangelo, C., Simon, R., Peter, F., 2004. Redundant feature elimination for multi-class problems. In:...