Improving multiclass pattern recognition with a co-evolutionary RBFNN

https://doi.org/10.1016/j.patrec.2007.10.019Get rights and content

Abstract

A new hybrid scheme of the radial basis function neural network (RBFNN) model and the co-operative co-evolutionary algorithm (Co-CEA) is presented for multiclass classification tasks. This combination of the conventional RBFNN training algorithm and the proposed Co-CEA enforces the strength of both methods. First, the decaying radius selection clustering (DRSC) method is used to obtain the initial hidden nodes of the RBFNN model, which are further partitioned into modules of hidden nodes by the K-means method. Then, subpopulations are initialized on modules, and the Co-CEA evolves all subpopulations to find the optimal RBFNN structural parameters. Matrix-form mixed encoding and special crossover and mutation operators are designed. Finally, the proposed algorithm is tested on 14 real-world classification problems from the UCI machine learning repository, and experimental results illustrate that the algorithm is able to produce RBFNN models that have better prediction accuracies and simpler structures than conventional algorithms of classification.

Introduction

Multiclass pattern recognition or multiclass classification tasks are ubiquitous in real-world, and there have been growing interests on the multiclass classification problems in the community of machine learning. A multiclass classification task attempts to learn a concept by the training instances with known labels in order to correctly label unknown instances.

There are basically two approaches to solve multiclass classification tasks. One is to create a group of pairwise classifications by matching one class with the others, and the total concept for the multiclass classification task consists of the pairwise classification concepts. The multiclass prediction is done based on the predictions of all two-class concepts. The other is to adapt directly learning algorithms to deal with multiclass problems. The first approach is feasible by restricting each instance taking only one label, and there are some well-known methods such as the error-correcting output codes (Dietterich and Bakiri, 1995), and round robin classification (Furnkranz, 2002). However, this approach fails to consider the correlations between the different labels of each instance in prediction (McCallum, 1999, Elisseeff and Weston, 2002). Nowadays, a lot of methods mainly belonging to the second approach have been proposed specially for multiclass learning tasks (Schapire and Singer, 2000, Kazawa et al., 2005).

The artificial neural network (ANN) has been verified to be a good method to learn multiclass classification concepts, and usually can yield the most predictive concepts for complicated problems. The mostly adopted network topology is radial basis function neural network (RBFNN) (Mitchell, 2003) due to a number of advantages compared with other types of ANNs, such as better prediction capabilities, simpler network structures, and faster learning process. Different variants of RBFNNs were invented to solve multiclass classification problems recently (Asim et al., 1995, Gao and Yang, 2002, Fu and Wang, 2003).

In this article, co-evolutionary RBFNN (CO-RBFNN) is proposed. It attempts to construct the RBFNN models for the multiclass classification problems by using a specially designed co-operative co-evolutionary algorithm (Co-CEA). The Co-CEA algorithm utilizes a divide-and-co-operative mechanism to evolve subpopulations with evolutionary algorithms in parallel (Zhao and Higuchi, 1996). After the initial hidden nodes are obtained by a decaying radius selection clustering (DRSC) method (Berthold and Diamond, 1995), a modified K-means method is employed to divide them further into modules. Then the hidden node modules are used to generate subpopulations for the Co-CEA to carry on the co-operative co-evolutionary searching. Collaborations among the modules are required to obtain complete solutions. The algorithm adopts a matrix-form mixed encoding which includes two determinant parameters of RBFNN’s topology (the network centers and the radius widths) and a control vector. The optimal hidden layer structure is obtained by co-evolving all of the parameters. The CO-RBFNN is applied to 14 real-world classification problems from the UCI machine learning repository, and the CO-RBFNN achieves higher accuracies of prediction with a much simpler network structure in fewer evolutionary trials on nearly all of them.

The rest of the paper is structured as follows: Section 2 presents the fundamentals of our models such as the RBFNN architecture, the Co-CEA as well as the preliminaries of the multiclass learning. In Section 3, the proposed algorithm is described in details. Section 4 illustrates the new algorithm’s performance on 14 UCI datasets in comparison with other learning techniques. Finally, Section 5 summarizes the key points of the paper and presents the concluding remarks.

Section snippets

Multiclass learning

A typical classification problem of n classes based on N patterns or examples with known class membership is defined as follows. Let S = {(x1, y1), (x2, y2),  , (xN, yN)} be a set of N training samples, where xi  Rm, and yi  Y, Y = {1,  , n}. The multiclass learning is to output a multi-label classifier h:X  2Y which optimizes some specific evaluation metric.

Researches on multiclass learning were initially motivated by the difficulty encountered in text categorization due to concept ambiguity, where each

Configuration of RBFNN with Co-CEA

The Co-CEA is particularly well-suited for the configuration of the RBFNN on complicated classification problems. The idea is that the multiclass classification tasks may benefit from partitioning potential solutions into smaller components that can be obtained much easier. Thus, a clustering process is introduced into the standard RBFNN configuration process. After the initial hidden nodes of RBFNN are generated by the DRSC method, the K-means approach is utilized to further group them into

Experimental studies

In order to evaluate the performance of the proposed method on various learning problems, the CO-RBFNN and standard algorithms are tested on 14 datasets from UCI Repository. These datasets are briefly characterized in Table 1. They are real-world classification problems that are different with respect to the number of available patterns (from 101 to 5000), attributes (from 9 to 60), and classes (from 2 to 21).

All datasets except the Sonar and the Vowel were divided into three subsets: 50% of

Conclusions

A hybrid framework for training RBFNN models with the Co-CEA has been presented in this paper. The Co-CEA was introduced to realize the co-evolution of the subpopulations in parallel. After the initial hidden nodes are obtained by the DRSC method, a modified K-means method is employed to divide them further into modules that are used to generate subpopulations for the Co-CEA. Collaborations among the modules are formed to obtain complete solutions. The algorithm adopts a matrix-form mixed

Acknowledgements

The work was supported by the National Science Foundation of China (Grant No.70171002, No. 70571057) and by the Program for New Century Excellent Talents in Universities of China (NCET-05-0253).

References (55)

  • R. Asim et al.

    An algorithm to generate radial basis function (RBF)-like nets for classification problem

    Neural Networks

    (1995)
  • M.R. Berthold et al.

    Boosting the performance of RBF networks with dynamic decay adjustment

  • E. Cantú-Paz et al.

    Inducing oblique decision trees with evolutionary algorithms

    IEEE Trans. Evolution. Comput.

    (2003)
  • S. Chen et al.

    Combined genetic algorithm optimization and regularized orthogonal least square learning for radial basis function networks

    IEEE Trans. Neural Networks

    (1999)
  • T.G. Dietterich

    An experimental comparison of three methods for constructing ensembles of decision trees: bagging, boosting, and randomization

    Machine Learning

    (2000)
  • T.G. Dietterich et al.

    Solving multiclass learning problems via error-correcting output codes

    J. Artif. Intell. Res.

    (1995)
  • G. Eibl et al.

    Multiclass boosting for weak classifiers

    J. Machine Learning Res.

    (2005)
  • Eklund, W.P., Hoang, A., 1999. Comparative study of public-domain supervised machine-learning accuracy on the UCI...
  • A. Elisseeff et al.

    A kernel method for multi-labelled classification

  • R. Eriksson et al.

    Cooperative coevolution in inventory control optimisation

  • E. Frank et al.

    Improving on bagging with input smearing

    PAKDD, Lect. Notes Artif. Intell.

    (2006)
  • E. Frank et al.

    Generating accurate rule sets without global optimization

  • E. Frank et al.

    Using model trees for classification

    Machine Learning

    (1998)
  • Y. Freund et al.

    The Alternating decision tree learning algorithm

  • N. Friedman et al.

    Bayesian network classifiers

    Machine Learning

    (1997)
  • J. Friedman et al.

    Additive logistic regression: a statistical view of boosting

    Ann. Statist.

    (2000)
  • X.J. Fu et al.

    Data dimensionality reduction with application to simplifying RBF network structure and improving classification performance

    IEEE Trans. System Man Cybernet.-Part B

    (2003)
  • Cited by (0)

    View full text