Elsevier

Applied Soft Computing

Volume 15, February 2014, Pages 231-242
Applied Soft Computing

An accuracy-oriented self-splitting fuzzy classifier with support vector learning in high-order expanded consequent space

https://doi.org/10.1016/j.asoc.2013.11.004Get rights and content

Highlights

  • A new fuzzy classifier (FC) with expanded high-order consequent space.

  • FC design through self-splitting clustering and SVM in high-order consequent space.

  • The FC achieves high classification accuracy with a small number of rules.

Abstract

This paper proposes a self-splitting fuzzy classifier with support vector learning in expanded high-order consequent space (SFC-SVHC) for classification accuracy improvement. The SFC-SVHC expands the rule-mapped consequent space of a first-order Takagi-Sugeno (TS)-type fuzzy system by including high-order terms to enhance the rule discrimination capability. A novel structure and parameter learning approach is proposed to construct the SFC-SVHC. For structure learning, a variance-based self-splitting clustering (VSSC) algorithm is used to determine distributions of the fuzzy sets in the input space. There are no rules in the SFC-SVHC initially. The VSSC algorithm generates a new cluster by splitting an existing cluster into two according to a predefined cluster-variance criterion. The SFC-SVHC uses trigonometric functions to expand the rule-mapped first-order consequent space to a higher-dimensional space. For parameter optimization in the expanded rule-mapped consequent space, a support vector machine is employed to endow the SFC-SVHC with high generalization ability. Experimental results on several classification benchmark problems show that the SFC-SVHC achieves good classification results with a small number of rules. Comparisons with different classifiers demonstrate the superiority of the SFC-SVHC in classification accuracy.

Introduction

Many classification models have been proposed for pattern classification using numerical data. Examples of the classification models are neural networks (NNs) [1], [2], fuzzy classifiers (FCs) [3], and statistical models [4], [5], such as a mixture of Gaussian classifier [4] and support vector machines (SVMs) [5]. FCs are based on fuzzy if-then classification rules. Neural networks and evolutionary computation approaches are characterized with optimization ability and have been applied to solve different optimization problems [6], [7], [8], [9], [10]. These approaches have also been applied to automate the design of classification rules using numerical data [11], [12], [13], [14], [15], [16], [17], [18], [19], [20]. One popular approach is to bring the learning ability of neural networks into a fuzzy system, and the model designed is usually called a fuzzy neural network (FNN) or a neural fuzzy system [11], [12], [13], [14], [15]. Another popular approach is to use the optimization ability of genetic algorithms (GAs) for fuzzy rule generation [16], [17], [18], [19], [20]. The NN- and GA-based approaches generate fuzzy rules based on empirical risk minimization, which does not account for small structural risk. The generalization performance may be poor when the FC is over-trained.

In contrast to the NN- and GA-based design approaches, a relatively new learning method, the support vector machine (SVM), has been proposed based on the principle of structural risk minimization [5]. Several studies on introducing SVMs into fuzzy-classification-rule generation have been proposed to improve the generalization performance of an FC [21], [22], [23], [24], [25]. This paper proposes a self-splitting fuzzy rule-based classifier with support vector learning in expanded high-order consequent space (SFC-SVHC). Based on the self-splitting clustering algorithm in [25], the antecedent parameters in the SFC-SVHC are determined using a variance-based self-splitting clustering (VSSC) algorithm. The SFC-SVHC differs from the NN, GA, and SVM-based FCs above in rule form and consequent parameter learning. That is, contributions of the SFC-SVHC are twofold. First, FCs typically use zero- or first-order TS-type fuzzy rules [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], where the consequent of a fuzzy rule is a linear decision function and may restrict the rule discrimination capability. For regression problems, the use of different nonlinear functions in the consequent of a fuzzy rule for regression performance improvement has been recently proposed in [26], [27]. This motivates the new idea of expanding the entire rule-mapped consequent space of a first-order TS-type fuzzy classifier, which is used in the SFC-SVHC. Different from the rule forms in previous FCs [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24], [25], the SFC-SVHC expands the entire rule-mapped consequent space of a first-order TS-type fuzzy system via trigonometric function transformations. The expanded rule-mapped consequent (ERMC) space can be regarded as the inclusion of high-order function terms for discrimination capability improvement. Second, the SFC-SVHC uses a linear SVM for consequent parameter optimization in the ERMC space. The cost function used in the optimization considers not only training error but also separation margin. The objective of using a linear SVM is to endow the SFC-SVHC with high generalization ability.

The rest of this paper is organized as follows. Section 2 presents surveys on design of FCs. Section 3 introduces the SFC-SVHC structure. Section 4 describes the SFC-SVHC structure learning using the VSSC algorithm. Section 5 introduces SFC-SVHC parameter learning using a linear SVM. Section 6 demonstrates the SFC-SVHC classification performance by applying it to several benchmark classification problems. This section also compares the performance of the SFC-SVHC with those of different classifiers. Section 7 presents discussion. Finally, Section 8 presents conclusions.

Section snippets

Literature survey

This section presents surveys of different data-driven FCs using NNs, GAs, and statistical learning. NN-based FCs are typically designed based on structure and parameter learning [11], [12], [13], [14], [15]. The neural fuzzy classifier in [11] uses k-means [28] for antecedent parameter initialization. The neural fuzzy classifier in [12] starts with a large rule base and a learning algorithm is used to prune the rules. The use of fuzzy C-means (FCM) [29] to determine the initial antecedent

SFC-SVHC structure and functions

The SFC-SVHC is based on functional expansion of the ERMC space in a first-order TS-type fuzzy system. Each rule in a first-order TS-type fuzzy system is of the following form:

Rule i: IF x1 is Ai1 and x2 is Ai2⋯and xn is Ain, thenyˆ=hi0+j=1nhijxjwhere x1, …, xn are inputs, Aij is a fuzzy set, and hij is a real number. Fig. 1 shows the SFC-SVHC structure, which has a total of six layers. Detailed mathematical functions of each layer are introduced layer by layer as follows.

Layer 1 (input layer

SFC-SVHC antecedent-part learning

The proposed SFC-SVHC uses a VSSC algorithm to determine antecedent fuzzy set parameters mij and di in (4). A cluster in the input space corresponds to a rule in SFC-SVHC. There are no rules in SFC-SVHC initially. All rules (clusters) are generated by performing the clustering algorithm on the input data x. In this algorithm, the variance σ¯i2 of cluster i is computed as follows:σ¯i2=j=1nσij2,where σij is the standard deviation of the cluster in the jth input dimension, i.e.,σij2=1Nix

Parameter learning using support vector machine

The parameter vector aˆm in the ERMC space of an SFC-SVHC is determined by a linear SVM. In this section, basic concepts of classification by a liner SVM are first described. The technique of using a linear SVM for parameter learning in the ERMC space is then introduced.

Experimental results

This section examines the performance of SFC-SVHC by using twenty benchmark pattern classification problems. Table 1 shows the specifications of these data sets. All of the first thirteen data sets are from the University of California, Irvine (UCI) machine learning repository [33], except for Tao which is from another repository [34]. The phoneme, ring, two-norm, and magic data sets are from the KEEL database [35]. For these seventeen data sets, the testing data sets are unavailable.

Performance comparison

The results in Table 3, Table 5 show that at the 5% significance level, the null hypothesis is rejected for each pair of comparison between the SFC-SVHC and the classification models used for comparisons in Experiments 1, 2, and 3, which verifies the significance in performance difference. For the data-driven FCs (SFC-GDHC, MDSOFN, SLAVE, LogitBoost, SGERD, and FH-GMML), the objective is to minimize the training classification error. As previously stated, in the SFC-SVHC, the cost function to

Conclusion

This paper proposes a new fuzzy classifier, the SFC-SVHC, which shows the advantage of good classification performance with a small number of fuzzy rules. In structure, the functional expansion of the first-order TS-type rule-mapped-space to a higher-order ERMC space improves the SFC-SVHC discrimination capability. For structure learning, the VSSC algorithm is used to automatically assign proper fuzzy set positions and shapes in the input space. For parameter learning, the use of linear SVM

References (40)

  • L.C. Cagnina et al.

    Solving engineering optimization problems with KKT Hopfield neural networks

    Int. Rev. of AT Eng.

    (2011)
  • C.H. Hsu et al.

    Evolutionary robot wall-following control using type-2 fuzzy controller with species-DE activated continuous ACO

    IEEE Trans. Fuzzy Syst.

    (2013)
  • J.S. Wang et al.

    Self-adaptive neuro-fuzzy inference systems for classification applications

    IEEE Trans. Fuzzy Syst.

    (Dec. 2002)
  • D. Chakraborty et al.

    A neuro-fuzzy scheme for simultaneous feature selection and fuzzy rule-based classification

    IEEE Trans. Neural Netw.

    (2004)
  • C.F. Juang et al.

    Human body posture classification by a neural fuzzy network and home care system application

    IEEE Trans. Syst. Man. Cyber. A, Syst. Humans

    (2007)
  • G.D. Wu et al.

    A maximizing-discriminability-based self-organizing fuzzy network for classification problems

    IEEE Trans. Fuzzy Syst.

    (2010)
  • C.F. Juang et al.

    Speedup of implementing fuzzy neural networks with high-dimensional inputs through parallel processing on graphic processing units

    IEEE Trans. Fuzzy Syst.

    (2011)
  • A. Gonzàlez et al.

    SLAVE: a genetic learning system based on an iterative approach

    IEEE Trans Fuzzy Syst.

    (1999)
  • H. Ishibuchi et al.

    Hybridization of fuzzy GBML approaches for pattern classification problems

    IEEE Trans. Syst. Man. Cyber. B, Cybern.

    (2005)
  • E.G. Mansoori et al.

    A steady-state genetic algorithm for extracting fuzzy classification rules from data

    IEEE Trans. Fuzzy Syst.

    (2008)
  • Cited by (0)

    View full text