Elsevier

Applied Soft Computing

Volume 8, Issue 3, June 2008, Pages 1222-1231
Applied Soft Computing

Type-2 fuzzy logic-based classifier fusion for support vector machines

https://doi.org/10.1016/j.asoc.2007.02.019Get rights and content

Abstract

As a machine-learning tool, support vector machines (SVMs) have been gaining popularity due to their promising performance. However, the generalization abilities of SVMs often rely on whether the selected kernel functions are suitable for real classification data. To lessen the sensitivity of different kernels in SVMs classification and improve SVMs generalization ability, this paper proposes a fuzzy fusion model to combine multiple SVMs classifiers. To better handle uncertainties existing in real classification data and in the membership functions (MFs) in the traditional type-1 fuzzy logic system (FLS), we apply interval type-2 fuzzy sets to construct a type-2 SVMs fusion FLS. This type-2 fusion architecture takes considerations of the classification results from individual SVMs classifiers and generates the combined classification decision as the output. Besides the distances of data examples to SVMs hyperplanes, the type-2 fuzzy SVMs fusion system also considers the accuracy information of individual SVMs. Our experiments show that the type-2 based SVM fusion classifiers outperform individual SVM classifiers in most cases. The experiments also show that the type-2 fuzzy logic-based SVMs fusion model is better than the type-1 based SVM fusion model in general.

Introduction

Support vector machines have been continuously gaining popularity as a machine-learning tool in the fields of pattern recognition and data classification since they have been developed by Vapnik [1] in 1995. Instead of applying empirical risk minimization (ERM) principle commonly used in the statistical learning methods, SVMs employ structural risk minimization (SRM) principle to achieve better generalization ability, the goal of machine-learning tools, than the conventional machine-learning algorithms, such as neural networks and decision tree.

The foundation of SVMs is based on statistical learning theory. For a binary classification problem with data examples labeled either positive or negative, SVMs aim to find an optimal separating hyperplane which separates the data into the two classes with maximum margin in a high or even infinite transformed feature space, like Hilbert spaces with infinite dimension created by RBF functions. The maximum margin is constituted of a set of positive and negative training examples which are closest to the separating hyperplane called support vectors. The transformation of feature spaces from input spaces can be made through kernel trick, which allows every dot product to be replaced simply by a kernel function. Different kernel functions can be chosen during the SVMs classification, corresponding to the different transformed feature spaces. So kernel functions play an essential role in the SVMs classification since they determine the feature spaces in which data examples are classified and can directly affect the SVMs classification results and performances.

When applying SVMs to solve real classification problems, one has to deal with the practical difficulty: how to select an appropriate kernel function which fits particular data better than any other kernel functions. One obvious way is to try many different kernels and choose the one which works best. But this approach could be time-consuming if the size or the number of attributes of training data is huge. Another less time-consuming way is to randomly choose several SVMs with different kernels and build an ensemble model to combine the different SVMs classifiers and generate a composite classifier. The resulting classifier is probably expected to outperform each of its composing single classifiers because different classifiers might complement each other well. Indeed, this advantage of complementation is an important feature of ensemble methods (by combining classifiers) [2]. It is also the significant difference between the ensemble methods and the exhaustive method mentioned above by simply trying many different SVMs classifiers. Bagging [3] and boosting [4] are two popular ensemble algorithms.

This paper proposes an ensemble model using the knowledge of fuzzy logic system (FLS) to combine multiple SVM classifiers. The model takes the classification results of a data example from different SVMs as the model inputs and generates one output indicating whether the classified data example belongs to positive or negative class. Classification data usually contain noise or outliers that may affect the results of SVMs classification. Therefore, the model inputs, such as SVMs accuracies and classification results, contain certain uncertainties. FLS is famous for dealing with uncertainties and imprecision. Type-1 fuzzy sets handle the uncertainties by using precise and crisp membership functions (MFs) and membership grades of type-1 fuzzy sets are any crisp values in [0,1]. But once the MFs are determined, all the uncertainties disappear [5], [6], [7]. However, the defined MFs in the type-1 FLS might not be the best choice to represent the linguistic variables. Type-2 fuzzy sets are especially useful to handle the situations where the shapes, positions or other parameters of MFs are uncertain. Unlike type-1 FLS, MFs of type-2 fuzzy sets themselves are fuzzy such that membership grades of type-2 fuzzy sets are fuzzy sets in [0,1] instead of crisp values in [0,1]. To better handle the uncertainties in classification data and in MFs, type-2 fuzzy sets and FLS are applied in the paper to construct the ensemble or fusion model for SVMs classifiers. General type-2 FLS is computationally difficult but the process can be simplified a lot if type-2 fuzzy sets are defined as interval type-2 fuzzy sets. So, all the type-2 fuzzy sets defined in the paper are interval type-2 fuzzy sets.

The organization of the paper is as follows. We will review the theories of SVMs in Section 2 and introduce type-2 fuzzy sets and interval type-2 FLS in Section 3. In Section 4, we will describe the interval type-2 SVMs fusion model. In Section 5, we will present the experiments and results. Finally, conclusions are drawn in Section 6.

Section snippets

Support vector machines (SVMs)

In this section, we summarize the basic SVMs theories [1]. Assume there is a training data set S: {(xi,yi)}i=1N, where each input xim and output yi  {±1}. The goal of SVMs is to map the input vector x into a feature space z = ϕ(x) and find an optimal hyperplane wz+b=0 in the feature space to separate the training data into two classes with the maximum margin, where w=i=1Nαiyizi, αi is a set of Lagrange multipliers to the following dual problem:Maximize:W(α)=i=1Nαi12i=j=1Nαiαjyiyj(zizj)

Type-1 versus type-2 fuzzy sets and MFs

When we have difficulty to measure the exact value of one object, we know we can apply type-1 fuzzy sets and FLS to solve the problem and obtain a fuzzy set, which is generally more reasonable than a crisp set [7]. This is the exact motivation that the concept of fuzzy logic was introduced by Zadeh [8]. The theory of fuzzy logic has been applied to many real applications to handle the uncertainties associated with FLS inputs and outputs. However, the ability of type-1 fuzzy sets and FLS to

Type-2 fuzzy SVMs fusion (T2FFSVM) model

Different SVMs classifiers may generate different separating hyperplanes for same training data. When the trained hyperplane models are used to classify unseen data examples, different decisions may be made. Since different decisions might complement each other, if we combine different SVMs together, the composite classifier might outperform all the individual base SVMs. For example, suppose we have two data examples labeled as data1 and data2. Data1 is a cancer tissue data and data2 is a

Experimental results

The type-2 fuzzy SVMs fusion model has been tested in n-fold cross-validation manner: onefold of data as testing data and all the other folds as training data. To obtain the SVMs accuracies needed in the type-2 FLS, we have several choices:

  • The first choice is to use testing accuracies as the accuracy inputs in the type-2 FLS. But apparently, this is not the right choice since testing accuracies should only be used to evaluate how good the model is and in the real applications, the testing

Conclusions

In this paper, we propose a type-2 SVMs fusion model to combine multiple individual SVMs classifiers. This model is constructed using interval type-2 fuzzy sets and interval type-2 FLS. The experimental results show that interval type-2 FLS is a suitable and feasible way to implement ensemble approaches in terms of performance and computational complexity. The proposed type-2 SVMs fusion system demonstrates more stable and more robust generalization ability than individual SVMs. When several

Acknowledgements

This work was supported in part by NIH under P20 GM065762, Georgia Cancer Coalition, and Georgia Research Alliance. Dr. Harrison is a GCC distinguished cancer scholar.

References (18)

  • L. Zadeh

    Fuzzy set

    Inf. Control

    (1965)
  • V. Vapnik

    The Nature of Statistical Learning Theory

    (1995)
  • J. Kittler et al.

    On combining classifiers

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1998)
  • L. Breiman

    Bagging predictors

    Mach. Learn. J.

    (1996)
  • Y. Freund et al.

    Experiments with a new boosting algorithm

  • R. John

    Type 2 fuzzy sets: an appraisal of theory and applications

    Int. J. Uncertainty Fuzziness Knowl. Based Syst.

    (1998)
  • J. Mendel

    Uncertain Rule-Based Fuzzy Logic Systems: Introduction and New Directions

    (2001)
  • N.N. Karnik et al.

    Introduction to type-2 fuzzy logic systems

  • Q. Liang et al.

    Connection admission control in ATM networks using survey-based type-2 fuzzy logic systems

    IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev.

    (2000)
There are more references available in the full text version of this article.

Cited by (58)

  • Proactive, uncertainty-driven queries management at the edge

    2021, Future Generation Computer Systems
  • A comprehensive review on type 2 fuzzy logic applications: Past, present and future

    2020, Engineering Applications of Artificial Intelligence
  • A hybrid model through the fusion of type-2 fuzzy logic systems and extreme learning machines for modelling permeability prediction

    2014, Information Fusion
    Citation Excerpt :

    In general, this loose coupling enables various loose forms of cooperation among modules. Several works have been reported in this direction, some of which include [50,56,64–66]. With the designed framework shown above, the hybridization of type-2 FLS (see description in Section 3.1) and extreme learning machine (see description in Section 3.2), becomes clear to develop and implement.

View all citing articles on Scopus
View full text