Elsevier

Applied Soft Computing

Volume 26, January 2015, Pages 531-544
Applied Soft Computing

TITS-FM: Transductive incremental Takagi-Sugeno fuzzy models

https://doi.org/10.1016/j.asoc.2014.09.024Get rights and content

Highlights

  • We developed transductive inference model for Takagi-Sugeno fuzzy models.

  • We introduce a novel model for transductive similarity.

  • Transductive similarity improves the accuracy of TS fuzzy models for classification.

  • Developed transductive similarity model is not limited only to TS fuzzy models.

  • Developed model improves the precision of real world recognition tasks.

Abstract

In this paper we present a novel model originating from Takagi-Sugeno fuzzy models. It is based on a concept of transductive similarity, where unlike a simple inductive similarity, it considers also local neighborhood of a given element. Transductive property of a local space is used in an inference process, what allows the technique to be used also in incremental settings. Since incremental model construction brings new challenges, we are unable to use the offline transductive approach as some of the previous works did. The key idea of our model is to adjust activation properties of each rule, based on cross-rule similarities. Our method is capable of using the transductive property for any metric. Besides the final model, we also present several improvements to the transductive similarity technique itself, where we alternate the similarity metric in several ways to better exploit the influence of local neighborhood in the final metric. At the end, we demonstrate a superior performance of our technique over the state-of-the-art techniques build on TS fuzzy models on several machine learning datasets.

Introduction

Similarity is a key concept for a majority of machine learning and patter recognition techniques, which represents the elements as multidimensional vectors. The similarity used in these techniques is frequently quite simple such as the L1 or L2 similarity. This is the most straight forward notion of a similarity, which can be translated from the human knowledge into the machine knowledge. But this kind of similarity might not necessarily be the best way to evaluate the correspondence between two items. It might not be the way, how people understand correspondence between objects. We can see that human notation of similarity is not always in the correspondence with simple metrics used in machine learning and pattern recognition algorithms. Even simple rules such as the triangular inequality can be too limiting for humans and algorithms which do not enforce this inequality can achieve better results than those, which require it [15]. Therefore we seek for a more complex notion of similarity, which can be used.

Specifically we rely on the concept of transductive similarity measure. The transductive inference itself was introduced as a concept by [24] and was used in combination with the SVM classifier in [28], [17]. In our work we understand the concept of the transductive similarity as a similarity induced by the transductive property of the space. More specifically, we say that the similarity measure s between two elements xi and xj is learned through the graph-based transductive learning algorithm. This is in accordance to the notion of the transductive similarity used in [11], who experimentally proved that transductive similarity can improve the accuracy of classification tasks. In case of the transductive similarity applied to individual elements, our intuition is to learn the similarity induced by the shape manifold represented by the feature space elements. The question that we try to answer is, whatever the fuzzy rules also create a manifold in a space and whatever we can learn a better fuzzy membership function based on the transductive property of the space.

Besides the similarity, second important aspect is the notion of belonging to a particular group or a class. In absolute world, this notion represents conditional relationship if x = C then y = V based on binary logic. This notion builds on an assumption that the relationship “belongs to” is absolute. However this is not the way how we understand this relationship in the real life. Given an object, which is “somehow” round and “somehow” red can be considered by different degrees of confidence as either an apple, a red ball or an orange, depending on the value of its properties. This notion is exactly what is expressed in the fuzzy logic and the conditional relationship is rather ifx=ˆCtheny=ˆV where the operator =ˆ tells the amount of correspondence of the variable x to C and the variable y to V. Thus, rather than a binary condition, a fuzzy rule represents variables having a continuous amount of values.

Given these two concepts, we should state a question, if the transductive similarity can help us to improve the performance of a fuzzy-based models. Our hypothesis is that for models based on fuzzy rules we can develop a cross-rule relationships, that can further be used in the transductive similarity to adjust activations (values of x) and results (values of y) of used fuzzy rules. Especially we build on the notion that the relationship x=ˆC expresses a degree of x being C, thus it can be understood as a similarity measure, where value of 1 expresses an absolute similarity and value 0 expresses an absolute dissimilarity.

In this paper we present novel techniques for using the transductive similarity for the inference in TS-fuzzy models. Since the simple transductive similarity model [11] did not prove to be effective for each of the tasks evaluated during our experiments, we propose a new model of the transductive similarity. This model does not need to be necessarily used in the context of TS-fuzzy models. The contributions of this paper are as following:

  • The transductive inference model for incremental TS-fuzzy models

  • 3 novel models for the transductive similarity, which can be combined and yield 7 possible variants

  • Applications of proposed models to tasks of gesture, sketch and object recognition

The originality of this paper is in both covered areas, the transductive similarity and the inference for incremental TS-fuzzy models. Our 3 novel models (can be combined into 1 general model) were not used in any prior works and as shown by our experiments this similarity measure, if used in similarity-based classifiers, yields superior results compared to the baseline techniques. The second major original contribution is the use of the transductive similarity itself during the inference time, giving us our novel transductive inference model for incremental TS-fuzzy models. The only similar work which captures the concept of transductive similarity for fuzzy models is such, that it uses the transductivity only as a similarity concept during the construction of rules, where it serves for a clustering which employs the transductive similarity in a cluster membership function. Therefore our idea is fully novel and also as shown by experiments it improves the performance of incremental TS-fuzzy models.

We evaluate the proposed techniques on 5 standardized datasets used for classification tasks. Further we present qualitative results for applications in 3 domains based on different classification models. To further advocate the proposed similarity models, we use them in a setting independent of fuzzy-based classification.

Our paper is organized as following: Section 2 summarizes previous works on models for transductive similarity, fuzzy models and their combinations, Section 3 introduces the transductive similarity, the baseline fuzzy model and describes our method for the incorporation of the transductive inference for incremental fuzzy models, Section 4 describes the proposed improvement to existing transductive similarity models, Section 5 presents experimental results, an evaluation protocol and applications of our method, Section 6 includes final conclusion and discussion.

Section snippets

Related works

Transductive similarity gives sense in any case, where notion of neighborhood could be considered in a topological space. It was widely used in a range of classifiers [17], [33], [54], [44] and was able to improve the results of these classifiers achieved without transductive property. Besides classification transductive similarity was successfully used in regression tasks [14], [19]. Motivated by these successful applications, we decided to explore a possible gain for incremental fuzzy models.

Transductive inference model

Now we describe our model, which is based on the notion of transductive similarity. At first we introduce the term of transductive similarity itself and building on this we present our new inference model for Takagi-Sugeno fuzzy models induced by the transductive similarity.

Improved transductive similarity measures

In this section we describe three models for transductive similarity, which improve the existing measure as presented in (3). These measures are proposed to capture additional properties of a used datasets and they allow the metric to adapt to a given dataset. Although we are presenting them as separate models, they can be used in a combination one with each other. This provides us with another free parameter which can be adapted to the given dataset.

Results

In this section we focus on an evaluation of our technique with a respect to each of the proposed methods. As a baseline technique we use implementation of TS fuzzy models from [40], which extends model from [1] based on the original online version of TS models presented in [5]. We evaluate each of the different factors of our method separately and in mutual combination giving altogether 8 possible testing settings (+1 for baseline method) as shown in Table 1.

Conclusions

In this work we presented a new inference model for incremental TS-fuzzy models which is based on a transductive similarity. The models brings a concept of transductivity into an inference for an incremental TS-fuzzy models. Unlike previous approaches which use the transductive inference for fuzzy models, our technique does not require access to the whole dataset in advance and can build the transductive model incrementally, one example at a time, starting from scratch. We show, that this

Acknowledgments

This work was supported by the Social Sciences and Humanities Research Council of Canada (SSHRC) as well as the Natural Sciences and Engineering Research Council of Canada (NSERC).

Lukas Tencer comes from Jalovec (Slovakia). He received his B.Sc. degree from Comenius University (Slovakia) in Applied Informatics in 2007 and his M.Sc. Degree from the same institution in Computer Graphics and Geometry in 2009. Since 2011, he joined the Synchromedia Laboratory for Multimedia Communication in Telepresence where he pursues his Ph.D. research, under the supervision of professor Mohamed Cheriet. His research interests include Pattern Recognition, Machine Learning, Similarity

References (55)

  • P. Angelov et al.

    Evolving fuzzy systems from data streams in real-time

  • P. Angelov et al.

    Evolving fuzzy rule-based classifiers

  • P.P. Angelov et al.

    An approach to online identification of Takagi-Sugeno fuzzy models

    IEEE Trans. Syst. Man Cybern. B: Cybern.

    (2004)
  • P.P. Angelov et al.

    Evolving fuzzy-rule-based classifiers from data streams

    IEEE Trans. Fuzzy Syst.

    (2008)
  • V. Athitsos et al.

    Boostmap: a method for efficient approximate similarity rankings

  • R. Babuska

    Fuzzy Modeling and Identification (Ph.D. Thesis)

    (1996)
  • K. Bache et al.

    UCI Machine Learning Repository

    (2013)
  • X. Bai et al.

    Co-transduction for shape retrieval

    IEEE Trans. Image Process.

    (2012)
  • X. Bai et al.

    Learning context-sensitive shape similarity by graph transduction

    IEEE Trans. Pattern Anal. Mach. Intell.

    (2010)
  • A. Bar-Hillel et al.

    Learning distance functions using equivalence relations

  • J.C. Bezdek

    Pattern Recognition with Fuzzy Objective Function Algorithms

    (1981)
  • O. Chapelle et al.

    Transductive inference for estimating values of functions

  • G. Chechik et al.

    Large scale online learning of image similarity through ranking

    J. Mach. Learn. Res.

    (2010)
  • S. Chiu

    Fuzzy model identification based on cluster estimation

    J. Intell. Fuzzy Syst.

    (1994)
  • R. Collobert et al.

    Large scale transductive SVMs

    J. Mach. Learn. Res.

    (2006)
  • C. Cortes et al.

    On transductive regression

  • M. Eitz et al.

    How do humans sketch objects?

    ACM Trans. Graph. (TOG)

    (2012)
  • Cited by (6)

    • Forgetting of unused classes in missing data environment using automatically generated data: Application to on-line handwritten gesture command recognition

      2017, Pattern Recognition
      Citation Excerpt :

      Another work also based on incremental Mountain Clustering is proposed in Almaksour et al. [3]. In Carse et al. [11], Gomez and Dasgupta [19], Kasabov [21] the authors use genetic algorithms for the evolution of the rules, in Lughofer [25,26] the authors propose to use incremental Vector Quantization; in Režnáková et al. [37] the authors propose an incremental clustering for the fuzzy rules based on the local results of these rules; in Režnáková et al. [38] the authors propose to use ART-2A Carpenter et al. [10] for the incremental generation of rules with the self-organized adaptation in Režnáková et al. [39]; in Tencer et al. [49] the authors propose transductive learning to improve the fuzzy model inference. Most of these models still require some pre-definition, especially in terms of classes; however, they are capable of on-line learning and some of them are capable of learning from scratch.

    • UFuzzy: Fuzzy models with Universum

      2017, Applied Soft Computing Journal
      Citation Excerpt :

      Although most semi-supervised techniques assume that the unlabeled data are relevant to the task, in [23] it was shown that even irrelevant data can help the learning process by providing an implicit knowledge about the data domain. [24,25] introduce the help-training method which trains generative “help” classifier to assign labels to most confident unlabeled data and then use them further in the training process for the discriminative classifier. [26] uses transductive properties of the data to induce better metric for data similarity.

    • SO-ARTIST: Self-organized ART-2A inspired clustering for online Takagi-Sugeno fuzzy models

      2015, Applied Soft Computing Journal
      Citation Excerpt :

      Režnáková et al. [55] proposed the use of incremental density measurement for the antecedent part to achieve a recognition rate comparable to a multivariate distribution, but to keep low computational cost. This work has been later embeded in Tencer et al. [64]. These methods use recursive least square (RLS, Ljung [43], Angelov et al. [5]) optimization to adjust the consequent parameters.

    • Evaluation of techniques for signature classification from accelerometer and gyroscope data

      2015, Proceedings of the International Conference on Document Analysis and Recognition, ICDAR

    Lukas Tencer comes from Jalovec (Slovakia). He received his B.Sc. degree from Comenius University (Slovakia) in Applied Informatics in 2007 and his M.Sc. Degree from the same institution in Computer Graphics and Geometry in 2009. Since 2011, he joined the Synchromedia Laboratory for Multimedia Communication in Telepresence where he pursues his Ph.D. research, under the supervision of professor Mohamed Cheriet. His research interests include Pattern Recognition, Machine Learning, Similarity Learning and Sketch-Based Retrieval. He also held at several occasions software engineering positions at various companies (IBM, imo.im, VIS GRAVIS, Whitestein).

    Marta Režnáková was born in Trnava (Slovakia). She received her B.Sc. and M.Sc. from Comenius University in Bratislava, Slovakia in Applied Informatics and Computer Graphics and Geometry. In 2011 she joined the Synchromedia Laboratory for Multimedia Communication in Telepresence at École de Technologie Supérieure (ÉTS) to pursue her Ph.D. studies. Her supervisor is professor Mohamed Cheriet and her main interests are Sketch Recognition, Fuzzy Models, Clustering and Online Learning.

    Mohamed Cheriet was born in Algiers (Algeria) in 1960. He received his B.Eng. from USTHB University (Algiers) in 1984 and his M.Sc. and Ph.D. degrees in Computer Science from the University of Pierre et Marie Curie (Paris VI) in 1985 and 1988 respectively. Since 1992, he has been a professor in the Automation Engineering department at the École de Technologie Supérieure (University of Quebec), Montreal, and was appointed full professor there in 1998. He co-founded the Laboratory for Imagery, Vision and Artificial Intelligence (LIVIA) at the University of Quebec, and was its director from 2000 to 2006. He also founded the SYNCHROMEDIA Consortium (Multimedia Communication in Telepresence) there, and has been its director since 1998. His interests include document image analysis, OCR, mathematical models for image processing, pattern classification models and learning algorithms, as well as perception in computer vision. He has published more than 250 technical papers in the field, and has served as chair or co-chair of the following international conferences: VI’1998, VI’2000, IWFHR’2002, and ICFHR’2008. He currently serves on the editorial board and is associate editor of several international journals: IJPRAI, IJDAR, and Pattern Recognition. He co-authored a book entitled, “Character Recognition Systems: A guide for Students and Practitioners,” John Wiley and Sons, Spring 2007. He is a senior member of the IEEE and the chapter chair of IEEE Montreal Computational Intelligent Systems (CIS).

    View full text