TITS-FM: Transductive incremental Takagi-Sugeno fuzzy models
Graphical abstract
Introduction
Similarity is a key concept for a majority of machine learning and patter recognition techniques, which represents the elements as multidimensional vectors. The similarity used in these techniques is frequently quite simple such as the L1 or L2 similarity. This is the most straight forward notion of a similarity, which can be translated from the human knowledge into the machine knowledge. But this kind of similarity might not necessarily be the best way to evaluate the correspondence between two items. It might not be the way, how people understand correspondence between objects. We can see that human notation of similarity is not always in the correspondence with simple metrics used in machine learning and pattern recognition algorithms. Even simple rules such as the triangular inequality can be too limiting for humans and algorithms which do not enforce this inequality can achieve better results than those, which require it [15]. Therefore we seek for a more complex notion of similarity, which can be used.
Specifically we rely on the concept of transductive similarity measure. The transductive inference itself was introduced as a concept by [24] and was used in combination with the SVM classifier in [28], [17]. In our work we understand the concept of the transductive similarity as a similarity induced by the transductive property of the space. More specifically, we say that the similarity measure s between two elements xi and xj is learned through the graph-based transductive learning algorithm. This is in accordance to the notion of the transductive similarity used in [11], who experimentally proved that transductive similarity can improve the accuracy of classification tasks. In case of the transductive similarity applied to individual elements, our intuition is to learn the similarity induced by the shape manifold represented by the feature space elements. The question that we try to answer is, whatever the fuzzy rules also create a manifold in a space and whatever we can learn a better fuzzy membership function based on the transductive property of the space.
Besides the similarity, second important aspect is the notion of belonging to a particular group or a class. In absolute world, this notion represents conditional relationship if x = C then y = V based on binary logic. This notion builds on an assumption that the relationship “belongs to” is absolute. However this is not the way how we understand this relationship in the real life. Given an object, which is “somehow” round and “somehow” red can be considered by different degrees of confidence as either an apple, a red ball or an orange, depending on the value of its properties. This notion is exactly what is expressed in the fuzzy logic and the conditional relationship is rather where the operator tells the amount of correspondence of the variable x to C and the variable y to V. Thus, rather than a binary condition, a fuzzy rule represents variables having a continuous amount of values.
Given these two concepts, we should state a question, if the transductive similarity can help us to improve the performance of a fuzzy-based models. Our hypothesis is that for models based on fuzzy rules we can develop a cross-rule relationships, that can further be used in the transductive similarity to adjust activations (values of x) and results (values of y) of used fuzzy rules. Especially we build on the notion that the relationship expresses a degree of x being C, thus it can be understood as a similarity measure, where value of 1 expresses an absolute similarity and value 0 expresses an absolute dissimilarity.
In this paper we present novel techniques for using the transductive similarity for the inference in TS-fuzzy models. Since the simple transductive similarity model [11] did not prove to be effective for each of the tasks evaluated during our experiments, we propose a new model of the transductive similarity. This model does not need to be necessarily used in the context of TS-fuzzy models. The contributions of this paper are as following:
- •
The transductive inference model for incremental TS-fuzzy models
- •
3 novel models for the transductive similarity, which can be combined and yield 7 possible variants
- •
Applications of proposed models to tasks of gesture, sketch and object recognition
We evaluate the proposed techniques on 5 standardized datasets used for classification tasks. Further we present qualitative results for applications in 3 domains based on different classification models. To further advocate the proposed similarity models, we use them in a setting independent of fuzzy-based classification.
Our paper is organized as following: Section 2 summarizes previous works on models for transductive similarity, fuzzy models and their combinations, Section 3 introduces the transductive similarity, the baseline fuzzy model and describes our method for the incorporation of the transductive inference for incremental fuzzy models, Section 4 describes the proposed improvement to existing transductive similarity models, Section 5 presents experimental results, an evaluation protocol and applications of our method, Section 6 includes final conclusion and discussion.
Section snippets
Related works
Transductive similarity gives sense in any case, where notion of neighborhood could be considered in a topological space. It was widely used in a range of classifiers [17], [33], [54], [44] and was able to improve the results of these classifiers achieved without transductive property. Besides classification transductive similarity was successfully used in regression tasks [14], [19]. Motivated by these successful applications, we decided to explore a possible gain for incremental fuzzy models.
Transductive inference model
Now we describe our model, which is based on the notion of transductive similarity. At first we introduce the term of transductive similarity itself and building on this we present our new inference model for Takagi-Sugeno fuzzy models induced by the transductive similarity.
Improved transductive similarity measures
In this section we describe three models for transductive similarity, which improve the existing measure as presented in (3). These measures are proposed to capture additional properties of a used datasets and they allow the metric to adapt to a given dataset. Although we are presenting them as separate models, they can be used in a combination one with each other. This provides us with another free parameter which can be adapted to the given dataset.
Results
In this section we focus on an evaluation of our technique with a respect to each of the proposed methods. As a baseline technique we use implementation of TS fuzzy models from [40], which extends model from [1] based on the original online version of TS models presented in [5]. We evaluate each of the different factors of our method separately and in mutual combination giving altogether 8 possible testing settings (+1 for baseline method) as shown in Table 1.
Conclusions
In this work we presented a new inference model for incremental TS-fuzzy models which is based on a transductive similarity. The models brings a concept of transductivity into an inference for an incremental TS-fuzzy models. Unlike previous approaches which use the transductive inference for fuzzy models, our technique does not require access to the whole dataset in advance and can build the transductive model incrementally, one example at a time, starting from scratch. We show, that this
Acknowledgments
This work was supported by the Social Sciences and Humanities Research Council of Canada (SSHRC) as well as the Natural Sciences and Engineering Research Council of Canada (NSERC).
Lukas Tencer comes from Jalovec (Slovakia). He received his B.Sc. degree from Comenius University (Slovakia) in Applied Informatics in 2007 and his M.Sc. Degree from the same institution in Computer Graphics and Geometry in 2009. Since 2011, he joined the Synchromedia Laboratory for Multimedia Communication in Telepresence where he pursues his Ph.D. research, under the supervision of professor Mohamed Cheriet. His research interests include Pattern Recognition, Machine Learning, Similarity
References (55)
- et al.
Ten years of genetic fuzzy systems: current framework and new trends
Fuzzy Sets Syst.
(2004) - et al.
Applications of neuro fuzzy systems: a brief review and future outline
Appl. Soft Comput.
(2014) - et al.
An experiment in linguistic synthesis with a fuzzy logic controller
Int. J. Man–Mach. Stud.
(1975) - et al.
A bi-fuzzy progressive transductive support vector machine (BFPTSVM) algorithm
Expert Syst. Appl.
(2010) - et al.
TWNFI – a transductive neuro-fuzzy inference system with weighted data normalization for personalized modeling
Neural Netw.
(2006) - et al.
Structure identification of fuzzy model
Fuzzy Sets Syst.
(1988) - et al.
Observer-based robust fuzzy control of nonlinear systems with parametric uncertainties
Fuzzy Sets Syst.
(2002) Fuzzy sets
Inf. Control
(1965)- et al.
Evolving fuzzy classifiers: application to incremental learning of handwritten gesture recognition systems
- et al.(2010)
Evolving fuzzy systems from data streams in real-time
Evolving fuzzy rule-based classifiers
An approach to online identification of Takagi-Sugeno fuzzy models
IEEE Trans. Syst. Man Cybern. B: Cybern.
Evolving fuzzy-rule-based classifiers from data streams
IEEE Trans. Fuzzy Syst.
Boostmap: a method for efficient approximate similarity rankings
Fuzzy Modeling and Identification (Ph.D. Thesis)
UCI Machine Learning Repository
Co-transduction for shape retrieval
IEEE Trans. Image Process.
Learning context-sensitive shape similarity by graph transduction
IEEE Trans. Pattern Anal. Mach. Intell.
Learning distance functions using equivalence relations
Pattern Recognition with Fuzzy Objective Function Algorithms
Transductive inference for estimating values of functions
Large scale online learning of image similarity through ranking
J. Mach. Learn. Res.
Fuzzy model identification based on cluster estimation
J. Intell. Fuzzy Syst.
Large scale transductive SVMs
J. Mach. Learn. Res.
On transductive regression
How do humans sketch objects?
ACM Trans. Graph. (TOG)
Cited by (6)
Generative adversarial network based telecom fraud detection at the receiving bank
2018, Neural NetworksForgetting of unused classes in missing data environment using automatically generated data: Application to on-line handwritten gesture command recognition
2017, Pattern RecognitionCitation Excerpt :Another work also based on incremental Mountain Clustering is proposed in Almaksour et al. [3]. In Carse et al. [11], Gomez and Dasgupta [19], Kasabov [21] the authors use genetic algorithms for the evolution of the rules, in Lughofer [25,26] the authors propose to use incremental Vector Quantization; in Režnáková et al. [37] the authors propose an incremental clustering for the fuzzy rules based on the local results of these rules; in Režnáková et al. [38] the authors propose to use ART-2A Carpenter et al. [10] for the incremental generation of rules with the self-organized adaptation in Režnáková et al. [39]; in Tencer et al. [49] the authors propose transductive learning to improve the fuzzy model inference. Most of these models still require some pre-definition, especially in terms of classes; however, they are capable of on-line learning and some of them are capable of learning from scratch.
UFuzzy: Fuzzy models with Universum
2017, Applied Soft Computing JournalCitation Excerpt :Although most semi-supervised techniques assume that the unlabeled data are relevant to the task, in [23] it was shown that even irrelevant data can help the learning process by providing an implicit knowledge about the data domain. [24,25] introduce the help-training method which trains generative “help” classifier to assign labels to most confident unlabeled data and then use them further in the training process for the discriminative classifier. [26] uses transductive properties of the data to induce better metric for data similarity.
SO-ARTIST: Self-organized ART-2A inspired clustering for online Takagi-Sugeno fuzzy models
2015, Applied Soft Computing JournalCitation Excerpt :Režnáková et al. [55] proposed the use of incremental density measurement for the antecedent part to achieve a recognition rate comparable to a multivariate distribution, but to keep low computational cost. This work has been later embeded in Tencer et al. [64]. These methods use recursive least square (RLS, Ljung [43], Angelov et al. [5]) optimization to adjust the consequent parameters.
Application of Takagi–Sugeno fuzzy model optimized with an improved Free Search algorithm to industrial polypropylene melt index prediction
2017, Transactions of the Institute of Measurement and ControlEvaluation of techniques for signature classification from accelerometer and gyroscope data
2015, Proceedings of the International Conference on Document Analysis and Recognition, ICDAR
Lukas Tencer comes from Jalovec (Slovakia). He received his B.Sc. degree from Comenius University (Slovakia) in Applied Informatics in 2007 and his M.Sc. Degree from the same institution in Computer Graphics and Geometry in 2009. Since 2011, he joined the Synchromedia Laboratory for Multimedia Communication in Telepresence where he pursues his Ph.D. research, under the supervision of professor Mohamed Cheriet. His research interests include Pattern Recognition, Machine Learning, Similarity Learning and Sketch-Based Retrieval. He also held at several occasions software engineering positions at various companies (IBM, imo.im, VIS GRAVIS, Whitestein).
Marta Režnáková was born in Trnava (Slovakia). She received her B.Sc. and M.Sc. from Comenius University in Bratislava, Slovakia in Applied Informatics and Computer Graphics and Geometry. In 2011 she joined the Synchromedia Laboratory for Multimedia Communication in Telepresence at École de Technologie Supérieure (ÉTS) to pursue her Ph.D. studies. Her supervisor is professor Mohamed Cheriet and her main interests are Sketch Recognition, Fuzzy Models, Clustering and Online Learning.
Mohamed Cheriet was born in Algiers (Algeria) in 1960. He received his B.Eng. from USTHB University (Algiers) in 1984 and his M.Sc. and Ph.D. degrees in Computer Science from the University of Pierre et Marie Curie (Paris VI) in 1985 and 1988 respectively. Since 1992, he has been a professor in the Automation Engineering department at the École de Technologie Supérieure (University of Quebec), Montreal, and was appointed full professor there in 1998. He co-founded the Laboratory for Imagery, Vision and Artificial Intelligence (LIVIA) at the University of Quebec, and was its director from 2000 to 2006. He also founded the SYNCHROMEDIA Consortium (Multimedia Communication in Telepresence) there, and has been its director since 1998. His interests include document image analysis, OCR, mathematical models for image processing, pattern classification models and learning algorithms, as well as perception in computer vision. He has published more than 250 technical papers in the field, and has served as chair or co-chair of the following international conferences: VI’1998, VI’2000, IWFHR’2002, and ICFHR’2008. He currently serves on the editorial board and is associate editor of several international journals: IJPRAI, IJDAR, and Pattern Recognition. He co-authored a book entitled, “Character Recognition Systems: A guide for Students and Practitioners,” John Wiley and Sons, Spring 2007. He is a senior member of the IEEE and the chapter chair of IEEE Montreal Computational Intelligent Systems (CIS).