Evolving fuzzy pattern trees for binary classification on data streams

Abstract

Fuzzy pattern trees (FPTs) have recently been introduced as a novel model class for machine learning. In this paper, we consider the problem of learning fuzzy pattern trees for binary classification from data streams. Apart from its practical relevance, this problem is also interesting from a methodological point of view. First, the aspect of efficiency plays an important role in the context of data streams, since learning has to be accomplished under hard time (and memory) constraints. Moreover, a learning algorithm should be adaptive in the sense that an up-to-date model is offered at any time, taking new data items into consideration as soon as they arrive and perhaps forgetting old ones that have become obsolete due to a change of the underlying data generating process. To meet these requirements, we develop an evolving version of fuzzy pattern tree learning, in which model adaptation is realized by anticipating possible local changes of the current model, and confirming these changes through statistical hypothesis testing. In experimental studies, we compare our method to a state-of-the-art tree-based classifier for learning from data streams, showing that evolving pattern trees are competitive in terms of performance while typically producing smaller and more compact models.

Introduction

Fuzzy pattern tree induction was recently introduced as a novel machine learning method for classification by Huang et al. [11]. Independently, the same type of model structure was proposed in [23] under the name “fuzzy operator tree”. An alternative to the original algorithm for learning pattern trees, as proposed in [11], was developed by Senge and Hüllermeier in [20]. Besides, an FPT variant for regression was introduced in [19].

Roughly speaking, a fuzzy pattern tree is a hierarchical, tree-like structure, whose inner nodes are marked with generalized (fuzzy) logical and arithmetic operators. It implements a recursive function that maps a combination of attribute values, entered in the leaf nodes, to a number in the unit interval, produced as an output by the root of the tree. The model class of fuzzy pattern trees is interesting for several reasons. Apart from some properties that make it appealing from a learning point of view (like a built-in feature selection mechanism and the possibility to guarantee monotonicity in certain attributes), FPTs are arguably attractive from an interpretation point of view. Generally, each tree can be considered as a kind of (generalized) logical description of a class.¹ In this regard, pattern trees can be considered as a viable alternative to classical fuzzy rule models. Compared to such models, the hierarchical structure of pattern trees further allows for a more compact representation and for trading off accuracy against model simplicity in a seamless manner.

In recent years, the idea of adaptive learning in dynamical environments has received considerable attention, especially under the slogan of “learning from data streams” [8]. Closely related to this, a special branch of data-driven fuzzy systems modeling has emerged under the notion of “evolving fuzzy systems” [2], [15], [1], [16]. Despite small differences regarding the basic assumptions and the technical setting, the emphasis of goals and performance criteria, or the focus on specific types of applications, the key motivation of these and related fields is the idea of a system that learns incrementally, and maybe even in real-time, on a continuous stream of data, and which is able to properly adapt itself to changes of environmental conditions or properties of the data-generating process.

Motivated by these developments, we propose an extended version of fuzzy pattern trees suitable for learning from data streams. More specifically, building on the (batch learning) algorithm for pattern tree induction as proposed in [20], we develop an evolving variant for the problem of binary classification. The rest of the paper is organized as follows. In Section 2, we start with a brief description of the data stream scenario and recall the special requirements it involves for learning. Fuzzy pattern trees are explained in Section 3, in which we also recall the basic algorithm for learning such trees in batch mode. An extension of this algorithm for learning from data streams in then proposed in Section 4. Finally, an empirical evaluation of this method is presented in Section 5, where evolving fuzzy pattern trees are compared with so-called Hoeffding trees [13] on different types of data streams, both in terms of performance and readability.