An evolving recurrent interval type-2 intuitionistic fuzzy neural network for online learning and time series prediction

Highlights

• A novel recurrent structure utilizing interval type-2 intuitionistic fuzzy set is proposed.

• Self-evolving structure based on density clustering and intuitionistic evaluation.

• The uncertain mean of interval type-2 intuitionistic fuzzy set is first constructed.

Abstract

The prediction of time series has both the theoretical value and practical significance in reality. However, since the high nonlinear and noises in the time series, it is still an open problem to tackle with the uncertainties and fuzziness in the forecasting process. In this article, an evolving recurrent interval type-2 intuitionistic fuzzy neural network (eRIT2IFNN) is proposed for time series prediction and regression problems. The eRIT2IFNN employs interval type-2 intuitionistic fuzzy sets to enhance the modeling of uncertainties by intuitionistic evaluation and noise tolerance of the system. In the eRIT2IFNN, the antecedent part of each fuzzy rule is defined using intuitionistic interval type-2 fuzzy sets, and the consequent realizes the Takagi–Sugeno–Kang type fuzzy inference mechanism. In order to utilize the prior knowledge including intuitionistic information, a local internal feedback is established by feeding the rule firing strength of each rule to itself eRIT2IFNN is fully adaptive to the evolving of sequence data by online learning of structure and parameters. A modified density-based clustering is implemented for the structure learning, where both densities and membership degrees are involved to determine the fuzzy rules. Performance of eRIT2IFNN is evaluated using a set of benchmark problems and compared with existing fuzzy inference systems. Moreover, the eRIT2IFNN is tested for identification of dynamics under both noise-free and noisy environments. Finally, a group of practical financial price-tracking problems including high-frequency data of financial future, commodity future and precious metal are used for the evaluation of the proposed inference system.

Introduction

Based on the fuzzy set theory established by Zadeh [1], fuzzy logic system (FLS) is an intelligent system that imitates human brain’s way to carry out fuzzy information processing, which can approximate nonlinear functions defined on compact set by arbitrary precision and the inference process is interpretable for domain experts as well as users [2]. In recent years, different machine learning methods have been proposed, such as neural networks [3], support vector machine [4] and artificial swarm intelligence [5]. Thereinto, fuzzy neural networks (FNNs), that combine the reasoning ability of the fuzzy logic system with adaptively adjusting parameters of the neural networks, increase effectiveness and practicability of inference [6]. Type-1 fuzzy sets (T1FS) are widely used in traditional fuzzy systems, where an element of the universe of discourse is mapped onto a precise number by using some kinds of membership functions (MFs). However, the application of T1FS is limited to the usage of the crisp values to evaluate the membership degrees. In order to improve the ability of handling uncertainty, Zadeh proposed the concept of type-2 fuzzy sets (T2FSs), where a fuzzy set instead of a crisp number is implemented to describe the membership values of elements to T2FSs [7]. Results in the literature have shown that type-2 FLSs appear to be a more promising method compared to their type-1 counterparts [8]. However, the general Type-2 fuzzy sets are computationally intensive, which seriously limits the scope of application fields. In order to reduce the computation burden, interval Type-2 fuzzy set (IT2FS) is designed to set the secondary memberships as zero or one [9], that have already been successfully applied into classification problems [10], pattern recognition [11] and time series prediction [12].

The above mentioned fuzzy sets are based on the assumption that the non-membership functions (NMFs) are complementary to the MFs, and the two opposite of a fuzzy concept are depicted by using a single membership value no matter to what type of uncertainty. However, because the information related to the decision is generally incomplete and additional factors such as economy, psychological behavior and ideology would be also involved, some hesitations always exist in the process of analysis and decision making [13]. The conventional fuzzy sets are incapable of dealing with the above situation. In 1986, Atanassov proposed a new kind of fuzzy sets known as the intuitionistic fuzzy set (IFS), which was characterized by independently defining MFs and NMFs together with some degree of hesitancy [14]. As an extension of the traditional fuzzy set theory, IFSs represent three kinds of information, i.e. the degrees of membership, non-membership and hesitation (intuitionistic fuzzy index (IF-index)). Furthermore, Atanassov and Gargov [15] proposed the interval valued intuitionistic fuzzy set (IVIFS), the core idea of which is that both MFs and NMFs separately defined in [0, 1] as intervals, and the summation of the upper MF and upper NMF is less than or equal to 1. IVIFS has been widely used in intelligent system fields, such as multi-attribute decision making [16].

Recently, a newly developed intuitionistic fuzzy system termed interval type-2 intuitionistic fuzzy system (IT2IFS) was introduced by Eyoh et al. [17], where non-membership functions and intuitionistic fuzzy indices as independent components were introduced into the classical IT2FLS models. Similar to the relationship between IT2FS and interval valued fuzzy sets, IT2IFS is a much broader concept compared to IVIFS, which can be seen as a special representation of IT2IFS [18]. Moreover, there are also some distinguished characteristics existing in IT2FS, such as the summation of upper bound MF and lower bound NMF is less than or equal to 1 as well as the summation of the lower bound MF and upper bound NMFS is less than or equal to 1 [18]. By means of IF-indices in the FOUs of the MFs and NMFs through a process of scaling and shifting, IT2IFS has the ability to capture additional uncertainties to improve the inference capability. In [19], the interval type-2 intuitionistic fuzzy logic system with Takagi–Sugeno–Kang type (IT2IFLS-TSK) was proposed, where decoupled extended Kalman filter (DEKF) was adopted as parameter learning algorithm; subsequently, a hybrid algorithm consisting of decouple extended DEKF and gradient descent (GD) was introduced into IT2IFLS and the empirical comparisons were made with different type-1 and type-2 variants [18]. Besides, IT2IFLS was also implemented in non-linear system prediction [20], time series prediction and regression problems [17]. The introduction of IT2IFS enhances the ability of anti-noise and enriches the details of reasoning information.

Because the continuous arrival of sequence data often evolves over time, it is challenging to make models learn data drift or shift adaptively. Over the past decades, evolving fuzzy systems (EFSs) as an important topic have been widely studied to be capable of keeping track of the variance of input data and learning from their environment in real time. Up to date, there are various proposed EFS, generally, which are mainly implemented in data streams [21]. For instance, Lughofer et al. addressed the visual inspection by integrating new classes on the fly into on-line evolving fuzzy classifier, which can adapt their structure and update their parameters incrementally due to embedded on-line adaptable classifier learning engines [22]. However, one of the key points of online learning models is “when to learn”. When the distribution or “concept” of data is not shifted, it is not necessary to learn each sample for a trained model. In order to decide “what-to-learn, when-to-learn and how-to-learn”, a series of online meta-cognitive learning algorithms were designed. Subramanian et al. proposed a neuro-fuzzy inference system called as meta-cognitive neuro-fuzzy inference system (McFIS), where a self-evolving learning mechanism was presented such that the learning process of the cognitive component was conducted in the real-value [23] and complex-value [24] fuzzy neural systems. Furthermore, in order to reduce the laborious pre- and/or post-training processes, Pratama et al. introduced Scaffolding theory into meta-cognitive algorithm [25]. Although the above studies well accomplished the online learning with less computational burden, however, they were all based on the assumption that the learned data were independent of each other. And, the corresponding inference systems were constructed by feed-forward mode. It should be noted that, in reality, sequence data generally show internal correlation such as the long-term self-similarity in time series, where prior knowledge in the history information is useful. Therefore, for dealing with sequence data, the inference system with recurrent structure is an indispensable research topic.

Since recurrent models exhibit some kind of memory properties, various recurrent fuzzy neural networks (RFNNs) have been proposed to solve the problems related to temporal characteristics [26]. Generally, RFNNs can be separated into two major categories, i.e. local feedbacks and global feedbacks, both of which have separately strengths. As to the local one, the feedback loop in a given fuzzy rule is connected only to itself, and the history information is reserved within a particular fuzzy rule. For instance, in [27], a recurrent self-evolving fuzzy neural network with local feedbacks (RSEFNN-LF) was proposed, which was composed of zero-order or first-order Takagi–Sugeno–Kang (TSK)-type recurrent fuzzy if–then rules. And, the recurrent model in a RSEFNN-LF was established by locally feeding the firing strength of a fuzzy rule back to itself. However, since the recurrent structure only existed in the rule layer, the reaction speed in consequent layer would be slow to deal with changing learning environments [28]. Therefore, the single layer of local recurrent connections was further extended into double layers structure such that the fuzzy inference system can both capture the temporal system dynamic and improve reaction speed of the Wavelet function [29]. On the other hand, for the global feedbacks, all of the rules in an inference system would be connected to each other in virtue of the external feedback loops, which transform the prior knowledge within the entire FNNs [30]. Although global feedbacks have more wide information reserved, the structure of fuzzy inference system is correspondingly increasing complexity. Besides, there are also some various variants of recurrent structures such as interactively recurrent structures [31]. Overall, from the existing studies, one can obtain that the involvement of recurrent structure would be in favor of the reasoning results.

In this paper, an evolving recurrent interval type-2 intuitionistic fuzzy neural network is first proposed. As discussed above, although recurrent fuzzy neural networks have been widely studied [26], [27], [29], [30], [31], the existing works mainly focused on the recurrent information related to membership degrees. And, to the best of our knowledge, there is few research on the recurrent FNN structure with intuitionistic fuzzy rules. Type-2 intuitionistic fuzzy rules can be beneficial for the improvement of the inference capability by capturing the additional uncertainties [15]. Therefore, the study on the recurrent interval IT2IFNN is a meaningful topic for the prediction problems with time dependency. Moreover, in the existing works [17], [18], [19], [20], all the inference systems based on interval type-2 intuitionistic fuzzy set are fixed system architecture, where network structure cannot be adaptively updated along with the inference process. As to the sequence data such as time series, since the existence of concept drifts and change points, invariant reasoning structure or/and fixed fuzzy rules cannot meet with real-time changes of data features [23]. However, how to efficiently achieve the evolving structure of inference system is still an open problem. In the literature, a pre-specified threshold is commonly utilized to determine the fuzzy rules, which only consider the fire strength of membership degree [27], [30]. Here, based on a modified density-based clusters method, the proposed eRIT2IFNN has a self-evolving ability that it can automatically evolve to obtain the appropriate intuitionistic fuzzy rules and adjust the network structure by learning the real-time data streams. And, by means of the designed feedback loops in the proposed FNNs, the captured information by intuitionistic fuzzy rules can be encoded in memory representation.

Mainly, there are three major contributions in this paper as follows. (1) A novel recurrent FNN structure utilizing interval type-2 intuitionistic fuzzy set is proposed. By means of internal feedbacks, the firing strength of intuitionistic fuzzy rules can be reserved and delivered along the timeline. Different from the existing models, the recurrent information involves both the membership and non-membership degrees of elements to the intuitionistic fuzzy sets, which implies that the contributions of the non-membership functions could be memorized and be fully used in the subsequent reasoning process. (2) The interval type-2 intuitionistic fuzzy inference system with self-evolving structure is first achieved. In order to handle the online stream data, a learning algorithm of structure is implemented in virtue of a modified density-based cluster method, where both density and MFs (NMFs) are involved to decide the addition and deletion of intuitionistic fuzzy rules. The main advantage of the proposed self-evolving learning method is the combination of dense-based cluster method with the fire strength of intuitionistic fuzzy sets. (3) To match the evolving fuzzy rules, different to the previous related studies, the uncertain mean of IT2IFSs is first applied in this paper. Accompanied with the online learning of structure and parameters, based on the centers of the fuzzy rules, this paper implements uncertain mean in eRIT2IFNN and derives its sequential learning algorithm. To demonstrate the performance of the proposed system, several groups of experiments have been carried out and comparative analysis with various kinds of type-1 and type-2 FNNs is implemented.

This article is organized as follows. Section 2 provides the related definitions. Section 3 illustrates the overall structure of the proposed eRIT2IFNN. Subsequently, the structure and parameters learning methods for eRIT2IFNN are introduced in Section 4. And, the performance evaluation is carried out in Section 5. Finally, a concluding remark is given.