A new hyperbox selection rule and a pruning strategy for the enhanced fuzzy min–max neural network

Abstract

In this paper, we extend our previous work on the Enhanced Fuzzy Min–Max (EFMM) neural network by introducing a new hyperbox selection rule and a pruning strategy to reduce network complexity and improve classification performance. Specifically, a new k-nearest hyperbox expansion rule (for selection of a new winning hyperbox) is first introduced to reduce the network complexity by avoiding the creation of too many small hyperboxes within the vicinity of the winning hyperbox. A pruning strategy is then deployed to further reduce the network complexity in the presence of noisy data. The effectiveness of the proposed network is evaluated using a number of benchmark data sets. The results compare favorably with those from other related models. The findings indicate that the newly introduced hyperbox winner selection rule coupled with the pruning strategy are useful for undertaking pattern classification problems.

Introduction

Artificial neural networks (ANNs), which are models of biological neural systems (Graupe, 1997, Li and Ma, 2010), have been widely used in many fields, which include healthcare (Lisboa, 2002), financial economics (Li & Ma, 2010), security (Obaidat & Macchairolo, 1994), power (Whei-Min, Chih-Ming, & Chiung-Hsing, 2011), robot motion control (Xia, Gang, & Jun, 2005), fault detection (Cho et al., 2010, Chow and Yee, 1991), Realization problem of multi-layer cellular (Ban & Chang, 2015), human action classification (Yu & Lee, 2015), and airline (Turkmen & Korkmaz, 2010). Pattern classification is one of the active ANN applications (Zhang, 2000). As an example, ANNs have been successfully applied to a variety of real-world pattern classification tasks in industry, business, and science (Zhang, 2000), medicine (Isola, Carvalho, & Tripathy, 2012), as well as industrial fault detection and diagnosis (Quteishat, Lim, Tweedale, & Jain, 2009). Besides that, ANNs are useful for handling noisy data collected from real environments. The learning properties of ANNs are robust against noise and are useful for recognizing different types of input patterns. In fact, one of the main problems of many learning algorithms is catastrophic forgetting that happens when they attempt to learn quickly in response to a changing world (Grossberg, 2013). Catastrophic forgetting, which is also known as the stability–plasticity dilemma, is concerned with the inability of a learning system to preserve what it has previously learned when new information is absorbed into its knowledge base. In other words, the learning systems forget previously learned information in the process of learning new information (Grossberg, 2013).

To overcome the stability–plasticity dilemma, a number of ANN models have been proposed, which include the adaptive resonance theory (ART) networks (Grossberg, 1976a, Grossberg, 1976b) and fuzzy min–max (FMM) networks (Simpson, 1992, Simpson, 1993). Among different ANN models, the FMM network and its variants have been the focus of many investigations (Davtalab et al., 2014, Gabrys and Bargiela, 2000, Nandedkar and Biswas, 2007a, Nandedkar and Biswas, 2007b, Quteishat and Lim, 2008, Quteishat et al., 2010, Simpson, 1992, Simpson, 1993, Zhang et al., 2011). The design of FMM variants is largely based on two original FMM networks introduced by Simpson, i.e., the supervised classification FMM network (Simpson, 1992) and later the unsupervised clustering FMM network (Simpson, 1993).

A number of neural–fuzzy models similar to FMM have been suggested in the literature, e.g. Abe and Lan (1995), Leite, Costa, and Gomide (2013) and Peters (2011). In Abe and Lan (1995), a new method to directly extract fuzzy rules (hyperboxes) from numerical data for pattern classification has been proposed. Specifically, a fuzzy classification model comparable with FMM for handling large-scale classification problems, but with less learning complexity, is formulated. In the proposed model, overlapping between different classes is solved by introducing two types of hyperboxes: activation and inhibition. The activation hyperboxes define the existence regions for classes, while the inhibition hyperboxes block the existence of data within the activation hyperboxes (Abe & Lan, 1995). While the concept of creating hyperboxes is similar to that of FMM, the number of hyperboxes in Abe and Lan (1995) increases by increasing the hyperbox size ($\theta$) (as opposed to that of FMM). In Peters (2011), the concept of granular box regression as a simple method that links independent and dependent variables by boxes (hyper-dimensional interval numbers) has been proposed. The idea of granular box regression is to establish relationships between independent and dependent variables, then to extract fuzzy rules from numerical data by a predefined number of boxes (Peters, 2011). These boxes are similar to the hyperboxes generated by FMM. However, the granular box regression remains transparent and does not behave like a black box as in FMM. Later, a granular neural network for evolving fuzzy system modeling from fuzzy data streams has been introduced by Leite et al. (2013).

The original FMM network uses hyperbox fuzzy sets to create and store knowledge (as hidden nodes) in its network structure. A number of hyperboxes are formed in the FMM structure. Each hyperbox occupies a region defined by its minimum (min) and maximum (max) points in the $n$-dimensional pattern space. The fuzzy notion in FMM arises from the combination of the hyperbox min–max points with a fuzzy membership function. The fuzzy membership function determines the degree to which an input pattern belongs to a particular class. FMM has a number of useful properties to handle pattern classification problems (Simpson, 1992), which include online learning, nonlinear separability, non-overlapping classes, short training time, as well as soft and hard decisions. All these salient properties make FMM a unique pattern classifier. Because of the advantages of FMM, a number of FMM variants have been introduced in the literature (Davtalab et al., 2014, Gabrys and Bargiela, 2000, Mohammed and Lim, 2015, Nandedkar and Biswas, 2007a, Nandedkar and Biswas, 2007b, Quteishat and Lim, 2008, Quteishat et al., 2010, Zhang et al., 2011). While different FMM variants have been proposed, they are built primarily based on the original FMM network, and are inherently affected by some, if not all, limitations of the original FMM learning algorithm. As a result, we have proposed the Enhanced FMM (EFMM) model (Mohammed & Lim, 2015) to solve the following limitations associated with FMM and its variants:

• The possibility of the hyperbox expansion procedure to increase the overlapping regions between different classes;

• The existing hyperbox overlap test rule is insufficient to detect all overlapping regions of hyperboxes;

• The hyperbox contraction procedure is affected owing the existence of undetected overlapping regions after performing the hyperbox overlap test.

Table 1 shows a summary of FMM and its variants that are susceptible to the above-mentioned limitations in the hyperbox expansion, hyperbox overlap test, and hyperbox contraction procedures, as well as whether the respective models are sensitive to noise. Even though EFMM has shown its effectiveness in addressing the first three limitations in Table 1, issues related to network complexity (in terms of the number of hyperboxes created) and noise tolerance remain unsolved. In EFMM, learning with large data sets increases the network complexity, while learning with noisy data samples results in spurious knowledge stored as hyperboxes in the network structure. These limitations affect the EFMM performance. Therefore, we investigate techniques and strategies to solve the limitations of EFMM and improve its robustness for tackling pattern classification problems. Accordingly, we propose an extended EFMM network, known as EFMM-II, in this paper, with the following contributions:

• A new hyperbox selection rule is formulated to reduce the network complexity by avoiding the creation of too many small hyperboxes within the vicinity of the winning hyperbox. It also helps reduce the misclassification errors by minimizing the overlapping regions of hyperboxes from different classes during the hyperbox expansion procedure.

• A new pruning strategy is devised to further reduce the network complexity pertaining to the presence of noise in the training data samples.

This paper is organized as follows. In Section  2, some related techniques for handling noise and complexity issues in neural network classifiers are described. The EFMM neural network is explained in Section  3. An analysis of the EFMM learning algorithm is presented in Section  4. The proposed hyperbox selection and expansion rule, as well as, a new pruning strategy for EFMM-II are detailed in Section  5. The network structure complexity, as well as algorithm complexity, are addressed in Section  6. Performance evaluation using a series of simulation studies is presented in Section  7. Finally, concluding remarks and suggestions for further work are included in Section  8.