Elsevier

Neurocomputing

Volume 124, 26 January 2014, Pages 63-71
Neurocomputing

TPPFAM: Use of threshold and posterior probability for category reduction in fuzzy ARTMAP

https://doi.org/10.1016/j.neucom.2013.07.042Get rights and content

Abstract

The issue of category proliferation caused by the overlapping classes in fuzzy ARTMAP (FAM) is addressed in this paper. A new FAM-based neural architecture called TTPFAM is proposed, which can reduce category proliferation by performing a threshold filtering mechanism before a new category created during training, and improve the classification accuracy by combining prediction distributed by the dynamic Q-max rule and posterior probability estimated during testing. The TPPFAM can produce a small size of neural network architecture without degradation of the classification accuracy. The algorithm is evaluated in terms of the classification accuracy and the number of categories by experiments on both artificial and real data, and the results show that the performance of TPPFAM is better than that of the other models.

Introduction

The fuzzy ARTMAP (FAM) is a supervised clustering algorithm which can be considered as one of the leading neural network (NN) algorithms for classification tasks [1]. It is based on Adaptive Resonance Theory (ART) which constitutes a model for artificial NN inspired by the studies of Grossberg and Carpenter [2]. These supervised learning systems have many merits, such as online real-time and incremental learning [3], the use of few training epochs to achieve reasonably accuracy [4], allowing learning new data without forgetting past data, and tacking the so-called “plasticity-stability dilemma” [5], and so on. The FAM and its modified versions have been widely used in many application fields, including robotic application scenarios [6], information fusion [7], data mining [8], image classification, genetic abnormality diagnosis [9], etc.

Though large [10] or noisy databases [11] can cause FAM generating too many categories, i.e., category proliferation, the problem is most serious when the network architecture is trained with data of overlapping classes [12]. When a similar input pattern lies in the overlapping region, it is difficult to make a more reliable and accurate prediction. Therefore, many categories are created to map onto this overlapping region, leading to heavy computational burden, more memory requirement and possible degradation of the classification accuracy [13], [14].

In order to cope with the category proliferation problem in FAM, many researchers have proposed various schemes which can be divided into two major classes—the new FAM-based algorithms and the modified versions of FAM. Nevertheless, the borderline between two classes is thin and sometimes we may have crossed it unintentially [15]. The former includes Boosted ARTMAP [16], Gaussian ARTMAP [11], ARTMAP-IC [17], and Bayesian ARTMAP [18]. The latter includes Distributed ARTMAP [19], rule pruning [20], μARTMAP [21], Koufakou et al. use cross-validation techniques to reduce category proliferation [14], and Sit et al. adopt a number of measures to reduce category proliferation [22].

In this paper, in order to reduce the number of categories and improve the classification accuracy, a threshold adjustment parameter is introduced in both training and testing algorithms, and a new algorithm is proposed, called TTPFAM. The proposed algorithm makes some modifications of the FAM algorithm in the following aspects: (1) During training, in order to reduce category proliferation, a threshold filtering is performed before creating a new category. Then the corresponding supervised class posterior probabilities of the given categories filtered are estimated, the category with the maximum posterior probability will be chosen as the winner. (2) During testing, in order to improve classification accuracy, the system combines the category information distributed by a dynamic Q-max rule with the estimated posterior probability matrix to get the most reliable and accurate class for each testing pattern. A comparison of TTPFAM with FAM, MT-, ARTMAP-IC and the Algorithm 1 in [22] is made on artificial and real data, and the results show that the TTPFAM algorithm is superior to the others in terms of the classification accuracy, the number of categories, and the sensitivity to statistical overlapping.

The remainder of the paper is organized as follows. Section 2 briefly summarizes the principles and dynamics of FAM, and discusses the category proliferation problem. The proposed algorithm is described in detail in Section 3, including the derivation of the threshold, the estimation of the posterior probability, and the description of the TTPFAM algorithm. In Section 4, the experiments and results analysis are presented. The conclusions are drawn in Section 5.

Section snippets

Fuzzy ARTMAP

The FAM is the most important neural network architecture derived from ART. It consists of two unsupervised fuzzy ART [23] modules, i.e., ARTa and ARTb, which cluster the patterns from the input space and the output space into categories, respectively. Two modules are linked by the map field which is used to ensure that each category from ARTa is associated with only one category from ARTb, i.e., a many-to-one mapping. However, the frequency of the associations between ARTa and ARTb categories

TTPFAM

As the description above, when an input pattern lies in the overlapping region, it is difficult to make one prediction for the input pattern. In FAM, if the activation values of two (or more) categories are the same and pass the vigilance test, the one with smaller index will be selected as the winner. Thus, it would be unfair to other categories which have the same activation value and pass the vigilance mapped to a class prediction accurately. Therefore, predicting only one category mapping

Experimental results

In the following sections, we evaluate the performance of TPPFAM in terms of the classification accuracy, the number of categories and the sensitivity to class overlapping. TPPFAM is tested on both artificial and real databases. In the paper, each database is split into 70% for training and 30% for testing. The overlapping degree of data directly affects the difficulty of the classification problem. Obviously, the problem of category proliferation in FAM becomes more and more serious as the

Conclusions

Overlapping classes present a difficulty in classification and cause category proliferation in FAM. In the paper, a new FAM-based neural architecture called TPPFAM has been proposed as a solution to the category proliferation problem. During training of TPPFAM, when an input pattern lies in the overlapping region, a threshold filtering mechanism is performed before creating a new category. Then according to the corresponding supervised class posterior probabilities of the given categories

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grant no. 60871074.

Yongquan Zhang was born in Gansu, China in 1985. He received B.S. (2007) degree in Computer Science and Technology, and M.S. (2010) degree in Computer Applications Technology from Lanzhou University of Technology, respectively.

Currently, he is pursuing a Ph.D. degree in the School of Electronic Engineering at Xidian University. His research interests include machine learning, signal processing, target tracking and data fusion.

References (35)

  • G.A. Carpenter et al.

    Biased ART: a neural architecture that shifts attention toward previously disregarded features following an incorrect prediction

    Neural Networks

    (2010)
  • G.A. Carpenter et al.

    Self-organizing ARTMAP rule discovery

    Neural Networks

    (2012)
  • R.P. Gorman et al.

    Analysis of hidden units in a layered network trained to classify sonar targets

    Neural Networks

    (1988)
  • G.A. Carpenter et al.

    Fuzzy ARTMAP: a neural-network architecture for incremental supervised learning of analog multidimensional maps

    IEEE Transactions on Neural Networks

    (1992)
  • G.A. Carpenter et al.

    The ART of adaptive pattern recognition by a self-organizing neural network

    Computer

    (1988)
  • E.P. Hernandez, E.G. Sanchez, Y.A. Dimitriadis, J.L. Coronado, A neuro-fuzzy system that uses distributed learning for...
  • S. Grossberg

    Adaptive pattern recognition and universal encoding II: feedback, expectation, olfaction, and illusions

    Biological Cybernetics

    (1976)
  • Cited by (22)

    • A novel fuzzy ARTMAP with area of influence

      2021, Neurocomputing
      Citation Excerpt :

      Over the years, many authors have proposed strategies to attenuate the category proliferation in FAM networks. These strategies are based on different approaches, such as changing the FAM architecture [8–15], applying post-processing methods for pruning categories [16,17], or even replacing the hyper-rectangular shape of FAM categories with hyper-spheres, Gaussians, polytopes, and so on [18–23]. In this work, our aim is to approach the category proliferation problem while maintaining the FAM categories geometry and its online and incremental learning capabilities – these properties are directly linked to the works that change the FAM architecture.

    View all citing articles on Scopus

    Yongquan Zhang was born in Gansu, China in 1985. He received B.S. (2007) degree in Computer Science and Technology, and M.S. (2010) degree in Computer Applications Technology from Lanzhou University of Technology, respectively.

    Currently, he is pursuing a Ph.D. degree in the School of Electronic Engineering at Xidian University. His research interests include machine learning, signal processing, target tracking and data fusion.

    Hong-bing Ji was born in Shaanxi, China in 1963. He graduated from Northern West Telecommunications Engineering College (the predecessor of Xidian University) and earned B.S. degree in Radar Engineering in 1983. He received M.S. (1989) degree in Circuit, Signals and Systems and Ph.D. (1999) degree in Signal and Information Processing from Xidian University, respectively.

    After graduation in 1989, he has been with the School of Electronic Engineering at Xidian University, a lecturer from 1990 to1995, an associate professor from 1995 to 2000, a professor from 2000. From 1996 to 2002 he served as a vice dean of School of Electronic Engineering. From 2002, he was the executive dean of graduate school of Xidian University, also a vice chairman of the Academic Degree Evaluation Committee. His primary areas of research have been radar signal processing, automatic targets recognition, multisensor information fusion and target tracking.

    Prof. Ji is a senior member of the Institute of Electrical and Electronics Engineers (IEEE), a member of IEEE Signal Processing Society and a member of IEEE Aerospace & Electronic Systems Society.

    Wenbo Zhang was born in Shaanxi, China in 1985. He received B.S. (2005) degree and M.S. (2009) degree in the School of Telecommunications Engineering at Xidian University.

    Currently, he is pursuing a Ph.D. degree in the School of Electronic Engineering at Xidian University. His research interests include pattern recognition, support vector machine, extreme learning machine.

    View full text