Pipelining of Fuzzy ARTMAP without matchtracking: Correctness, performance bound, and Beowulf evaluation

Abstract

Fuzzy ARTMAP neural networks have been proven to be good classifiers on a variety of classification problems. However, the time that Fuzzy ARTMAP takes to converge to a solution increases rapidly as the number of patterns used for training is increased. In this paper we examine the time Fuzzy ARTMAP takes to converge to a solution and we propose a coarse grain parallelization technique, based on a pipeline approach, to speed-up the training process. In particular, we have parallelized Fuzzy ARTMAP without the match-tracking mechanism. We provide a series of theorems and associated proofs that show the characteristics of Fuzzy ARTMAP’s, without matchtracking, parallel implementation. Results run on a BEOWULF cluster with three large databases show linear speedup as a function of the number of processors used in the pipeline. The databases used for our experiments are the Forrest CoverType database from the UCI Machine Learning repository and two artificial databases, where the data generated were 16-dimensional Gaussian distributed data belonging to two distinct classes, with different amounts of overlap (5% and 15%).

Introduction

Neural Networks have been used extensively and successfully to tackle a wide variety of problems. As computing capacity and electronic databases grow, there is an increasing need to process considerably larger databases. In this context, the algorithms of choice tend to be ad hoc algorithms (Agrawal & Srikant, 1994) or tree based algorithms such as CART (King, Feng, & Shutherland, 1995) and C4.5 (Quinlan, 1993). Variations of these tree learning algorithms, such as SPRINT (Shafer, Agrawal, & Mehta, 1996) and SLIQ (Mehta, Agrawal, & Rissanen, 1996) have been successfully adapted to handle very large data sets.

Neural network algorithms, on the other hand, can have a prohibitively slow convergence to a solution, especially when they are trained on large databases. Even one of the fastest (in terms of training speed) neural network algorithms, the Fuzzy ARTMAP algorithm ((Carpenter et al., 1992, Carpenter et al., 1991), and its faster variations (Kasuba, 1993, Taghi et al., 2003)) tend to converge slowly to a solution as the size of the network increases.

One obvious way to address the problem of slow convergence to a solution is by the use of parallelization. Extensive research has been done on the properties of parallelization of feed-forward multi-layer perceptrons (Mangasarian and Solodov, 1994, Torresen et al., 1995, Torresen and Tomita, 1998). This is probably due to the popularity of this neural network architecture, and also because the backpropagation algorithm (Rumelhart, Hinton, & Williams, 1986), used to train these types of networks, can be characterized mathematically by matrix and vector multiplications, mathematical structures that have been parallelized with extensive success.

Regarding the parallelization of ART neural networks, the work by Manolakos (Manolakos, 1998) implements the ART1 neural network (Carpenter et al., 1991) on a ring of processors. To accomplish this Manolakos divides the communication in two bidirectional rings, one for the $F_1$ layer of ART1 and another for the $F_2$ layer of ART1. Learning examples are pipelined through the ring to optimize network utilization. Experimental results of Manolakos’ work indicate close to linear speed-up as a function of the number of processors. This approach is efficient for ring networks and it is an open question of whether it can be extended for Fuzzy ARTMAP. Another parallelization approach that has been used with ART and other types of neural networks is the systems integration approach where the neural network is not implemented on a network of computers but on parallel hardware. Zhang (Zhang, 1998) shows how a fuzzy competitive neural network similar to ARTMAP can be implemented using a systolic array. Asanović (Asanović et al., 1998) uses a special purpose parallel vector processor SPERT-II to implement back-propagation and Kohonen neural networks. In Malkani and Vassiliadis (1995), a parallel implementation of the Fuzzy-ARTMAP algorithm, similar to the one investigated here, is presented. However, in his paper, a hypercube topology is utilized for transferring data to all of the nodes involved in the computations. While it is trivial to map the hypercube to the more flexible switched network typically found in a Beowulf, this would likely come with a performance hit. In this approach each one of the processors maintains a subset of the architecture’s templates, and finds the template with the maximum match in its local collection. Finally, in the $d$-dimensional hypercube, all the processors cooperate to find the global maximum through $d$ different synchronization operations. This can eventually limit the scalability of this approach, since the value $d$ grows with the size of the hypercube, while the network bandwidth remains constant.

Mining of large databases is an issue that has been addressed by many researchers. Mehta (Mehta et al., 1996), developed SLIQ, a decision-tree based algorithm that combines techniques of tree-pruning and sorting to efficiently manage large datasets. Furthermore, Shafer (Shafer et al., 1996), proposed SPRINT, another decision-tree based algorithm, that removed memory restrictions imposed by SLIQ and is designed to be amenable to parallelization. The Fuzzy ARTMAP neural network has many desirable characteristics, such as the ability to solve any classification problem, the capability to learn from data in an on-line mode, the advantage of providing interpretations for the answers that it produces, the capacity to expand its size as the problem requires, and the ability to recognize novel inputs, among others. Due to all these virtues we investigate Fuzzy ARTMAP’s parallelization in an effort to improve its convergence speed to a solution when it is trained with large datasets.

There are many variants within the Fuzzy ARTMAP family of neural networks. Kasuba (Kasuba, 1993), with only classification problems in mind, develops a simplified Fuzzy ARTMAP structure (simplified Fuzzy ARTMAP), while Taghi et al., in Taghi et al. (2003), describe variants of simplified Fuzzy ARTMAP, called Fast Simplified Fuzzy ARTMAP, variants. These Fuzzy ARTMAP variants are faster than the original Fuzzy ARTMAP algorithm, because they eliminated all the computations performed in the ${\mathrm{ART}}_b$ module of Fuzzy ARTMAP, and because they have simplified the computations performed in the ${\mathrm{ART}}_{ab}$ module of Fuzzy ARTMAP; the results produced by these simplified Fuzzy ARTMAP variants are the same as the results produced by the original Fuzzy ARTMAP, when the problem at hand is a classification problem.

One of the Fuzzy ARTMAP fast algorithmic variants presented in Taghi et al. (2003) is called SFAM2.0 and it is this algorithmic Fuzzy ARTMAP variant (that is equivalent to Fuzzy ARTMAP for classification problems) that is the focus of our paper. Furthermore, in this paper, we only concentrate on the no-match tracking version of SFAM2.0. No-match tracking was a concept introduced by Anagnostopoulos in the framework of the ART networks (Anagnostopoulos & Georgiopoulos, 2003). No match-tracking is a specific ART network behavior, where whenever an input pattern is presented to the ART network and a category is chosen that maximizes the bottom-up input, passes the vigilance, but is mapped to the incorrect output, this category is deactivated and a new category (uncommitted category) is activated next that will encode the input pattern. As a reminder, in that case, the typical ART network behavior is to engage the match-tracking mechanism that deactivates the chosen category, increases the vigilance threshold and searches for another appropriate category that might be or might not be an uncommitted category. Anagnostopoulos has shown through experimentation in Anagnostopoulos and Georgiopoulos (2003) that no-match-tracking Fuzzy ARTMAP increases the number of categories created in the category representation layer compared to Fuzzy ARTMAP but it does so while providing improved generalization performance. No Match-tracking in Fuzzy ARTMAP should not be confused with the on-line operation in Fuzzy ARTMAP. On-line Fuzzy ARTMAP operation implies that an input–output pair is presented only once in Fuzzy ARTMAP’s training phase, and it can be used in a match-tracking or a no-match tracking Fuzzy ARTMAP. The reason that we focus on the no match-tracking Fuzzy ARTMAP is because it gives us the opportunity to first parallelize the competitive aspect of Fuzzy ARTMAP, while ignoring the complications of the feedback mechanism that matchtracking introduces. Finally, we focus on the on-line version of this network, since a parallelization of the on-line version extends in a straightforward fashion to the off-line version of the network.

For simplicity, we refer to this Fuzzy ARTMAP variant (on-line, no-matchhtracking SFAM2.0) simply as Fuzzy ARTMAP, or FAM. If we demonstrate the effectiveness of our parallelization strategies for FAM, extension to other ART structures can be accomplished without a lot of effort. This is due to the fact that the other ART structures share a lot of similarities with FAM, and as a result, the advantages of the proposed parallelization approach can be readily extended to other ART variants (for instance Gaussian ARTMAP (Williamson, 1996) and Ellipsoidal ARTMAP (Anagnostopoulos & Georgiopoulos, 2001), among others).

The remainder of this paper is organized as follows: Section 2 presents the Fuzzy ARTMAP neural network architecture and a few Fuzzy ARTMAP variants. Section 3 continues with the pseudo-code of the off-line, match-tracking Fuzzy ARTMAP, on-line match tracking Fuzzy ARTMAP, and on-line no-match tracking Fuzzy ARTMAP (referred to simply as FAM). Section 4 focuses on the computational complexity of the on-line, match-tracking Fuzzy ARTMAP, and serves as a necessary motivation for the parallelization approach introduced in this paper. Section 5 presents a discussion of the Beowulf cluster as our platform of choice. Section 6 continues with the pseudocode of the parallel Fuzzy ARTMAP, referred to as Pfam, and associated discussion to understand the important aspects of this implementation. Section 7 focuses on theoretical results related to the proposed parallelization approach. In particular, we prove there that Pfam is equivalent to FAM, and that the processors in the parallel implementation will be reasonably balanced by considering a worst case scenario. Furthermore, Section 8 proceeds with experiments and results comparing the performance of Pfam and FAM on three databases, one of them real and two artificial. The article concludes with Section 9, where a summary of our experiences, from the conducted work, and future research are delineated.