A novel competitive learning algorithm for the parametric classification with Gaussian distributions

doi:10.1016/S0167-8655(00)00004-0

Pattern Recognition Letters

Volume 21, Issue 5, May 2000, Pages 375-380

https://doi.org/10.1016/S0167-8655(00)00004-0 Get rights and content

Abstract

A competitive learning algorithm for the parametric classification of Gaussian sources is presented in this letter. The algorithm iteratively estimates the mean and prior probability of each class during the training. Bayes rule is then used for classification based on the estimated information. Simulation results show that the proposed algorithm outperforms k-means and LVQ algorithms for the parametric classification.

Introduction

Neural networks are known to be effective for pattern recognition Haykin, 1994, Lin and Lee, 1996. For the non-parametric problems, high classification accuracy can be achieved using the supervised learning techniques (Kohonen, 1990a). In the parametric cases, since the input distributions are available, further improvement in the classification accuracy is possible if the learning algorithms can estimate and fully utilize the input distribution information.

Under the assumptions that the input distributions are Gaussian with known covariance matrices, the objective of this letter is to present a novel competitive learning (CL) algorithm which correctly estimates the prior probability and mean of each class so that the classification error can be minimized. The number of neural units in the network is identical to the number of classes to be classified. During the training process, the CL training process iteratively estimates the prior probability and mean of each class. Then the Bayes rule is used based on the estimated information for the classification. The algorithm is simple to implement and has a higher parameter estimation accuracy than that of the other estimation algorithms such as k-means technique (Duda and Hart, 1973). In addition, although the algorithm is unsupervised in nature, simulation results show that it can have less classification error than that of the many existing supervised learning techniques (Kohonen, 1990a) for classification.

Section snippets

The Algorithm

Consider an N-class parametric problem. Suppose an n-dimensional observation vector x is drawn from a Gaussian mixture with distribution $p(x)=∑_{i=1}^{N} p(x /ω_{i})p(ω_{i}),$ where $ω_{i}, i=1,…,N$ , represents the class, p(ω_i) is the prior probability of class i and $p(x /ω_{i})$ is Gaussian with mean $u_{i}$ and covariance matrix Σ_i. Different observation vectors are independently classified. The minimum probability of error rule for determining ω_i from an observation $x$ follows directly from Bayes' Theorem (Schalkoff, 1992,

Simulation results

To demonstrate the effectiveness of the algorithm, we first consider a three-class Gaussian mixture with diagonal covariances. The dimension of the vectors drawn from the mixture is 4. The covariance matrices Σ₁,Σ₂, and Σ₃ are given as $I,4 I$ and $9 I$ , respectively, where $I$ is a 4×4 identity matrix.

Table 1 shows the true and estimated mean and prior probabilities for each class measured from the training data consisting of 70,000 vectors. The learning rates η_i(j),i=1,…,N, at the jth iteration of

Conclusion

A novel competitive learning algorithm is presented for parametric classification. The algorithm is able to accurately estimate the means and prior probabilities of Gaussian sources. The classification error rate of the Bayes classifiers based on the estimated parameters obtained by our algorithm is almost identical to the optimal Bayes classifiers based on the true parameters. In addition, our algorithm also outperforms the k-means and LVQ techniques for classification.

References (11)

W.J. Hwang et al.
A novel entropy-constrained competitive learning algorithm for vector quantization
Neurocomputing
(1999)
Dony, R.D., Haykin, S., 1995. Neural network approaches to image compression. In: Proceedings of the IEEE, Vol. 83, No....
R.O. Duda et al.
Pattern Classification and Scene Analysis
(1973)
A. Gersho et al.
Vector Quantization and Signal Compression
(1992)
S. Haykin
Neural Networks
(1994)

There are more references available in the full text version of this article.

Cited by (5)

Efficient pipelined architecture for competitive learning
2011, Journal of Parallel and Distributed Computing
Citation Excerpt :
Clustering techniques are effective for data approximation, data categorization and information extraction. They are widely used in numerous applications such as signal compression [9,14,15], pattern classification [18,30], data mining [26] and others [6]. Such techniques have also gained a crucial role in many data intensive applications [4,7,8,25], which involve the processing of massive data sets.
This paper presents a novel pipelined architecture for competitive learning (CL). The architecture is implemented by the field programmable gate array (FPGA). It is used as a hardware accelerator in a system on programmable chip (SOPC) for reducing the computation time. In the architecture, a novel codeword swapping scheme is adopted so that neuron competitions for different training vectors can be operated concurrently. The neuron updating process is based on a hardware divider with simple table lookup operations. The divider performs finite precision calculations for area cost reduction at the expense of slight degradation in training performance. The CPU time of the NIOS processor executing the CL training with the proposed architecture as an accelerator is measured. Experimental results show that the NIOS processor with the proposed architecture as an accelerator can achieve up to a speedup of 254 over its software counterpart running on a general purpose processor Pentium IV without hardware support.
Efficient K-winner-take-all competitive learning hardware architecture for on-chip learning
2012, Sensors (Switzerland)
An efficient pipelined architecture for fast competitive learning
2010, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Hardware implementation of κ-winner-take-all neural network with on-chip learning
2010, Proceedings - 2010 13th IEEE International Conference on Computational Science and Engineering, CSE 2010
A generalized competitive learning algorithm on Gaussian mixture with automatic model selection
2006, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

View full text

A novel competitive learning algorithm for the parametric classification with Gaussian distributions

Abstract

Introduction

Section snippets

The Algorithm

Simulation results

Conclusion

Neurocomputing

Pattern Classification and Scene Analysis

Vector Quantization and Signal Compression

Neural Networks