The regularized LVQ1 algorithm

Abstract

This paper introduces a straightforward generalization of the well-known LVQ1 algorithm for nearest neighbour classifiers that includes the standard LVQ1 and the k-means algorithms as special cases. It is based on a regularizing parameter that monotonically decreases the upper bound of the training classification error towards a minimum. Experiments using 10 real data sets show the utility of this simple extension of LVQ1.

Introduction

Nearest neighbour (NN) methods [16] are still among the simplest and most successful ways of solving pattern recognition problems. As several comparative studies on real-world problems [39], [29] suggest, NN methods are often very competitive in comparison with more sophisticated and modern algorithms. Their success can be explained from a theoretical point of view, since they converge to Bayes classifier as the number of neighbours K and the number of prototypes M tend to infinite at an appropriate rate for all distributions. Also, with NN methods, there is a slightly higher probability of misclassifications occurring than with Bayes error Perr_B for K finite and M→∞ (i.e.: Perr_K-NN⩽(1+√(2/K))Perr_B and Perr_1-NN⩽2Perr_B [17]). These facts, combined with recent advances on memory-based systems [46], lazy methods [1] and local regression [20], have revived the interest for these techniques in the last decade. A plethora of new learning algorithms has been recently studied (e.g. [18], [19], [36] and other commented in Section 2.2).

NN classifiers are local learning systems [15]: they fit the training data only in a region around the location of an input pattern. Given a pattern x to classify, the K-nearest-neighbour classification rule is based on the following algorithm:

• Find the K nearest patterns to x in the set of prototypes P={(m_j, cl(m_j), j=0,…,M−1} where m_j is a prototype that belongs to one of the classes and cl(m_j) is the class indicator variable.

• Establish the classification by a majority vote amongst these K patterns.

NN classifiers allow for a variety of design choices that can be adjusted automatically from data such as the metric d to measure closeness between patterns, the number of neighbours K, the set of prototypes P and the size M. The most common similarity measurement d(x,y) is the Euclidean distance d(x,y)=||x−y||², while K can be automatically selected using a validation set, although K=1 is a common choice due to the following:

• Euclidean 1-NN classifiers form class boundaries with piecewise linear hyperplanes. They can therefore be used to solve a large class of classifiers since any border can be approximated by a series of locally defined hyperplanes.

• Most of the learning algorithms that compute P from training data work with 1-NN classifiers.

Finally, the design of the set of prototypes is the most difficult and challenging task. The simplest method would be to select the whole training set D_N={(x_i, cl(x_i)), i=0,…,N−1} (where x_i is a random sample of X and cl(x_i)) is the class label associated to x_i) as P. Nevertheless, this option would result in large memory and execution requirements in large databases. Therefore, in practice, a small set of prototypes of size M (with M⪡N) is mandatory. There are three main classes of learning algorithms that are designed to reduce the number of prototypes stored:

• Condensing algorithms: Since only near class border training data are useful in the classification process, condensing procedures aim to keep those points from training data which form class boundaries [17, Section 19].

• Editing algorithms: These retain any training patterns that fall inside class borders estimated to be within the same training set. Such patterns tend to form homogeneous clusters because only points at the centre of the natural groups in the data are retained [17, Section 26].

• Clustering algorithms: It is also feasible to use any NN vector quantization algorithm [23] (e.g. K-means [38]) to form a set of labelled prototypes). Firstly, we obtain a set of unlabelled prototypes from training data using the clustering algorithm. These prototypes can then be used to divide the input space in K-nearest-neighbour cells. Finally, we can assign labels to prototypes according to a majority vote by the training data in each cell [17, Section 21.5]. However, it is also possible to compute labelled centroids using a one-step learning strategy such as learning vector quantization (LVQ) algorithms [30].

As pointed out in [17, Section 19.3], clustering algorithms seem preferable to condensing and editing algorithms for the following reason: if the values of P are allowed to be arbitrary, prototypes are not constrained to training points. A more flexible class of classifiers can therefore be designed. However, the most preferable strategy for designing prototypes is to minimize the empirical classification error produced in the training set D_N [17, p. 311], since generalization error bounds for the 1-NN classifier based on the VC theory [17, Section 19] can be applied. Our work here shows that LVQ1 does not minimize the classification error, but a simple and straightforward generalization that introduces a regularizing parameter monotonically decreases the upper bound of the misclassification rate of the 1-NN classifier and thus improves the classification results obtained by LVQ1.

The paper is organized as follows. In Section 2, a review of LVQ1 and its basic limitations is presented. Section 3 introduces and analyses RegLVQ1, a regularized form of the LVQ1 algorithm, which controls the upper bound of the training error. Section 4 includes a comparative empirical study of RegLVQ1 and other learning algorithms for 1-NN classifiers in ten real-world problems. Finally, some conclusions are given in Section 5.