Improving the k-NCN classification rule through heuristic modifications

doi:10.1016/S0167-8655(98)00108-1

Pattern Recognition Letters

Volume 19, Issue 13, November 1998, Pages 1165-1170

https://doi.org/10.1016/S0167-8655(98)00108-1 Get rights and content

Abstract

This paper presents an empirical investigation of the recently proposed k-Nearest Centroid Neighbours (k-NCN) classification rule along with two heuristic modifications of it. These alternatives make use of both proximity and geometrical distribution of the prototypes in the training set in order to estimate the class label of a given sample. The experimental results show that both alternatives give significantly better classification rates than the k-Nearest Neighbours rule, basically due to the properties of the plain k-NCN technique.

Introduction

The k-Nearest Neighbours (k-NN) rule (Duda and Hart, 1973) is one of the most remarkable choices among non-parametric classification rules. This is a distance-based technique which classifies a test sample according to the classes of its k closest cases in a set of n previously labelled prototypes, X={x₁,…,x_n}. It is well known that the error for the k-NN rule tends towards the Bayes error in the asymptotic case (n→∞). However, in practice, due to the finite sample size, the k-NN estimates are no longer optimal. This problem becomes more relevant when the number of prototypes is not large enough compared to the dimensionality of the feature space (Fukunaga, 1990), which constitutes a very usual practical situation.

A number of alternative neighbourhood definitions have been applied to classification problems, trying to partially overcome the practical drawbacks pointed out for the k-NN rule. In particular, the concept of Nearest Centroid Neighbourhood (NCN) (Chaudhuri, 1996) along with the neighbourhood relation derived from the Gabriel and the Relative Neighbourhood graphs (Jaromczyk and Toussaint, 1992) have successfully been used in finite sample size situations (Sánchez et al., 1997). The resulting classification approaches have been generically referred to as surrounding rules because they try to look for prototypes not only close enough (in the basic distance sense) but also homogeneously or symmetrically distributed around a sample.

Although the surrounding classification schemes have been proven to outperform the k-NN rule in most cases, this kind of neighbourhood also suffers from some drawbacks due to the fact that it may contain some prototypes which are not sufficiently close to the test sample. Thus, this paper proposes some modifications of the k-NCN rule which try to solve this problem and then to achieve better results.

The organization of the rest of this paper is as follows. Section 2describes the NCN concept and the derived k-NCN classifier, as well as the conceptual differences with respect to the k-NN rule. In Section 3, two modifications of the k-NCN rule are introduced. Section 4provides an experimental study for both synthetic and real data sets. Finally, some concluding remarks are given in Section 5.

Section snippets

Surrounding neighbourhood

The k-NN rule consists of estimating the class of a given sample through its k closest prototypes in the training set. This technique considers that all the information required to classify a new sample can be obtained from a small subset of prototypes close to it. However, it does not take into account the geometrical distribution of those k prototypes with respect to the given sample, that is, in general the nearest prototypes do not completely surround the sample since the k-NN rule defines

Using proximity and spatial homogeneity for classification

We here propose two heuristic modifications of the k-NCN decision rule in order to improve its correct classification rate. These alternative schemes try to jointly use information about proximity as well as about the spatial distribution of prototypes around a given sample. In fact, although it has been empirically proven that the k-NCN rule may outperform the k-NN classifier (Sánchez et al., 1997), some nearest centroid neighbours may be too far from the sample to classify, which can

Empirical comparison

Several experiments using both synthetic and real databases (Murphy and Aha, 1991) have been carried out in order to compare the efficiency of the classification schemes considered in this work. Five different random partitions (half of prototypes for training and half for testing purposes) of each original data set, have been used to obtain averaged measures about the performance of each classification rule. In particular, the focus of the present experimental study is on a comparison of the

Conclusions

Alternative approaches to neighbourhood-based classification have been considered in this work. In particular, the recently introduced k-NCN decision rule along with two heuristic modifications have been used. These extensions to the k-NCN technique try to take into account both proximity and geometrical distribution of the prototypes.

From the experiments carried out, it can be concluded that the modifications proposed here achieve even higher classification rates than the plain k-NCN rule,

References (10)

B.B. Chaudhuri
A new definition of neighbourhood of a point in multi-dimensional space
Pattern Recognition Letters
(1996)
J.S. Sánchez et al.
On the use of neighbourhood-based non-parametric classifiers
Pattern Recognition Letters
(1997)
Dasarathy, B.V., 1990. Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques. IEEE Computer Soc. Press, Los...
Devijver, P.A., Kittler, J., 1982. Pattern Recognition: A statistical Approach. Prentice-Hall, Englewood Cliffs,...
Duda, R., Hart, P.E., 1973. Pattern Classification and Scene Analysis. Wiley, New...

There are more references available in the full text version of this article.

Cited by (25)

A new two-layer nearest neighbor selection method for kNN classifier
2022, Knowledge-Based Systems
The $k$ -nearest neighbor (kNN) classifier is a classical classification algorithm that has been applied in many fields. However, the performance of the kNN classifier is limited by a simple neighbor selection method, called nearest neighbor (NN) rule, where only the neighborhood of the query is considered when selecting the nearest neighbors of the query. In other words, the NN rule only uses one-layer neighborhood information of the query.
In this paper, we propose a new neighbor selection method based on two-layer neighborhood information, called two-layer nearest neighbor (TLNN) rule. The neighborhood of the query and the neighborhoods of all selected training instances in this neighborhood are considered simultaneously, then the two-layer nearest neighbors of the query are determined according to the distance, distribution relationship, and backward nearest neighbor relationship between the query and all selected training instances in the above neighborhoods. In order to verify the effectiveness of the proposed TLNN rule, a $k$ -two-layer nearest neighbor (kTLNN) classifier is proposed to measure the classification ability of the two-layer nearest neighbors.
Extensive experiments on twenty real-world datasets from UCI and KEEL repositories show that the kTLNN classifier outperforms not only the kNN classifier but also seven other state-of-the-art NN-based classifiers.
A new locally adaptive k-nearest neighbor algorithm based on discrimination class
2020, Knowledge-Based Systems
The $k$ -nearest neighbor (kNN) rule is a classical non-parametric classification algorithm in pattern recognition, and has been widely used in many fields due to its simplicity, effectiveness and intuitiveness. However, the classification performance of the kNN algorithm suffers from the choice of a fixed and single value of $k$ for all queries in the search stage and the use of simple majority voting rule in the decision stage.
In this paper, we propose a new kNN-based algorithm, called locally adaptive $k$ -nearest neighbor algorithm based on discrimination class (DC-LAKNN). In our method, the role of the second majority class in classification is for the first time considered. Firstly, the discrimination classes at different values of $k$ are selected from the majority class and the second majority class in the $k$ -neighborhood of the query. Then, the adaptive $k$ value and the final classification result are obtained according to the quantity and distribution information on the neighbors in the discrimination classes at each value of $k$ .
Extensive experiments on eighteen real-world datasets from UCI (University of California, Irvine) Machine Learning Repository and KEEL (Knowledge Extraction based on Evolutionary Learning) Repository show that the DC-LAKNN algorithm achieves better classification performance compared to standard kNN algorithm and nine other state-of-the-art kNN-based algorithms.
Prototype selection for dissimilarity-based classifiers
2006, Pattern Recognition
A conventional way to discriminate between objects represented by dissimilarities is the nearest neighbor method. A more efficient and sometimes a more accurate solution is offered by other dissimilarity-based classifiers. They construct a decision rule based on the entire training set, but they need just a small set of prototypes, the so-called representation set, as a reference for classifying new objects. Such alternative approaches may be especially advantageous for non-Euclidean or even non-metric dissimilarities.
The choice of a proper representation set for dissimilarity-based classifiers is not yet fully investigated. It appears that a random selection may work well. In this paper, a number of experiments has been conducted on various metric and non-metric dissimilarity representations and prototype selection methods. Several procedures, like traditional feature selection methods (here effectively searching for prototypes), mode seeking and linear programming are compared to the random selection. In general, we find out that systematic approaches lead to better results than the random selection, especially for a small number of prototypes. Although there is no single winner as it depends on data characteristics, the k-centres works well, in general. For two-class problems, an important observation is that our dissimilarity-based discrimination functions relying on significantly reduced prototype sets (3–10% of the training objects) offer a similar or much better classification accuracy than the best k-NN rule on the entire training set. This may be reached for multi-class data as well, however such problems are more difficult.
On global, local, mixed and neighborhood kernels for support vector machines
1999, Pattern Recognition Letters
The basic principles of the support vector machine (SVM) are analyzed. Two approaches to constructing a kernel function which takes into account some local properties of a problem are considered. The first one deals with interactions between neighboring pixels in an image and the second with proximity of the objects in the input space. In the former case, this is equivalent to feature selection and the efficiency of this approach is demonstrated by an application to Texture Recognition. In the latter case, this approach may be considered as either a kind of local algorithm or as a mixture of local and global ones. We demonstrate that the use of such kernels increases the domain of SVM applications.
Geometric proximity graphs for improving nearest neighbor methods in instance-based learning and data mining
2005, International Journal of Computational Geometry and Applications
A new globally adaptive k-nearest neighbor classifier based on local mean optimization
2021, Soft Computing

View all citing articles on Scopus

¹: This work has partially been supported by projects P1B96-13 (Fundació Caixa-Castelló), and AGF95-0712-C03-01 and TIC95-676-C02-01 (Spanish CICYT).

View full text

Improving the k-NCN classification rule through heuristic modifications1

Abstract

Introduction

Section snippets

Surrounding neighbourhood

Using proximity and spatial homogeneity for classification

Empirical comparison

Conclusions

Pattern Recognition Letters

Pattern Recognition Letters

Improving the k-NCN classification rule through heuristic modifications¹