Fast implementations of nearest neighbor classifiers

doi:10.1016/S0031-3203(96)00098-2

Pattern Recognition

Volume 30, Issue 3, March 1997, Pages 459-465

https://doi.org/10.1016/S0031-3203(96)00098-2 Get rights and content

Abstract

Standard implementations of non-parametric classifiers have large computational requirements. Parzen classifiers use the distances of an unknown vector to all N prototype samples, and consequently exhibit O(N) behavior in both memory and time. We describe four techniques for expediting the nearest neighbor methods: replacing the linear search with a new kd tree method, exhibiting approximately $O (N^{12})$ behavior; employing an L_∞ instead of L₂ distance metric; using variance-ordered features; and rejecting prototypes by evaluating distances in low dimensionality subspaces. We demonstrate that variance-ordered features yield significant efficiency gains over the same features linearly transformed to have uniform variance. We give results for a large OCR problem, but note that the techniques expedite recognition for arbitrary applications. Three of four techniques preserve recognition accuracy.

References (14)

J.L. Blue et al.
Evaluation of pattern classifiers for fingerprint and OCR applications
Pattern Recognition
(1994)
K. Fukunaga
Introduction to Statistical Pattern Recognition
D.F. Specht
Probabilistic neural networks
Neural Networks
(1990)
T.M. Cover et al.
Nearest neighbor pattern classification
IEEE Trans. Inf. Process.
(1967)
J.H. Friedman et al.
An algorithm for finding best matches in logarithmic expected time
ACM Trans. Math. Software
(1977)
P.E. Hart
The condensed nearest neighbor rule
IEEE Trans. Inf. Theory
(1968)

There are more references available in the full text version of this article.

Cited by (58)

A scheme for high level data classification using random walk and network measures
2018, Expert Systems with Applications
Supervised classification techniques are known to exploit physical information of the analysed data, such as similarity, distribution and other low level features. Despite the relevance of such features, recent works have showed that a higher variety of patterns can be detected by combining low level and high level features. In this paper, it is proposed a supervised classification technique which applies limiting probabilities of the random walk theory over underlying networks constructed from input labeled data. The appealing feature of the proposed approach is that the adjacency matrix which carries both physical and structural information about the data. Structural information are given by features extracted from network connections. The class of a given unlabeled sample is estimated by a heuristic called ease of access, which is measured by the random walk process over the adjacency matrix. Such approach makes the technique quite general as one can put distinct data measures of interest in the connection matrix of the underlying data network to guide the random walker. Specifically, we show examples of combining low and high level features in the proposed classification scheme. Simulation results using artificial and real data sets suggest that the proposed technique is not only competitive with current and established classification techniques, but it also can reveal intrinsic structural patterns formed by the input data.
Improving scalability of ART neural networks
2017, Neurocomputing
Citation Excerpt :
The idea here is the hierarchical organization of data into a tree structure using the features in order to accelerate access to it. Although k-d Trees can be fast in finding neighbors, as in the case of the k-Nearest-Neighbors (kNN) classifier [17], their representation of a hyperbox by only one point is too inaccurate for classification. Approaches using hyperboxes directly as R-Tree (and even X-tree, its extension for high dimensions) are slower when searching for prototypes in the neighborhood of a test sample than the activation of all prototypes of ML-ARAM, especially in classification problems with thousands of features.
With the increasing amount of available data, the need for classification of large data volumes is permanently growing. In order to cope with this challenge, neural classifiers should be adapted to large-scale data. We present here a well scalable extension to the fuzzy Adaptive Resonance Associative Map (ARAM) neural network, which was specially developed for the quick classification of high-dimensional and large data. This extension aims at increasing the classification speed by adding an extra layer for clustering learned prototypes into large clusters. This enables the activation of only one or a few clusters i.e. a small fraction of all prototypes, reducing the classification time significantly. Further we introduce two methods to adapt this extension to a multi-label classification task.
NNMap: A method to construct a good embedding for nearest neighbor classification
2015, Neurocomputing
Citation Excerpt :
To overcome the above limitations of NN classification, in this work, we propose a new method called as NNMap, which speeds up NN classifier through data embedding. The proposed method makes three key contributions to the current state of the art [9–14]. The first contribution is that the proposed method obtains an efficient distance metric to take place of the original expensive distance.
This paper aims to deal with the practical shortages of nearest neighbor classifier. We define a quantitative criterion of embedding quality assessment for nearest neighbor classification, and present a method called NNMap to construct a good embedding. Furthermore, an efficient distance is obtained in the embedded vector space, which could speed up nearest neighbor classification. The quantitative quality criterion is proposed as a local structure descriptor of sample data distribution. Embedding quality corresponds to the quality of the local structure. In the framework of NNMap, one-dimension embeddings act as weak classifiers with pseudo-losses defined on the amount of the local structure preserved by the embedding. Based on this property, the NNMap method reduces the problem of embedding construction to the classical boosting problem. An important property of NNMap is that the embedding optimization criterion is appropriate for both vector and non-vector data, and equally valid in both metric and non-metric spaces. The effectiveness of the new method is demonstrated by experiments conducted on the MNIST handwritten dataset, the CMU PIE face images dataset and the datasets from UCI machine learning repository.
Fuzzy nearest neighbor algorithms: Taxonomy, experimental analysis and prospects
2014, Information Sciences
Citation Excerpt :
The aforementioned drawbacks have been analyzed extensively by the research community. As a result, many approaches have been proposed regarding, for example, the computation of similarity measures [16], the optimum choice of the k parameter [60], the definition of weighting schemes for patterns and attributes [96,51], the adaptation of the algorithm to data [43], the development of fast and approximate versions of the NN rule, devised to quicken the computation of the nearest neighbors [37,74,5,70], and the reduction of the training data [31,91,22,26]. Fuzzy Sets Theory (FST) [107] has been the basis of a remarkable number of these approaches.
In recent years, many nearest neighbor algorithms based on fuzzy sets theory have been developed. These methods form a field, known as fuzzy nearest neighbor classification, which is the source of many proposals for the enhancement of the k nearest neighbor classifier. Fuzzy sets theory and several extensions, including fuzzy rough sets, intuitionistic fuzzy sets, type-2 fuzzy sets and possibilistic theory are the foundations of these hybrid techniques, designed to tackle some of the drawbacks of the nearest neighbor rule.
In this paper the most relevant approaches to fuzzy nearest neighbor classification are reviewed, as are applications and theoretical works. Several descriptive properties are defined to build a full taxonomy, which should be useful as a future reference for new developments. An experimental framework, including implementations of the methods, datasets, and a suggestion of a statistical methodology for results assessment is provided. A case of study is included, featuring a comparison of the best techniques with several state of the art crisp nearest neighbor classifiers. The work concludes with the suggestion of some open challenges and ways to improve fuzzy nearest neighbor classification as a machine learning technique.
Neighbors' distribution property and sample reduction for support vector machines
2014, Applied Soft Computing Journal
Citation Excerpt :
The time complexity of naïve algorithm to solve kNN is O(n2). To speed up searching k-nearest neighbors, many scholars researched on it and proposed many faster algorithms [21–25]. In this paper, kNN is solved by naïve algorithm.
For data pre-processing of SVMs, many scholars tried to find those samples, which would become support vectors. Generally, support vectors locate in the overlap regions, which are between different classes. But overlap region does not always exist. In this paper, a new method is proposed to find the boundary regions of each class instead of overlap regions. This method could deal with the dataset without overlap regions. Summing the cosine of the sample-neighbor angle, the sum ranges from 0 to k. When the sample locates in the boundary region of data distribution, the sum would be close to k; when the sample locates in the interior of the data distribution, the sum would be close to 0. Using cosine sum, the samples locating in the interior of each class can be disposed before SVMs training. Experimental results show that the proposed method can solve the problem, which the methods based on finding overlap regions cannot deal with.
Boundary detection and sample reduction for one-class Support Vector Machines
2014, Neurocomputing
Citation Excerpt :
Without considering speedup strategies, the time complexity of solving kNN is O(n2). To speed up searching k-nearest neighbors, a large number of faster algorithms have been proposed [18–22]. This paper only uses the naïve algorithm to solve kNN.
There are a large number of target data and fewer outlier data in the problem of one-class classification. The sample reduction methods used for two-class or multi-class classification cannot be applied to this problem. This paper presents a novel method to reduce the scale of training set for one-class classification. This method only preserves the sample locating near the data distribution which may become support vector. If summing the cosine value of the angle which is between the difference between the sample and the neighbor, and the difference between the sample and mean of neighbors, the cosine sum will be close to k (k is the number of the neighbors), while the sample locating near the boundary; close to 0, while the sample locating within the data distribution. Experimental results demonstrate that the proposed method can reduce the scale of training set to get faster training speed and use less support vectors in the model of one-class SVMs, and the performance does not deteriorate.

View all citing articles on Scopus

View full text

Fast implementations of nearest neighbor classifiers

Abstract

Pattern Recognition

Neural Networks

Nearest neighbor pattern classification

IEEE Trans. Inf. Process.

An algorithm for finding best matches in logarithmic expected time

ACM Trans. Math. Software

The condensed nearest neighbor rule

IEEE Trans. Inf. Theory