A novel two-level nearest neighbor classification algorithm using an adaptive distance metric

Abstract

When there exist an infinite number of samples in the training set, the outcome from nearest neighbor classification (kNN) is independent on its adopted distance metric. However, it is impossible that the number of training samples is infinite. Therefore, selecting distance metric becomes crucial in determining the performance of kNN. We propose a novel two-level nearest neighbor algorithm (TLNN) in order to minimize the mean-absolute error of the misclassification rate of kNN with finite and infinite number of training samples. At the low-level, we use Euclidean distance to determine a local subspace centered at an unlabeled test sample. At the high-level, AdaBoost is used as guidance for local information extraction. Data invariance is maintained by TLNN and the highly stretched or elongated neighborhoods along different directions are produced. The TLNN algorithm can reduce the excessive dependence on the statistical method which learns prior knowledge from the training data. Even the linear combination of a few base classifiers produced by the weak learner in AdaBoost can yield much better kNN classifiers. The experiments on both synthetic and real world data sets provide justifications for our proposed method.

Introduction

The nearest neighbor (kNN) rule is one of the oldest and most accurate methods to obtain nonlinear decision boundaries in classification problems [1], [2], [3]. Given a training data set, the kNN predicts the class label of an unlabeled test sample by the majority label among its k-nearest neighbors in the training set. It has been shown that the 1 nearest neighbor rule has asymptotic error rate that is at most twice the Bayes error rate, independent on the used distance metric.

When there are an infinite number of training samples in the training set, the class conditional probabilities are constant in some infinitesimal region around an unlabeled test sample. Thus, Euclidean distance is the natural choice as the distance metric to identify the nearest neighbors because Euclidean distance implies that the input space is isotropic or homogeneous. In the absence of prior knowledge, most kNN based methods use simple Euclidean distances to measure the similarities between samples. However, it is impossible that the number of training samples is infinite in the training set. Therefore, the assumption of locally constant class conditional probabilities is invalid with limited samples, and that is, Euclidean distance fails to capitalize on any statistical regularity in the training data. Hence, selecting a distance measure becomes crucial in determining the outcome of nearest neighbor classification.

Distance metric learning has played a significant role in both statistical classification and information retrieval. For instance, previous studies [4], [5] have shown that appropriate distance metrics can significantly improve the classification accuracy of kNN algorithm. Most of the work in distance metrics learning can be organized into the following two categories:

• The metric is learned based on discriminant analysis [4], [5], [6], [7], [8], [9], [10], [11], [12].

The majority of the previous works in this area attempt to determine a location in input space where nearest neighbors of an unlabeled test sample always belong to the same class while samples from different classes are separated. However, the nearest neighbors of an unlabeled test sample should be chosen so that class conditional probabilities are equal to or approximately equal to the unlabeled test sample, which is not taken into account by the previous studies. In addition, Yang et al. [8] show that the goals of keeping data points within the same classes close together while separating data points from different classes conflict and cannot be simultaneously satisfied when classes exhibit multimodal data distributions. Graepel and Herbrich [10] show that most of previous works in this area cannot incorporate data invariance to known transformations which has been shown to improve the accuracy of a classifier. For example, Weinberger and Saul [5] learn a Mahalanobis distance metric for kNN classification so that the k-nearest neighbors always belong to the same class while samples from different classes are separated by a large margin (LMNN). However, LMNN does not incorporate data invariance to known transformations. Domeniconi and Gunopulos [6] learn a local flexible metric technique using support vector machines (LFM-SVM), where SVM is used as guidance to define a local flexible metric. However, when the kernel-based methods transform data in the input space to a high-dimensional feature space, data invariance cannot be maintained. Furthermore, it is difficult to choose a proper kernel function to perform this task. Finally, the majority of previous works in this area rely heavily on the discriminant analysis which learns prior knowledge from training samples. It is not surprising that poor performance of SVM in some cases, such as under-fitting or over-fitting of SVM, will result in a significant degradation in classification accuracy for LFM-SVM classifier.

• The metric is learned based on local statistical analysis.

Popular algorithms in this category include [13], [14], [15], [16], [17]. Zhang et al. [13] present an algorithm which learns a distance metric from a training data set by knowledge embedding. Using the new distance metric can determine the nearest neighbors of an unlabeled test sample in the input space. Their coordinates are close and local features are similar. A kNN algorithm based on the best distance measurement (kNN-BDM) is given in [14], [15]. The best distance measurement is optimized in order to minimize the mean-square error of misclassification rate of kNN with finite and infinite number of training samples. However, these methods depend on the local statistical information and lack global discriminant analysis. This makes the predictions of kNN highly sensitive to the noise or the sampling fluctuations associated with the random nature of the process producing the training data and leads to high variance predictions.

In this paper, we proposed a novel two-level nearest neighbor algorithm (TLNN). We firstly use AdaBoost as guidance to define a local flexible metric. AdaBoost has been successfully used as a classification tool in a variety of areas, and it is able to take advantage of weak hypotheses to maximize the minimum margin even if the training error of the combination of hypotheses is zero [18], [19], [20]. There are theoretical bounds on the generalization error of linear classifiers [18], [21], [22], which inspired AdaBoost convey desirable properties to the AdaBoost based learning algorithm, and therefore AdaBoost is the natural choice to guide the local information extraction. At the low-level of the TLNN, we use Euclidean distance to determine a local subspace centered at an unlabeled test sample. Then at the high-level of the TLNN, AdaBoost is chosen to guide the local information extraction in the subspace centered at the unlabeled test sample.

The TLNN algorithm has several advantages. Firstly, data invariance is maintained and at the same time the highly stretched or elongated neighborhoods along different directions are produced. Secondly, the TLNN algorithm reduces the excessive dependence on the statistical methods which learn prior knowledge from labeled or unlabeled data. Even the linear combinations of a few base classifiers produced by the weak learner in AdaBoost can yield much better kNN classifiers. Thirdly, with the improvement of performance of AdaBoost, the TLNN can minimize the mean-absolute error of misclassification rate of kNN with finite and infinite number of training samples. Compared to other methods, our classification model is non-parametric.