Optimization Strategies for the k-Nearest Neighbor Classifier

Abstract

In this paper, we propose six (6) fast and efficient classification schemes for different type of images (digits, objects, characters) using the classical k-nearest neighbor (kNN) classifier. It is common knowledge that kNN is one of the most popular supervised classification algorithm. However, for large data collections, the classification is time consuming because of the distance calculations which increase with the number of training samples and the data dimensionality. To reduce the number of training samples, we propose two techniques that employ the notions of convex envelope and stratified sampling. For the convex envelope, only the data points building the convex hull will serve as designated prototypes (training samples), thus reducing considerably the computational burden. To reduce the dimensionality of the data, we considered auto-encoder networks and vector embedding. The former is learning a way to compact data representation using statistical machine learning and the latter is changing the data itself emphasizing how each data sample is organized in space compared to others. The experiments on multiple benchmark data collections such as MNIST, Fashion-MNIST and Lampung characters show a considerable classification speed up (up to 32$\times$) with no significant drop in accuracy when compared to results obtained using the complete data.