Image-To-Class distance is first proposed in Naive-Bayes Nearest-Neighbor. NBNN is a feature-based image classifier, and can achieve impressive classification accuracy. However, the performance of NBNN relies heavily on the large number of training samples. If using small number of training samples, the performance will degrade. The goal of this paper is to address this issue. The main contribution of this paper is that we propose a robust Image-to-Class distance by local learning. We define the patch-to-class distance as the distance between the input patch to its nearest neighbor in one class, which is reconstructed in the local manifold space; and then our image-to-class distance is the sum of patch-to-class distance. Furthermore, we take advantage of large-margin metric learning framework to obtain a proper Mahalanobis metric for each class. We evaluate the proposed method on four benchmark datasets: Caltech, Corel, Scene13, and Graz. The results show that our defined Image-To-Class Distance is more robust than NBNN and Optimal-NBNN, and by combining with the learned metric for each class, our method can achieve significant improvement over previous reported results on these datasets.