Exploring Root Causes of CNN-Based Image Classifier Failures Using 3-Nearest Neighbors

Abstract

The black-box characteristics of deep learning models make it imperative to explain decision-making mechanisms of these models in a way that humans can understand or interpret. This requirement has become more important as artificial intelligence-based applications take increasingly more critical roles in our lives. To this end, we, in this study, develop a model-agnostic post-hoc explanation method that brings explanations such as underfitting, overfitting, or outlier particularly about the possible root cause of inaccurate decisions produced by convolutional neural network (CNN) based image classification models. Our approach relies on analysis of the model’s response to the k-nearest neighbors in training dataset of the mispredicted test instance. To find visually and semantically similar images in the process of extracting k-nearest neighbors, we measure the distance between the features extracted from internal layers of the model. For the experimental analysis, we first build several underfitted, overfitted and well-fitted CNN models for MNIST and CIFAR-10 datasets. Then, for each different model, we identify the mispredicted test samples in these datasets, and extract their 3-nearest neighbors from the related training sets. We feed the extracted 3-nearest neighbors into the associated model and perform both sample-based and statistical post-hoc explanation for the inaccurately predicted test samples based on the models’ responses to the 3-nearest neighbors.