Image classification using spatial pyramid robust sparse coding

Abstract

Recently, the sparse coding based codebook learning and local feature encoding have been widely used for image classification. The sparse coding model actually assumes the reconstruction error follows Gaussian or Laplacian distribution, which may not be accurate enough. Besides, the ignorance of spatial information during local feature encoding process also hinders the final image classification performance. To address these obstacles, we propose a new image classification method by spatial pyramid robust sparse coding (SP-RSC). The robust sparse coding tries to find the maximum likelihood estimation solution by alternatively optimizing over the codebook and local feature coding parameters, hence is more robust to outliers than traditional sparse coding based methods. Additionally, we adopt the robust sparse coding technique to encode visual features with the spatial constraint. Local features from the same spatial sub-region of images are collected to generate the visual codebook and encode local features. In this way, we are able to generate more discriminative codebooks and encoding parameters which eventually help to improve the image classification performance. Experiments on the Scene 15 dataset and the Caltech 256 dataset demonstrate the effectiveness of the proposed spatial pyramid robust sparse coding method.

Introduction

In recent years, the bag-of-visual-word (BoW) model has become popular in image classification. This model extracts appearance descriptors from local patches and quantizes them into discrete “visual words”, and then a compact histogram representation is used to represent images. The descriptive power of the BoW model is severely limited because it discards the spatial information of local descriptors. To overcome this problem, one popular extension method, called the spatial pyramid matching (SPM) by Lazebnik et al. (2006), is proposed and has been shown to be effective for image classification. The SPM partitions an image into several segments in different scales, then computes the BoW histogram within each segment and concatenates all the histograms to form a high dimension vector representation of the image.

To obtain good performance, researchers have empirically found that the SPM should be used together with SVM classifier using nonlinear Mercer kernels. However, the computational complexity is $O(n^3$) and the memory complexity is $O(n^2$) in the training phase, where n is the size of training dataset. This constrains the scalability of the SPM-based nonlinear SVM method. To reduce the training complexity and improve image classification performance, sparse coding based linear spatial pyramid matching methods (Yang et al., 2009, Serre et al., 2005, Wang et al., 2010) are proposed which help to improve classification performance. In fact, there is another constraint which was neglected in Yang et al., 2009, Wang et al., 2010, i.e., the spatial locality constraint. For example, ‘sky’ often lies on the upper side of images, while ‘beach’ often lies on the lower side of images. When we try to encode an image region about the upper ‘sky’, it is more semantically meaningful to use the bases which are generated by the local features on the upper side of images. Similarly, it is more meaningful to encode the lower ‘beach’ with the bases generated from the local features on the lower side of images. We believe this spatial information should be combined with the codebook generation in order to encode local features more efficiently.

Besides, the sparse coding used in Yang et al. (2009) for local feature encoding tried to minimize the reconstruction error of local features by learning the optimal codebook and coding parameters simultaneously with sparsity constraints. After the codebook is learned, the rest local features are encoded by minimizing the reconstruction error with the learnt codebook and sparsity constraints. To ensure coding parameter’s smoothness and reduce encoding information loss, Laplacian sparse coding and non-negative sparse coding are proposed by Gao et al. (2010) and Zhang et al. (2011) respectively. Actually, these sparse coding models assume that the reconstruction error should be Gaussian or Laplacian distribution, which is unable to model real world applications. It would be more effective if we can construct a more robust model than simply assuming the Gaussian or Laplacian distribution of reconstruction error.

In this paper, we present a novel image classification method by using spatial pyramid robust sparse coding (SP-RSC). We give the flowchart in Fig. 1. We first partition images into sub-regions on multiple scales. Then we adopt the robust sparse coding approach to generate the codebook and encode local features of images with the spatial constraint. Different from SPM (Lazebnik et al., 2006), the proposed SP-RSC based visual vocabulary is concatenated with each encoding results from the sub-regions which have the same spatial locality and segmentation scale. For the robust sparse coding, we adopt the maximum likelihood estimation (MLE) approach and try to minimize some function of the coding residuals. This function is associated with the distribution of the coding residuals which robustly encodes the given local feature with sparse regression coefficients. Experimental evaluations on two public datasets demonstrate the effectiveness of the proposed method.

Compared with our previous work (Zhang et al., 2010), we extended the spatial pyramid coding by using robust sparse coding instead of sparse coding both for codebook construction and local feature encoding. The sparse coding assumes the reconstruction error follows the Gaussian or Laplacian distribution while the robust sparse coding has no such constraints, hence helps to encode the local features more efficiently. Besides, more experiments are added to clarify the effectiveness of the proposed spatial pyramid robust sparse coding method.

The rest of the paper is organized as follows. Section 2 gives an overview of some related work. In Section 3, we present the details of the proposed spatial pyramid robust sparse coding method. Experimental results and analysis are given in Section 4. Finally, we give the conclusions in Section 5.