Image classification by multimodal subspace learning

Abstract

In recent years we witnessed a surge of interest in subspace learning for image classification. However, the previous methods lack of high accuracy since they do not consider multiple features of the images. For instance, we can represent a color image by finding a set of visual features to represent the information of its color, texture and shape. According to the “Patch Alignment” Framework, we developed a new subspace learning method, termed Semi-Supervised Multimodal Subspace Learning (SS-MMSL), in which we can encode different features from different modalities to build a meaningful subspace. In particular, the new method adopts the discriminative information from the labeled data to construct local patches and aligns these patches to get the optimal low dimensional subspace for each modality. For local patch construction, the data distribution revealed by unlabeled data is utilized to enhance the subspace learning. In order to find a low dimensional subspace wherein the distribution of each modality is sufficiently smooth, SS-MMSL adopts an alternating and iterative optimization algorithm to explore the complementary characteristics of different modalities. The iterative procedure reaches the global minimum of the criterion due to the strong convexity of the criterion. Our experiments of image classification and cartoon retrieval demonstrate the validity of the proposed method.

Introduction

In many applications of computer vision and multimedia management, images are usually represented in different ways. This kind of data is termed as multimodal data. A typical example is a color image which has different features from different modalities (Tao et al., 2006, Bian and Tao, 2010), e.g., color, texture, and shape. Classifying images into meaningful categories is a challenging and important task. Many methods (Vapnik, 1998, Domingos and Pazzani, 1996) have been used for image classification in order to organize, represent and browse images better; and also they are to improve the performances of the related applications such as Content Based Image Retrieval (CBIR) (Liu, 2004) and image annotation and image indexing (Datta et al., 2008). In these methods, an image of n₁ × n₂ pixels is described by a feature vector with n₁ × n₂ dimensions. Because of the large dimension of this data (Donoho, 2000), these methods cannot work well in practice. Some previous studies show that the performance of image classification can be improved significantly in a low dimensional subspace (Belhumeur et al., 1997). Representatives of the subspace learning methods are the Principal Component Analysis (PCA) (Belhumeur et al., 1997), Linear Discriminant Analysis (LDA) (Belhumeur et al., 1997), Locally Linear Embedding (LLE) (Roweis and Saul, 2000), Laplacian Eigenmaps (LE) (Belkin and Niyogi, 2002, Belkin and Niyogi, 2003) and Local Tangent Space Alignment (LTSA) (Zhang and Zha, 2005). However, these methods cannot deal with data represented by multiple feature spaces.

Although much progress has been made for multimodal data learning including classification (Zien and Ong, 2007), clustering (Bickel and Scheffer, 2004) and feature selection (Zhao and Liu, 2008), little progress has been made in multimodal subspace learning. A possible solution is to concatenate vectors from different modalities to form a new vector, and to apply dimensionality reduction directly on the concatenated vector. However, further analysis shows that this solution also has its own problem as these features describe different aspects of an image’s properties, that is, they are intrinsically embedded in heterogeneous feature spaces. This concatenation ignores the diversity of modalities and thus cannot efficiently explore their complementary nature. The other considerable solution is the Distributed Multiple-view Subspace Learning (DMSL) proposed in (Long et al., 2008). DMSL performs a subspace learning algorithm on each modality independently, and then based on the obtained low dimensional representations, it learns a common low dimensional representation which is as close as possible to each representation. Although DMSL allows selecting different subspace learning algorithms for different modalities, the original data are invisible to the final learning process, and thus, it cannot effectively explore the complementary nature.

In this paper, we propose a novel method, termed Semi-Supervised Multimodal Subspace Learning (SS-MMSL), which learns a unified low dimensional subspace over all modalities simultaneously. We consider the problem of multimodal subspace learning for image classification based on the “Patch Alignment” Framework (PAF) (Zhang et al., 2009; Guan et al., 2011a, Guan et al., 2011b; Wang et al., 2011). According to PAF, the subspace learning techniques are applied in the stages of local patch construction and whole alignment. We apply the discriminative information revealed by the labeled data in building local patches. Besides, we introduce the unlabeled data (Belkin et al., 2005, Zhu et al., 2003, Zheng et al., 2008, Belkin and Niyogi, 2004) into the local patch construction and incorporate them in the whole alignment stage to obtain the optimal low dimensional subspace for each modality. In order to find a unified low dimensional subspace wherein the distribution of each modality is sufficiently smooth, we derive an iterative algorithm by using alternating optimization (Bezdek and Hathaway, 2002) to obtain a group of appropriate weights which effectively learns the complementary nature. Our experimental results of image classification and cartoon retrieval demonstrate the effectiveness of the proposed method.

The rest of this paper is organized as follows. In Section 2, we present the proposed Semi-Supervised Multimodal Subspace Learning (SS-MMSL) method and the solution to image classification using SS-MMSL. Experimental results are presented in Section 3. The application for cartoon retrieval is described in Section 4. And finally, conclusions are drawn in Section 5.