Zero-shot learning via a specific rank-controlled semantic autoencoder

Highlights

• The proposed LSA model solves the domain shift problem, and considers the low-rank structure of the reconstruction data.

• The proposed SRSA model can avoid simultaneously minimizing the variance of reconstruction data.

• Comprehensive experiments on some datasets demonstrate the effectiveness of the proposed approaches.

Abstract

Existing embedding zero-shot learning models usually learn a projection function from the visual feature space to the semantic embedding space, e.g. attribute space or word vector space. However, the projection learned based on seen samples may not generalize well to unseen classes, which is known as the projection domain shift problem in ZSL. To address this issue, we propose a method named Low-rank Semantic Autoencoder (LSA) to consider the low-rank structure of seen samples to maintain the sparse feature of reconstruction error, which can further improve zero-shot learning capability. Moreover, to obtain a more robust projection for unseen classes, we propose a Specific Rank-controlled Semantic Autoencoder (SRSA) to accurately control of the projection’s rank. Extensive experiments on six benchmarks demonstrate the superiority of the proposed models over most existing embedding ZSL models under the standard zero-shot setting and the more realistic generalized zero-shot setting.

Introduction

Humans can identify a lot of categories, specifically, about 30,000 basic classes and even more sub-classes. At the same time, humans are also very good at recognizing objects even if they have never seen any of their examples, which is considered as the problem of zero-shot learning (ZSL) in machine learning. For example, if a child has seen cattle before, he/she can easily recognize a cow and learn that a cow looks like cattle with black-and-white color. Therefore, in machine learning, ZSL tries to recognize classes whose samples haven’t been available during training time [1], [2], [3]. Recently, the number of new models has been increasing rapidly.

Zero-shot recognition mainly makes use of the training set with labeled seen classes and the semantic relationship between seen and unseen classes. The semantic embedding space is defined as a high dimensional vector space where seen and unseen classes are usually related, which can be a semantic attribute space [4] or a semantic word vector space [5]. The class prototype denotes that both unseen and seen classes are embedded as vectors and the relationships between classes in the semantic embedding space are usually measured by a distance. For example, the prototypes of cows and bison should be closer than those of cows and desks. With respect to the bridge between the visual feature space and the semantic embedding space, most existing ZSL methods tend to learn a projection function that can project the data from the visual feature space to the semantic embedding space with the labeled training set including seen classes only. When classifying unseen objects, the function is used to project the visual representation of unseen classes into the same semantic embedding space including unseen and seen classes. Then the nearest neighbor (NN) search method is used to recognize the sample of unseen class, which is the testing and recognizing process.

In early works of ZSL [6], the attributes within a two-stage approach are widely used to predict the label of a sample that belongs to one of the unseen classes. For example, Direct Attributes Prediction (DAP) [4] learns probabilistic attribute classifiers to estimate the posterior of each attribute for a sample. Then it calculates class posteriors and infers class labels with maximum a posterior (MAP) estimate. In addition, Indirect Attribute Prediction (IAP) [4] infers the class posterior of seen classes, and it makes use of the probability of each class to predict the attribute posteriors of each sample. Then, the class posterior of unseen classes can be predicted with the help of a multiclass classifier. Moreover, the two-stage approach is also used when attributes are not available. For instance, CONSE [7] first infers seen class posteriors, then the image feature will be projected into the word2vec space [8].

Recently, most ZSL models directly learn a mapping from the feature space to the semantic space. For example, ALE [9] learns a bilinear compatibility function between the feature space and the semantic space with ranking loss. Likewise, SJE [10] learns the bilinear compatibility but with the structure SVM loss optimized. Alternatively, DeViSE [11] learns a linear mapping using an efficient ranking loss formulation, which is evaluated on the large-scale ImageNet dataset. ESZSL [12] learns the bilinear compatibility using the square loss and regularizes the objective using the Frobenius norm.

However, ZSL suffers from the domain shift problem [13] that the visual representations of the same attributes may look completely different in unseen classes. To solve this problem, Kodirov et al. [14] proposed a novel approach called Semantic Autoencoder (SAE) for ZSL. Firstly, the visual feature representation of an image is projected into a semantic space by an encoder, which is the same as the conventional ZSL model. Secondly, the visual feature projection in the semantic space is projected back to the original space as the reconstructed visual feature representation by the decoder. Although SAE works well in many datasets, it does not take the low-rank structure of the data into account, which means that the robustness of the model is not strong when encountering the images in the wild. For example, sharks, dolphins and blue whales all have similar features, e.g.,“a tail”, so we hope to jointly optimize low-rank embedding and reconstruction to capture shared discriminative features across seen and unseen classes (See Fig. 1). We propose a novel approach called Low-rank Semantic Autoencoder (LSA) for zero-shot learning in this paper. The LSA model not only solves the domain shift problem, but also considers the low-rank structure of the reconstruction data and the sparse feature of reconstruction error. In addition, we additionally propose a Specific Rank-controlled Semantic Autoencoder (SRSA) for achieving an explicit control on the rank of projection. This model can avoid simultaneously minimizing the variance of reconstruction data while minimizing the objective’s rank. Experimental results on six benchmark datasets demonstrate that our proposed LSA and SRSA significantly outperform existing state-of-the-art ZSL models.