Discrete asymmetric zero-shot hashing with application to cross-modal retrieval

Abstract

In recent years, cross-modal retrieval technology has attracted extensive attention with the massive growth of multimedia data. However, most cross-modal hashing methods mainly focus on exploring the retrieval of seen classes, while ignoring the retrieval of unseen classes. Therefore, traditional cross-modal hashing methods cannot achieve satisfactory performances in zero-shot retrieval. To mitigate this challenge, in this paper, we propose a novel zero-shot cross-modal retrieval method called discrete asymmetric zero-shot hashing(DAZSH), which fully exploits the supervised knowledge of multimodal data. Specifically, it integrates pairwise similarity, class attributes and semantic labels to guide zero-shot hashing learning. Moreover, our proposed DAZSH method combines the data features with the class attributes to obtain a semantic category representation for each category. Therefore, the relationships between seen and unseen classes can be effectively captured by learning a category representation vector for each instance. Therefore, the supervised knowledge can be transferred from the seen classes to the unseen classes. In addition, we develop an efficient discrete optimization strategy to solve the proposed model. Massive experiments on three benchmark datasets show that our proposed approach has achieved promising results in cross-modal retrieval tasks. The source code of this paper can be obtained from https://github.com/szq0816/DAZSH.

Introduction

Over the past decade, cross-modal retrieval tasks have become a great challenge owing to the exponential growth of multimodal data. In general, multimodal data are usually dependent and have essential connections in most cases. Therefore, it is fundamental to learn the correlation information between multimodalities in pattern recognition and machine learning, which is referred to as the heterogeneity gap. To resolve this discrepancy, traditional approaches try to project multimedia data into a common semantic space and then perform the retrieval task. However, real-value projections require more storage space and expensive computational costs due to the increase in multimedia data. This has become a significant obstacle in cross-modal retrieval applications. Therefore, hashing is an effective way to perform the retrieval task in large-scale datasets due to its low storage and high computational efficiency. It aims to project original samples into compact binary codes, which preserves their similarity in Hamming space. Recently, researchers have made many efforts to bridge the heterogeneity gap between multiple modalities and have achieved promising performances in many real applications [1], [2], [3], [4].

To the best of our knowledge, most existing cross-modal hashing retrieval methods are studied in the seen classes dataset [5], [6], [7]. However, with the explosive growth of multimedia data, some new concepts, such as unseen classes, have emerged in the past few years. Retraining the existing cross-modal hashing model after collecting new concept data is high cost and requires much storage space. Therefore, it is necessary to adopt a new cross-modal hashing model to deal with training data containing new concepts. Therefore, zero-shot learning aims to identify previously unseen data categories. Specifically, the trained classifier not only identifies the existing data categories in the training set, but also distinguishes the data from the unseen categories [8].

In the past several years, many zero-shot learning approaches have been applied for cross-modal retrieval [9], [10], [11]. Yang et al. [9] realized the potential semantic transfer by projecting the labels into the word embedding space. Shi et al. [10] proposed a zero-shot hashing method based on the asymmetric ratio similarity matrix, which can significantly improve the ability of knowledge transfer from seen classes to unseen classes. Transductive zero-shot hashing (T-MLZSH) [11] was proposed as a multilabel image retrieval model based on zero-shot learning. In this model, the labels of unseen classes are predicted by the instance-concept coherence ranking. Nevertheless, all the abovementioned works were applied to single-modality retrieval tasks, and there are still few efforts on unseen cross-modality retrieval tasks. With the continuous emergence of new concepts, existing cross-modal retrieval methods have the following limitations. (1) They only consider data from the seen categories and ignore the unseen cases. Therefore, these models are unsuitable for cross-modal retrieval with mixed unseen data. (2) Most of them neglect the class attribute information in hash code learning and thus are inconducive to knowledge transfer from seen classes to unseen classes. (3) Existing zero-shot hashing approaches fail to consider the pairwise similarity, class labels and class attributes to train models at the same time.

To address the above challenges, in this work, a novel zero-shot hashing method, called discrete asymmetric zero-shot hashing(DAZSH), is proposed for cross-modal retrieval. It integrates pairwise similarity, semantic labels and category attributes into a framework to fully explore semantic information. Specifically, we combine the data features with category attributes to obtain the category representation vector of each instance. In addition, the relationship between seen classes and unseen classes can be better captured, and the supervision knowledge can be transferred from the seen classes to the unseen classes. Fig. 1 shows the framework of our proposed DAZSH method in zero-shot cross-modal retrieval. The experimental results on three datasets show that our proposed DAZSH method can achieve better retrieval performances in dealing with unseen data.

The contributions of this work are as follows:

• We propose a unified discrete asymmetric zero-shot framework for learning hash codes. It combines data features with class attributes to learn an attribute space for each modality. Therefore, we explore the relationship between the seen classes and the unseen classes, which can transfer the supervision information from the seen classes to the unseen classes. In addition, our proposed approach aims at embedding the labels into the attribute space to improve retrieval accuracy. Therefore, our proposed model can generate more discriminative hash codes than traditional hashing methods.

• To maintain the characteristics of each modality, we generate different hash codes for different modalities using the asymmetric similarity strategy. Furthermore, we employ the maximum likelihood estimation algorithm to explore the pairwise similarity of multimodality data. To the best of our knowledge, this study is the first to utilize the pairwise similarity of different modalities using the maximum likelihood estimation algorithm in the cross-modal zero-shot learning field. Compared with traditional cross-modal zero-shot learning methods, our proposed method can effectively alleviate the heterogeneity gap between different modalities and more closely connect the seen and unseen classes, simultaneously.

• We develop a discrete optimization scheme to solve our proposed model, and then give its complexity analysis. Comprehensive experimental results on three benchmark datasets have shown the superiority of our proposed DAZSH method in different retrieval tasks.

The remainder of this paper is organized as follows: Section 2 introduces the previous work on cross-modal retrieval. Section 3 details our approach and its optimization scheme. Section 4 gives the experimental results and their analysis. Section 5 draws the conclusion of this work.