Multi-label enhancement based self-supervised deep cross-modal hashing

Abstract

Deep cross-modal hashing which integrates deep learning and hashing into cross-modal retrieval, achieves better performance than traditional cross-modal retrieval methods. Nevertheless, most previous deep cross-modal hashing methods only utilize single-class labels to compute the semantic affinity across modalities but overlook the existence of multiple category labels, which can capture the semantic affinity more accurately. Additionally, almost all existing cross-modal hashing methods straightforwardly employ all modalities to learn hash functions but neglect the fact that original instances in all modalities may contain noise. To avoid the above weaknesses, in this paper, a novel multi-label enhancement based self-supervised deep cross-modal hashing (MESDCH) approach is proposed. MESDCH first propose a multi-label semantic affinity preserving module, which uses ReLU transformation to unify the similarities of learned hash representations and the corresponding multi-label semantic affinity of original instances and defines a positive-constraint Kullback–Leibler loss function to preserve their similarity. Then this module is integrated into a self-supervised semantic generation module to further enhance the performance of deep cross-modal hashing. Extensive evaluation experiments on four well-known datasets demonstrate that the proposed MESDCH achieves state-of-the-art performance and outperforms several excellent baseline methods in the application of cross-modal hashing retrieval. Code is available at: https://github.com/SWU-CS-MediaLab/MESDCH.

Introduction

Recent years have witnessed a huge surge of multimedia data such as images, texts, audios and videos on the website. Because different data modalities may describe the same events or topics, we can take advantage of the potential semantically correlation among these multi-modal data to realize large-scale cross-modal data retrieval. Hence, cross-modal retrieval [1], [2], which is proposed to search semantically relevant instances from one modality by using a query of another modality, has drawn escalating attention. These different modality data have different feature representations and distributions. Thus, how to efficiently and effectively unify these massive yet heterogeneous modality data and further reduce their semantic gaps is still a big challenge.

The goal of cross-modal retrieval is to learn an isomorphic latent space, where the original heterogeneous feature representation of data from different modalities can be unified by a latent embedding. This is based on the hypothesis that different modalities having semantically related properties can be mapped and grouped into a common latent space. Existing cross-modal retrieval methods can be divided into two main categories [3]: real-valued representation learning methods and binary representation learning methods. The real-valued representation (such as subspace learning [4], [5], [6], [7], [8], [9], topic models [10], [11], [12], and deep models [13], [14], [15], [16], [17], [18], [19], [20], [21]) are usually measured by Euclidean distance to ensure that semantically relevant data are close to each other. However, the similarity measure in the real-valued representation space suffers from low search response and high computational complexity. Thus, binary codes are used, which have both low data storage requirements and a highly efficient distance measure (XOR operation) [22], [23]. Binary representation learning methods, also referred to as cross-modal hashing (CMH) [24] methods, could effectively project a high dimensional real-valued representation of multi-modal data into an isomorphic Hamming space, endowing similar cross-modal data representations with similar hash codes.

In general, existing cross-modal hashing methods can be further categorized into unsupervised and supervised methods. Unsupervised cross-modal hashing methods, such as inter-media hashing (IMH) [25], collective matrix factorization hashing (CMFH) [26], latent semantic sparse hashing (LSSH) [27], and unsupervised generative adversarial cross-modal hashing (UGACH) [28], learn the hash projection functions by exploring the underlying distributions and structures among the similarities of multi-modal data representations without using any further supervised information. Supervised cross-modal hashing methods learn the hash functions by mapping pairwise instances into pairwise binary codes and preserve the semantic relevance of the pairwise instances with the guidance of supervised information (such as semantic labels). Representative supervised CMH methods, such as semantic correlation maximization (SCM) [29], semantics preserving hashing (SePH) [30], dictionary learning cross-modal hashing (DLCMH) [31] and semi-relaxation supervised hashing (SRSH) [32], can effectively distill semantic correlations of cross-modal data by exploiting semantic labels and achieve superior performance compared to unsupervised CMH methods. Nevertheless, these methods are based on shallow architectures and cannot describe the complicated nonlinear correlations among different modalities. Moreover, in these methods, the hand-crafted feature extraction and hash function learning are independently performed, which might not be optimally compatible with each other and may result in suboptimal performance.

Recently, deep convolutional neural networks (CNNs) [33], [34], [35], [36] have made significant progress in various computer vision applications [37], [38], [39], [40], [41], [42], [43], [44], [45], [46], and have been adopted for cross-modal hashing retrieval. The deep neural network based cross-modal hashing methods, such as deep cross-modal hashing (DCMH) [47], pairwise relationship guided deep hashing (PRDH) [48], correlation hashing network (CHN) [49], collective deep quantization (CDQ) [50], self-supervised adversarial hashing (SSAH) [51], cross-modal hamming hashing (CMHH) [52], fast discrete cross-modal hashing (FDCH) [53], deep multiscale fusion hashing (DMFH) [54]. and triplet-based deep hashing (TDH) [55], integrate the learning of hash representations and hash functions into an end-to-end trainable architecture. Meanwhile, deep model based CMH methods can effectively capture nonlinear heterogeneous cross-modal correlations and obtain better performance than methods using shallow architectures.

However, most existing deep CMH methods simply leverage the single-class labels to define the semantic affinity of the original pairwise instances, and meanwhile, minimize the difference between the semantic affinity and the similarities of learned hash representations to preserve the semantically correlation of pairwise instances in the Hamming space. Nonetheless, this simple definition of semantic affinity cannot effectively preserve the semantic correlation and may lead to inferior retrieval performance. Actually, in cross-modal retrieval benchmark datasets as well as practical applications, these data from different modalities are usually labeled with multiple categories, e.g., multi-labels. This allows us to refine the definition of semantic affinity and base it on multi-labels information, thereby exploiting more accurate semantic relevance both for inter-modality and intra-modality pairwise instances (as shown in Fig. 1). However, the problem to effectively minimize the gap between the multi-label semantic affinity and the corresponding similarity of learned hash representations still remains. A common solution to minimize this gap is optimizing a MSE (Mean Square Error) based loss function [56]. However, the measure of MSE is based on Euclidean distance and is difficult to optimize. Moreover, MSE based loss functions are not robust to outlier pairs of instances [49]. To address this issue, a straight forward solution is constraining the similarities of learned hash representations (S) to fit their corresponding multi-label semantic affinity of original pairwise instances (P) by using a classical Kullback–Leibler divergence [57] based loss function (KL Loss) together with stochastic gradient descent optimization [58]. This leads to two other problems. Firstly, the ranges of S and P are not always the same, and it has been shown that it is ineffective to try to unify the ranges of S and P with simple linear transformations. Secondly, if most of the initial values of S are bigger than the corresponding values of P, the Kullback–Leibler divergence based loss function may generate negative loss values, which may work against the fitting goal during the optimization procedure (as the shown in Fig. 2).

Furthermore, most existing deep CMH methods simply use all modalities of the data to learn hash functions but neglect the fact that original instances in all data modalities may contain noise. These noises may reduce the performance and robustness of the learned hash representations and hash functions. At the same time, the assigned labels of instances constitute refinements for the original features in each modality and contain much semantic information, which usually shows little noise. Thus, a self-supervised semantic network based on multi-label annotations is usually utilized to improve the performance of deep CMH methods (as shown in Fig. 3). However, they define the semantic affinity matrix of instances based on single-label information, which cannot accurately capture the semantic affinity of original pairwise instances.

Taking the above problems into consideration, in this paper, we propose a novel and efficient multi-label enhancement based self-supervised deep cross-modal hashing (MESDCH) method to improve the robustness of learned hash representations and hash functions. As shown in Fig. 4, two novel modules are introduced in our MESDCH. The first one is the multi-label semantic affinity preserving module. This module mainly consists of three parts: the first part is a new definition of multi-labels semantic affinity, which aims to accurately exploit the semantic affinity of original pairwise instances under the supervision of multi-label information. The second part is a novel space transformation using a ReLU function to effectively unify the ranges of the similarities of learned hash representations and the corresponding multi-label semantic affinity of original pairwise instances in the Kullback–Leibler divergence based loss function. The third part is the proposed positive-constraint Kullback–Leibler divergence loss function which mainly prevents the Kullback–Leibler divergence based loss function from having negative loss values. The second one is self-supervised semantic generation module. This module effectively use multi-label annotations as a modal to supervise the hash representation and hash function learning with the aim to which alleviate the impact of noisy data in all modalities data. The proposed MESDCH merges the multi-label semantic affinity preserving module and self-supervised semantic generation module into deep cross-modal hashing based on three deep neural networks, LabelNet for multi-label modality, ImgNet for the image modality and TxtNet for the text modality. LabelNet plays as a supervisor role to guide the training of TxtNet and ImgNet, and the multi-label semantic affinity of both intra-modalities and inter-modalities are preserved by minimizing the difference between the multi-label semantic affinity matrix and the corresponding hash representation affinity matrix. Superior hash binary codes can be obtained by utilizing a sign function on the learned hash representations.

The main contributions of our work can be summarized as follows.

• A novel multi-label semantic affinity preserving module is proposed. In this module, a multi-label semantic affinity matrix is defined to accurately calculate the semantic relevance of original pairwise instances. A ReLU transformation is put forward to transform the range of similarities of learned hash representations close to the range of semantic affinity of original pairwise instances. A positive-constraint Kullback–Leibler divergence based loss function is defined to ensure the value of the loss function is non-negative during the hash functions learning procedure.

• To effectively lessen the influence of noisy data in original training instances during the hash functions learning procedure, the proposed multi-label enhancement based self-supervised deep cross-modal hashing method incorporate both the multi-label semantic affinity preserving module and the self-supervised semantic generation module into an end-to-end trainable architecture, which can further enhance the robustness of the learned hash representations and hash functions.

• Extensive experiments on four cross-modal retrieval benchmark datasets demonstrate that MESDCH significantly enhances the performance compared to CMH methods without the proposed modules. Furthermore, experimental results also show that our proposed MESDCH outperforms other state-of-the-art CMH methods.

The remainder of this paper is organized as follows. We briefly review the related works on cross-modal hashing retrieval in Section 2. Section 3 elaborates our proposed multi-label enhancement based self-supervised deep cross-modal hashing method. Section 4 presents the detailed optimizations used in our framework. Section 5 provides the experimental results and the corresponding analysis. Section 6 concludes our work.