Intra-class low-rank regularization for supervised and semi-supervised cross-modal retrieval

Abstract

Cross-modal retrieval aims to retrieve related items across different modalities, for example, using an image query to retrieve related text. The existing deep methods ignore both the intra-modal and inter-modal intra-class low-rank structures when fusing various modalities, which decreases the retrieval performance. In this paper, two deep models (denoted as ILCMR and Semi-ILCMR) based on intra-class low-rank regularization are proposed for supervised and semi-supervised cross-modal retrieval, respectively. Specifically, ILCMR integrates the image network and text network into a unified framework to learn a common feature space by imposing three regularization terms to fuse the cross-modal data. First, to align them in the label space, we utilize semantic consistency regularization to convert the data representations to probability distributions over the classes. Second, we introduce an intra-modal low-rank regularization, which encourages the intra-class samples that originate from the same space to be more relevant in the common feature space. Third, an inter-modal low-rank regularization is applied to reduce the cross-modal discrepancy. To enable the low-rank regularization to be optimized using automatic gradients during network back-propagation, we propose the rank-r approximation and specify the explicit gradients for theoretical completeness. In addition to the three regularization terms that rely on label information incorporated by ILCMR, we propose Semi-ILCMR in the semi-supervised regime, which introduces a low-rank constraint before projecting the general representations into the common feature space. Extensive experiments on four public cross-modal datasets demonstrate the superiority of ILCMR and Semi-ILCMR over other state-of-the-art methods.

Appendices

Appendix A:: Derivation of (13)

We acquired the approximation of rank calculation as shown in (7):

$$ rank(A) \approx \rho(A,r) = \frac{{\sum}_{i=r+1}^{s} \delta_{i}}{{\sum}_{i=1}^{s} \delta_{i}} = 1- \frac{{\sum}_{i=1}^{r} \delta_{i}}{{\sum}_{i=1}^{s} \delta_{i}}, $$

and combine it with the SVD knowledge as shown in (21):

$$ A = U \varSigma V^{T} = \delta_{1} u_{1} {v_{1}^{T}} + \delta_{2} u_{2} {v_{2}^{T}} + {\dots} + \delta_{s} u_{s} {v_{s}^{T}}. $$

(21)

We acquire the derivative in terms of A by the chain rule as shown in (13):

$$ \frac{\partial rank(A)}{\partial A} \approx \frac{\partial \rho(A,r)}{\partial A} = {\sum}_{i=1}^{s} \frac{\partial \rho}{\partial \delta_{i}} \cdot \frac{\partial \delta_{i}}{\partial A}. $$

A.1 $\frac {\partial \rho }{\partial \delta _{i}}$

For i ∈ [1,r], we have the following derivation from (7):

$$ \frac{\partial \rho}{\partial \delta_{i}} = \frac{0*{\sum}_{j=1}^{s} \delta_{j} - 1*{\sum}_{j=r+1}^{s} \delta_{j}}{{({\sum}_{j=1}^{s} \delta_{j})}^{2}} = - \frac{{\sum}_{j=r+1}^{s} \delta_{j}}{{({\sum}_{j=1}^{s} \delta_{j})}^{2}},\! $$

(22)

and for i ∈ [r + 1,s], we have the following derivation from (7):

$$ \frac{\partial \rho}{\partial \delta_{i}} = \frac{1*{\sum}_{j=1}^{s} \delta_{j} - 1*{\sum}_{j=r+1}^{s} \delta_{j}}{{({\sum}_{j=1}^{s} \delta_{j})}^{2}} = \frac{{\sum}_{j=1}^{r} \delta_{j}}{{({\sum}_{j=1}^{s} \delta_{j})}^{2}}. $$

(23)

Therefore, we can derive (14):

$$ \frac{\partial \rho}{\partial \delta_{i}}=\left\{ \begin{array}{rcl} - \frac{{\sum}_{j=r+1}^{s} \delta_{j}}{{\left( {\sum}_{j=1}^{s} \delta_{j} \right)}^{2}}, & & {i=1, \dots, r}\\ \\ \frac{{\sum}_{j=1}^{r} \delta_{j}}{{\left( {\sum}_{j=1}^{s} \delta_{j} \right)}^{2}}, & & {i=r+1, \dots, s} \end{array}. \right. $$

A.2 $\frac {\partial \delta _{i}}{\partial A}$

From (21), we have the following equation:

$$ \begin{array}{lll} &&\begin{pmatrix} a_{11}&a_{12}&\cdots&a_{1d}\\a_{21}&a_{22}&\cdots&a_{2d}\\\vdots&\vdots&\ddots&\vdots\\a_{n1}&a_{n2}&\cdots&a_{nd}\end{pmatrix} \\&=&\delta_{1} \begin{pmatrix} u_{11}v_{11} & u_{11}v_{12} & {\dots} & u_{11}v_{1d} \\ u_{12}v_{11} & u_{12}v_{12} & {\dots} & u_{12}v_{1d} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ u_{1n}v_{11} & u_{1n}v_{12} & {\dots} & u_{1n}v_{1d} \end{pmatrix} \\ &&+ \delta_{2} \begin{pmatrix} u_{21}v_{21} & u_{21}v_{22} & {\dots} & u_{21}v_{2d} \\ u_{22}v_{21} & u_{22}v_{22} & {\dots} & u_{22}v_{2d} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ u_{2n}v_{21} & u_{2n}v_{22} & {\dots} & u_{2n}v_{2d} \end{pmatrix} \\ &&+{\dots} + \delta_{s} \begin{pmatrix} u_{s1}v_{s1} & u_{s1}v_{s2} & {\dots} & u_{s1}v_{sd} \\ u_{s2}v_{s1} & u_{s2}v_{s2} & {\dots} & u_{s2}v_{sd} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ u_{sn}v_{s1} & u_{sn}v_{s2} & {\dots} & u_{sn}v_{1d} \end{pmatrix} \end{array}. $$

(24)

Equivalently, we have

$$ \left\{ \begin{array}{rcl} a_{11} &=& {\sum}_{i=1}^{s} \delta_{i} u_{i1} v_{i1}\\ \\ a_{12} &=& {\sum}_{i=1}^{s} \delta_{i} u_{i1} v_{i2}\\ &\vdots& \\ a_{nd} &=& {\sum}_{i=1}^{s} \delta_{i} u_{in} v_{id} \end{array} .\right. $$

(25)

Then, we can obtain the derivatives as follows:

$$ \left\{ \begin{array}{rcl} \frac{\partial a_{11}}{\partial \delta_{i}} &=& u_{i1} v_{i1} \rightarrow \frac{\partial \delta_{i}}{\partial a_{11}} = \frac{1}{u_{i1} v_{i1}}\\ \\ \frac{\partial a_{12}}{\partial \delta_{i}} &=& u_{i1} v_{i2} \rightarrow \frac{\partial \delta_{i}}{\partial a_{12}} = \frac{1}{u_{i1} v_{i2}}\\ &\vdots& \\ \frac{\partial a_{nd}}{\partial \delta_{i}} &=& u_{in} v_{id} \rightarrow \frac{\partial \delta_{i}}{\partial a_{nd}} = \frac{1}{u_{in} v_{id}} \end{array} .\right. $$

(26)

Therefore, the derivative in terms of A can be obtained as follows:

$$ \frac{\partial \delta_{i}}{\partial A} = 1./ \begin{pmatrix} u_{i1}v_{i1} & u_{i1}v_{i2} & {\dots} & u_{i1}v_{id} \\ u_{i2}v_{i1} & u_{i2}v_{i2} & {\dots} & u_{i2}v_{id} \\ {\vdots} & {\vdots} & {\ddots} & {\vdots} \\ u_{in}v_{i1} & u_{in}v_{i2} & {\dots} & u_{in}v_{id} \end{pmatrix} =1./u_{i} {v_{i}^{T}}, $$

(27)

where ‘./’ denotes the elementwise division operation.