Efficient cross-modal retrieval via flexible supervised collective matrix factorization hashing

Abstract

Cross-modal retrieval has recently drawn much attention in multimedia analysis, and it is still a challenging topic mainly attributes to its heterogeneous nature. In this paper, we propose a flexible supervised collective matrix factorization hashing (FS-CMFH) to efficient cross-modal retrieval. First, we exploit a flexible collective matrix factorization framework to jointly learn the individual latent space of similar semantic with respected to each modality. Meanwhile, the label consistency across different modalities is simultaneously exploited to preserve both intra-modal and inter-modal semantics within these similar latent semantic spaces. Accordingly, these two ingredients are formulated as a joint graph regularization term in an overall objective function, through which the similar hash codes of different modalities in an instance can be discriminatively obtained to flexibly characterize such instance. As a result, these derived hash codes incorporating higher discrimination power are able to improve the cross-modal searching accuracy significantly. The extensive experiments tested on three popular benchmark datasets show that the proposed approach performs favorably compared to the state-of-the-art competing approaches.

Appendix

In this section, we show the equivalent derivations for (12), (13) and (14), respectively. Let $\mathbf {X}=\{\mathbf {x}_{i}\}_{i = 1}^{n}\in \mathbb {R}^{d\times n}$, $\mathbf {Y}=\{\mathbf {y}_{i}\}_{i = 1}^{n}\in \mathbb {R}^{d\times n}$, and $\mathbf {w}(i,j)$ denotes the similarity measurement between $\mathbf {x}_{i}$ and x_j, we first construct a function $\boldsymbol {{\Phi }}$ of following form:

$$\begin{array}{@{}rcl@{}} \boldsymbol{{\Phi}}&{=}& \frac{1}{2}{\sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {{\mathbf{w}_{ij}}\|{\mathbf{x}_{i}}{-}{\mathbf{y}_{j}}\|}^{2} }}\\ &=& \frac{1}{2}\left( \sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {{\mathbf{w}_{ij}}\left( \mathbf{x}_{i}^{\mathrm{T}{\mathbf{x}_{i}}} - 2\mathbf{x}_{i}^{\mathrm{T}{\mathbf{y}_{j}}} + \mathbf{y}_{j}^{\mathrm{T}{\mathbf{y}_{j}}}\right)} } \right)\\ &=& \frac{1}{2}\left( \sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {{\mathbf{w}_{ij}}\mathbf{x}_{i}^{\mathrm{T}{\mathbf{x}_{i}}{-}}\sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {2{\mathbf{w}_{ij}}\mathbf{x}_{i}^{\mathrm{T}{\mathbf{y}_{j}}}} }{+}\sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {{\mathbf{w}_{ij}}\mathbf{y}_{j}^{\mathrm{T}{\mathbf{y}_{j}}}} } } } \right) \end{array} $$

(25)

Accordingly, we can also generalize function $\boldsymbol {{\Phi }}$ as:

$$\begin{array}{@{}rcl@{}} \boldsymbol{{\Phi}}&{=}& \left[ {\begin{array}{*{20}{c}} \mathbf{X}& \mathbf{Y} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} \mathbf{A} & \mathbf{B}\\ \mathbf{E} & \mathbf{F} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{\mathbf{X}^{\mathrm{T}}}}\\ {{\mathbf{Y}^{\mathrm{T}}}} \end{array}} \right] \\ &=& \left[ {\begin{array}{*{20}{c}} {\mathbf{XA}{+}\mathbf{YE}}& ~{\mathbf{XB}{+}\mathbf{YF}} \end{array}} \right]\left[ {\begin{array}{*{20}{c}} {{\mathbf{X}^{\mathrm{T}}}}\\ {{\mathbf{Y}^{\mathrm{T}}}} \end{array}} \right] \\&=& \mathbf{XAX}^{\mathrm{T}} + \mathbf{YEX}^{\mathrm{T}} + \mathbf{XBY}^{\mathrm{T}} + \mathbf{YFY}^{\mathrm{T}} \end{array} $$

(26)

Therefore, we can obtain $\boldsymbol {{\Phi }}=tr(\mathbf {XAX}^{\mathrm {T}} {+} \mathbf {YEX}^{\mathrm {T}} {+} \mathbf {XBY}^{\mathrm {T}} {+} \mathbf {YFY}^{\mathrm {T}})$. By comparing the (25) and (26), we can obtain that $\mathbf {A}_{ii}{=} \sum \nolimits _{j = 1}^{m} {{\mathbf {w}_{ij}} = {\mathbf {D}_{ii}}}$, $\mathbf {B} {=} - \mathbf {W}$, $\mathbf {F}_{jj}{=}\sum \nolimits _{i = 1}^{n} {{\mathbf {w}_{ij}} = {\mathbf {D}_{jj}}}$, $\mathbf {E} {=} - {\mathbf {W}^{\mathrm {T}}}$. Let $\mathbf {U} = [\mathbf {X}{\kern 1pt} ~~ {\kern 1pt} \mathbf {Y}]$, we can obtain:

$$\begin{array}{@{}rcl@{}} \frac{1}{2}{\sum\limits_{i = 1}^{n} {\sum\limits_{j = 1}^{m} {{\mathbf{w}_{ij}}||{\mathbf{x}_{i}} - {\mathbf{y}_{j}}||^{2}} }} = tr(\mathbf{ULU}^{\mathrm{T}}) \end{array} $$

(27)

where $\mathbf {L} = \left [ {\begin {array}{*{20}{c}} {{\mathbf {D}}}&{ - \mathbf {W}}\\ { - {\mathbf {W}^{\mathrm {T}}}}&{{\mathbf {D}}} \end {array}} \right ]$. According to this general formation, the three items in (11) can be directly converted as:

$$ \frac{\lambda_{1}}{2}\sum\limits_{i,j = 1}^{n} \mathbf{r}_{ij}^{(1)}||\mathbf{s}_{i}^{(1)}-\mathbf{s}_{j}^{(1)}||^{2} \Leftrightarrow tr(\mathbf{S}^{(1)}{(\mathbf{D}_{1}- \lambda_{1} \mathbf{R}_{1})}({\mathbf{S}^{(1)}})^{\mathrm{T}}) $$

(28)

$$ \frac{\lambda_{2}}{2}\sum\limits_{i,j = 1}^{n} \mathbf{r}_{ij}^{(2)}||\mathbf{s}_{i}^{(2)}-\mathbf{s}_{j}^{(2)}||^{2} \Leftrightarrow tr(\mathbf{S}^{(2)}{(\mathbf{D}_{2}- \lambda_{2} \mathbf{R}_{2})}({\mathbf{S}^{(2)}})^{\mathrm{T}}) $$

(29)

$$ \begin{array}{lllllllll} \sum\limits_{i,j = 1}^{n} \mathbf{c}_{ij}{||\mathbf{s}_{i}^{(1)}{-}\mathbf{s}_{j}^{(2)}||}^{2} &{\Leftrightarrow} tr\left( {\left[ {\begin{array}{*{20}{c}} {{\mathbf{S}^{(1)}}}&{{\mathbf{S}^{(2)}}} \end{array}} \right]\left( {\begin{array}{*{20}{c}} \mathbf{D}_{3}&{ - \mathbf{C}}\\ {-\mathbf{C}^{\mathrm{T}}}&\mathbf{D}_{3} \end{array}} \right)\left[ \begin{array}{l} ({\mathbf{S}^{{{(1)}}}})^{\mathrm{T}}\\ ({\mathbf{S}^{{{(2)}}}})^{\mathrm{T}} \end{array} \right]} \right)\\ &= tr\left( {\mathbf{S}^{(1)}}{\mathbf{D}_{3}}({\mathbf{S}^{(1)}})^{\mathrm{T}{+}{\mathbf{S}^{(2)}}{\mathbf{D}_{3}}}({\mathbf{S}^{(2)}})^{\mathrm{T}{-}{2\mathbf{S}^{(1)}}}\mathbf{C}({\mathbf{S}^{(2)}})^{\mathrm{T}}\right) \end{array} $$

(30)

where $\mathbf {D}_{1}$, $\mathbf {D}_{2}$, $\mathbf {D}_{3}\in {\mathbb {R}}^{n\times n}$ are diagonal matrices with entries being the column sum of $\lambda _{1} \mathbf {R}^{(1)}$, $\lambda _{2} \mathbf {R}^{(2)}$ and $\mathbf {C}$, respectively.