Learning multi-task local metrics for image annotation

Abstract

The goal of image annotation is to automatically assign a set of textual labels to an image to describe the visual contents thereof. Recently, with the rapid increase in the number of web images, nearest neighbor (NN) based methods have become more attractive and have shown exciting results for image annotation. One of the key challenges of these methods is to define an appropriate similarity measure between images for neighbor selection. Several distance metric learning (DML) algorithms derived from traditional image classification problems have been applied to annotation tasks. However, a fundamental limitation of applying DML to image annotation is that it learns a single global distance metric over the entire image collection and measures the distance between image pairs in the image-level. For multi-label annotation problems, it may be more reasonable to measure similarity of image pairs in the label-level. In this paper, we develop a novel label prediction scheme utilizing multiple label-specific local metrics for label-level similarity measure, and propose two different local metric learning methods in a multi-task learning (MTL) framework. Extensive experimental results on two challenging annotation datasets demonstrate that 1) utilizing multiple local distance metrics to learn label-level distances is superior to using a single global metric in label prediction, and 2) the proposed methods using the MTL framework to learn multiple local metrics simultaneously can model the commonalities of labels, thereby facilitating label prediction results to achieve state-of-the-art annotation performance.

Appendices

Appendix A: Optimization in MTLM-LDML

For the logistic regression model in (7), the bias parameter b _m can be considered to be an additional entry in the weight vector w _m , as b _m = w ₀ and d ⁰(⋅) = −1. Thus, the probability of a similar pair in (7) can be concisely expressed as

$$ p_{qp}(s_{qp} = 1|y_{m}; D_{y_{m}}(I_{q}, I_{p}), b) = \sigma(-D_{y_{m}}(I_{q}, I_{p})). $$

(13)

Intuitively, to simplify the denotation, the cost function in (9) can be rewritten as

$$ E(\boldsymbol{w}) = {\gamma_{\ast} \| \boldsymbol{w}_{\ast} \|_{2}^{2}} + \sum\limits_{m=1}^{M}\left[\gamma_{m}{\| \boldsymbol{w}_{m} \|_{2}^{2}} + {\underset{(I_{q}, I_{p}) \in \mathcal{P}_{y_{m}}}\sum}{\log(1 + \exp(s_{qp} D_{y_{m}}(I_{q}, I_{p})))} \right]. $$

(14)

Based on the definitions of $D_{y_{m}}$ and the sigmoid function $\sigma (z) = \frac {1}{1 + \exp (-z)}$, we first consider the gradients with respect to weight vectors $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ of the label-specific metrics as

$$ E(\boldsymbol{w}_{m}) = \gamma_{m}{\| \boldsymbol{w}_{m} \|_{2}^{2}} + {\underset{(I_{q}, I_{p}) \in \mathcal{P}_{y_{m}}}\sum}{\log(1 + \exp(s_{qp} D_{y_{m}}(I_{q}, I_{p})))}. $$

(15)

Hence, the gradient of w _m is given by

$$ \boldsymbol{G}(\boldsymbol{w}_{m}) = \frac{\partial{E(\boldsymbol{w})}}{\partial{\boldsymbol{w}_{m}}} = 2\gamma_{m} \boldsymbol{w}_{m} + {\underset{(I_{q}, I_{p}) \in \mathcal{P}_{y_{m}}}\sum}{s_{qp} \sigma(s_{qp}D_{y_{m}}(I_{q}, I_{p}))\boldsymbol{d}(I_{q}, I_{p})}. $$

(16)

Then, we consider gradients with respect to weight vector w _∗ of the shared metric as

$$ \boldsymbol{G}(\boldsymbol{w}_{\ast}) = \frac{\partial{E(\boldsymbol{w})}}{\partial{\boldsymbol{w}_{\ast}}} = 2\gamma_{\ast} \boldsymbol{w}_{\ast} + \sum\limits_{m=1}^{M} ~~~~~~{\underset{(I_{q}, I_{p}) \in \mathcal{P}_{y_{m}}}\sum}{s_{qp} \sigma(s_{qp}D_{y_{m}}(I_{q}, I_{p}))\boldsymbol{d}(I_{q}, I_{p})}. $$

(17)

Note that the gradients of $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ in (16) can be calculated separately according to the pairwise constraints of labels $\{{y_{m}}\}_{m=1}^{M}$, while the gradient of w _∗ in (17) depends on all gradients of $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ and must be calculated thereafter.

Finally, the iterative update solutions for weight vectors $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ and w _∗ can be formulated as

$$ \boldsymbol{w}_{m}^{t+1} = [\boldsymbol{w}_{m}^{t} - \eta_{t} \boldsymbol{G}(\boldsymbol{w}_{m}^{t}) ]_{+}, $$

(18)

$$ \boldsymbol{w}_{\ast}^{t+1} = [\boldsymbol{w}_{\ast}^{t} - \eta_{t} \boldsymbol{G}(\boldsymbol{w}_{\ast}^{t}) ]_{+}, $$

(19)

where η _t is the step size of iteration t, and $[f]_{+} = \max (0, f)$ truncates any negative entries in w and sets them to zero for the non-negative constraints of w. Algorithm 1 summarizes the learning process in the proposed MTLM-LDML method.

Appendix B: Optimization in MTLM-LMNN

Similar to the computation of gradients in Appendix A, from the cost function in (12) we can derive the gradients of weight vectors $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ and w _∗ as

$$ \boldsymbol{G}(\boldsymbol{w}_{m}) = 2\gamma_{m} \boldsymbol{w}_{m} + {\underset{(I_{q}, I_{n}) \in \mathcal{P}_{y_{m}}^{+}}\sum}{\boldsymbol{d}(I_{q}, I_{p})} + \mu {\underset{(I_{q}, I_{n}) \in \mathcal{P}_{y_{m}}^{-}}\sum}\left(\boldsymbol{d}(I_{q}, I_{p}) - \boldsymbol{d}(I_{q}, I_{n}) \right), $$

(20)

$$ \boldsymbol{G}(\boldsymbol{w}_{\ast}) = 2\gamma_{\ast} \boldsymbol{w}_{\ast} + \sum\limits_{m=1}^{M} \left[~~~~~ {\underset{(I_{q}, I_{n}) \in \mathcal{P}_{y_{m}}^{+}}\sum}{\boldsymbol{d}(I_{q}, I_{p})} + \mu {\underset{(I_{q}, I_{n}) \in \mathcal{P}_{y_{m}}^{-}}\sum}\left(\boldsymbol{d}(I_{q}, I_{p}) - \boldsymbol{d}(I_{q}, I_{n}) \right) \right]. $$

(21)

Here we make the similar observations that the gradient of $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ in (20) can be calculated separately according to the pairwise constraints $\mathcal {P}_{y_{m}}$ of each label y _m , while the gradient of w _∗ in (21) depends on all the gradients of $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$.

Since naive computation of the gradients in (20) and (21) would be extremely expensive, we consider only the “active” triplets in $\mathcal {P}_{y_{m}}$ that change from iteration t to iteration t + 1 based on the following update rules:

$$\begin{array}{@{}rcl@{}} \boldsymbol{G}(\boldsymbol{w}_{m}^{t+1})&=&\boldsymbol{G}(\boldsymbol{w}_{m}^{t}) - \eta_{t} {\underset{(I_{q}, I_{p}, I_{n}) \in \mathcal{P}_{y_{m}}^{t} - P_{y_{m}}^{t+1}}\sum}(\boldsymbol{d}(I_{q}, I_{p}) - \boldsymbol{d}(I_{q}, I_{n})) + \eta_{t}\\&&{\kern4pc} {\underset{(I_{q}, I_{p}, I_{n}) \in \mathcal{P}_{y_{m}}^{t+1} - P_{y_{m}}^{t}}\sum}(\boldsymbol{d}(I_{q}, I_{p}) - \boldsymbol{d}(I_{q}, I_{n})), \end{array} $$

(22)

$$\begin{array}{@{}rcl@{}} \boldsymbol{G}(\boldsymbol{w}_{\ast}^{t+1}) = \boldsymbol{G}(\boldsymbol{w}_{\ast}^{t}) - \sum\limits_{m=1}^{M} \left[ \eta_{t} {\underset{(I_{q}, I_{p}, I_{n}) \in \mathcal{P}_{y_{m}}^{t} - \mathcal{P}_{y_{m}}^{t+1}}\sum}(\boldsymbol{d}(I_{q}, I_{p}) - \boldsymbol{d}(I_{q}, I_{n})) + \eta_{t}\right.\\ \left.{\underset{(I_{q}, I_{p}, I_{n}) \in \mathcal{P}_{y_{m}}^{t+1} - \mathcal{P}_{y_{m}}^{t}}\sum}(\boldsymbol{d}(I_{q}, I_{p}) - \boldsymbol{d}(I_{q}, I_{n})) \right], \end{array} $$

(23)

where μ _t is the step size in the t-th iteration, $\mathcal {P}_{y_{m}}^{t+1} - \mathcal {P}_{y_{m}}^{t}$ represents the new triplets appearing in $\mathcal {P}_{y_{m}}^{t+1}$, and $\mathcal {P}_{y_{m}}^{t} - \mathcal {P}_{y_{m}}^{t+1}$ represents the old triplets that have disappeared in $\mathcal {P}_{y_{m}}^{t+1}$. For a small step size, the set $\mathcal {P}_{y_{m}}^{t}$ changes little in each iteration. In this case, computing the gradients in (22) and (23) is much faster.

Finally, we utilize a gradient descent based learning process similar to that depicted in Algorithm 1 to learn weight vectors $\{\boldsymbol {w}_{m}\}_{m=1}^{M}$ and w _∗ for the proposed MTLM-LMNN model.