Image auto-annotation via tag-dependent random search over range-constrained visual neighbours

Abstract

The quantity setting of visual neighbours can be critical for the performance of many previously proposed visual-neighbour-based (VNB) image auto-annotation methods. And in those methods, each candidate tag of a to-be-annotated image would be better to have its own trustworthy part of visual neighbours for score prediction. Hence in this paper we propose to use a constrained range rather than an identical and fixed number of visual neighbours for VNB methods to allow more flexible choices of neighbours, and then put forward a novel tag-dependent random search process to estimate the tag-dependent trust degrees of visual neighbours for each candidate tag. We further propose an effective image auto-annotation method termed TagSearcher based on a widely-used conditional probability model for auto-annotation, considering image-dependent weights of visual neighbours, tag-dependent trust degrees of visual neighbours and votes for a candidate tag from visual neighbours. Extensive experiments conducted on both a benchmark dataset and real-world web images present that the proposed TagSearcher can yield inspiring annotation performance and also reduce the performance sensitivity to the quantity setting of visual neighbours.

Appendix

1.1 A1. Convergence proof

Note that in formula (15), each column of the matrix $\mathbf {S}^{\left (n-h\right )^{T}}$ denoting the successive relations between vertices is L1 normalized to sum up to 1, and the entries of any $\mathbf {S}^{\left (n-h\right )^{T}}$ or F ^{n − h} are all between 0 and 1. For any δ lying in (0, 1), there always exists γ < 1 subject to 1 − δ < γ, and thus we can derive that:

$$\begin{array}{l} \sum\limits_{j}\left(\prod\limits_{h=1}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ji} \\ = \sum\limits_{j}\sum\limits_{k}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-1\right)^{T}}\mathbf{F}^{\left(n-1\right)}\right)_{jk}\left(\prod\limits_{h=2}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ki} \\ = \sum\limits_{k}\left(\prod\limits_{h=2}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ki}\left(1-\delta\right)\sum\limits_{j}\left(\mathbf{S}^{\left(n-1\right)^{T}}\mathbf{F}^{\left(n-1\right)}\right)_{jk} \\ \leqslant \left(1-\delta\right)\sum\limits_{k}\left(\prod\limits_{h=2}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ki}\sum\limits_{j}\left(\mathbf{S}^{\left(n-1\right)^{T}}\right)_{jk} \\ = \left(1-\delta\right)\sum\limits_{k}\left(\prod\limits_{h=2}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ki} \\ \leqslant \gamma\sum\limits_{k}\left(\prod\limits_{h=2}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ki} \\ \leqslant \gamma\left(\gamma\sum\limits_{k}\left(\prod\limits_{h=3}^{n-1}\left(\left(1-\delta\right)\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)_{ki}\right) \\ \leqslant \ldots \\ \leqslant \gamma^{n-1} \end{array} $$

Therefore, the sum of each column of $\prod _{h=1}^{n-1}\left (\left (1-\delta \right )\mathbf {S}^{\left (n-h\right )^{T}}\mathbf {F}^{\left (n-h\right )}\right )$ will tend to be zero as n goes to positive infinity. Since (1 − δ), $\mathbf {S}^{\left (n-h\right )^{T}}$ and F ^{n − h} are all non-negative, $\prod _{h=1}^{n-1}\left (\left (1-\delta \right )\mathbf {S}^{\left (n-h\right )^{T}}\mathbf {F}^{\left (n-h\right )}\right )$ will also be non-negative and thus $\left (\prod _{h=1}^{n-1}\left (\left (1-\delta \right ) \mathbf {S}^{\left (n-h\right )^{T}}\mathbf {F}^{\left (n-h\right )}\right )\right )\mathbf {p}^{\left (1\right )}$ will converge to a zero vector. Then formula (15) can be simplified as follows.

$$\mathbf{p}_{\pi} = \lim\limits_{n\rightarrow\infty}\delta\left(1+\sum\limits_{k=1}^{n-1}\left(1-\delta\right)^{k}\prod\limits_{h=1}^{k}\left(\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)\mathbf{p}^{\left(0\right)} $$

Since we only focus on the ratios between entries of p _π rather than their values, the formula above can be further simplified as follows.

$$\mathbf{p}_{\pi} \sim \lim\limits_{n\rightarrow\infty}\left(1+\sum\limits_{k=1}^{n-1}\left(1-\delta\right)^{k}\prod\limits_{h=1}^{k}\left(\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)\mathbf{p}^{\left(0\right)} $$

And it can be seen that entries of p _π keep increasing as n increases. Since entries of any $\mathbf {S}^{\left (n-h\right )^{T}}$, F ^{n − h}and p ⁽⁰⁾ are all non-negative and lie in [0, 1], we can further derive that:

$$\begin{array}{l} \lim\limits_{n\rightarrow\infty}\left(1+\sum\limits_{k=1}^{n-1}\left(1-\delta\right)^{k}\prod\limits_{h=1}^{k}\left(\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\right)\mathbf{p}^{\left(0\right)} \\ = \lim\limits_{n\rightarrow\infty}\mathbf{p}^{\left(0\right)}+\sum\limits_{k=1}^{n-1}\left(1-\delta\right)^{k}\prod\limits_{h=1}^{k}\left(\mathbf{S}^{\left(n-h\right)^{T}}\mathbf{F}^{\left(n-h\right)}\right)\mathbf{p}^{\left(0\right)} \\ \preceq \lim\limits_{n\rightarrow\infty}\left(\mathbf{p}^{\left(0\right)}+\sum\limits_{k=1}^{n-1}\left(1-\delta\right)^{k}\mathbb{I}\right) \end{array} $$

where 𝕀 is a column vector with entries all being 1, and “ ≼ ” means entry-wise “ ≤ ”. Since (1 − δ) lies in (0, 1), $\sum _{k=1}^{n-1}\left (1-\delta \right )^{k}$ will converge as n goes to positive infinity. Then p _π is a monotonically increasing function w.r.t the step n and has an upper bound. Therefore, it will converge as n goes to positive infinity, meaning that convergence of the proposed tag-dependent random search process is guaranteed.