End-to-end multitask Siamese network with residual hierarchical attention for real-time object tracking

Abstract

Object tracking with deep networks has recently achieved substantial improvement in terms of tracking performance. In this paper, we propose a multitask Siamese neural network that uses a residual hierarchical attention mechanism to achieve high-performance object tracking. This network is trained offline in an end-to-end manner, and it is capable of real-time tracking. To produce more efficient and generative attention-aware features, we propose residual hierarchical attention learning using residual skip connections in the attention module to receive hierarchical attention. Moreover, we formulate a multitask correlation filter layer to exploit the missing link between context awareness and regression target adaptation, and we insert this differentiable layer into a neural network to improve the discriminatory capability of the network. The results of experimental analyses conducted on the OTB, VOT and TColor-128 datasets, which contain various tracking scenarios, demonstrate the efficiency of our proposed real-time object-tracking network.

Appendix

Based on the principle of convex functions and the circulant matrix, the closed-form solution of the filter parameter w in the proposed objective function can be deduced. The appendix shows the detailed solution process.

The objective function in (5) can be rewritten as follows using a circulant matrix, $\mathbf {B}= \left [\begin {array}{cccccc} \mathbf {X}_{0}& \sqrt {\theta _{3}}\mathbf {X}_{1}&\sqrt {\theta _{3}}\mathbf {X}_{2}&\ldots &\sqrt {\theta _{3}}\mathbf {X}_{k} \end {array}\right ]^{T}$:

$$ \begin{array}{@{}rcl@{}} \arg\underset{\mathbf{w},\mathbf{y}}{\min} \mathbb{O} &=& ||\begin{array}{cccccc} \left[\begin{array}{cccc} \mathbf{X}_{0}\\ \sqrt{\theta_{3}}\mathbf{X}_{1}\\\vdots\\\sqrt{\theta_{3}}\mathbf{X}_{k} \end{array}\right] \mathbf{w}- \left[\begin{array}{cccc} \mathbf{y}\\ 0\\\vdots\\0 \end{array}\right] \end{array}|| ^{2}_{2}+\theta_{1}||\mathbf{w}||^{2}_{2}\\ &&+\theta_{2} ||\begin{array}{cccccc} \left[\begin{array}{cccc} \mathbf{y}\\ 0\\\vdots\\0 \end{array}\right] -\left[\begin{array}{cccc} \mathbf{y}_{0}\\ 0\\\vdots\\0 \end{array}\right] \end{array}|| ^{2}_{2}\\ &=&||\mathbf{B}\mathbf{w}-\mathbf{y}^{\prime}||^{2}_{2}+\theta_{1}||\mathbf{w}||^{2}_{2}+\theta_{2}||\mathbf{y}^{\prime}-\mathbf{y}^{\prime}_{0}||^{2}_{2}. \end{array} $$

(10)

We define that $\mathbf {z}=\left [\begin {array}{cccc}\mathbf {w}\\\mathbf {y}^{\prime } \end {array}\right ]$; then, we attempt to search for a z to optimize (10):

$$ \begin{array}{@{}rcl@{}} \arg\underset{\mathbf{z}}{\min} \mathbb{O}(\mathbf{z})&=& ||\left[\begin{array}{cccc}\mathbf{B} &-\mathbf{I} \end{array}\right]\mathbf{z}||^{2}_{2}+\theta_{1}||\left[\begin{array}{cccc}\mathbf{I} &0 \end{array}\right]\mathbf{z}||^{2}_{2}\\ &&+\theta_{2}||\left[\begin{array}{cccc}0&\mathbf{I} \end{array}\right]\mathbf{z}-\mathbf{y}^{\prime}_{0}||^{2}_{2}.\ \end{array} $$

(11)

Because our proposed objective function is convex, (11) can be optimized using its first derivative:

$$ \begin{array}{@{}rcl@{}} \nabla_{\mathbf{z}}\mathbb{O}(\mathbf{z})&=& \left[\begin{array}{cccc}\mathbf{B}^{T}\mathbf{B} & -\mathbf{B}^{T}\\-\mathbf{B}&\mathbf{I} \end{array}\right]\mathbf{z}+\theta_{1}\left[\begin{array}{cccc}\mathbf{I}&0\\0&0 \end{array}\right]\mathbf{z}\\ &&+\theta_{2}\left[\begin{array}{cccc}0&0\\0&\mathbf{I} \end{array}\right]\mathbf{z} -\theta_{2}\left[\begin{array}{cccc}0\\\mathbf{I} \end{array}\right]\mathbf{y}^{\prime}_{0}=0,\\ &\Rightarrow&\left[\begin{array}{cccc}\mathbf{B}^{T}\mathbf{B}+\theta_{1}\mathbf{I} & -\mathbf{B}^{T}\\-\mathbf{B}&(1+\theta_{2})\mathbf{I} \end{array}\right]\mathbf{z}=\theta_{2}\left[\begin{array}{cccc}0\\\mathbf{I} \end{array}\right]\mathbf{y}^{\prime}_{0}.\ \end{array} $$

(12)

B is a circulant matrix. Thus, we can use its characteristics as follows:

$$ \begin{array}{lllll} &\mathbf{B}=\mathbf{F}\left[\begin{array}{cccc}diag(\hat{\mathbf{x}}_{0})&\sqrt{\theta_{3}}diag(\hat{\mathbf{x}}_{1})&\ldots&\sqrt{\theta_{3}}diag(\hat{\mathbf{x}}_{k}) \end{array}\right]^{T}\mathbf{F}^{H}=-\mathbf{F}\mathbf{D}\mathbf{F}^{H},\\ & \mathbf{B}^{T}=\mathbf{F}\left[\begin{array}{cccc}diag(\hat{\mathbf{x}}^{*}_{0})&\sqrt{\theta_{3}}diag(\hat{\mathbf{x}}^{*}_{1})&\ldots&\sqrt{\theta_{3}}diag(\hat{\mathbf{x}}^{*}_{k}) \end{array}\right]\mathbf{F}^{H} =-\mathbf{F}\mathbf{C}\mathbf{F}^{H}. \end{array} $$

(13)

where F is the DFT matrix.

By substituting (13) into the result of (12), the equation can be transformed into (14) and (15):

$$ \begin{array}{@{}rcl@{}} &&\left[\begin{array}{cccc} \mathbf{F}&0\\0&\mathbf{F} \end{array}\right] \left[\begin{array}{cccc} diag(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{x}}_{0}+\theta_{3}{\sum}^{k}_{i=1}\hat{\mathbf{x}}^{*}_{i}\odot\hat{\mathbf{x}}_{i}+\theta_{1}) & \mathbf{C}\\\mathbf{D}&diag(1+\theta_{2}) \end{array}\right]\\ &&\times \left[\begin{array}{cccc} \mathbf{F}^{H}&0\\0&\mathbf{F}^{H} \end{array}\right] \mathbf{z}=\theta_{2} \left[\begin{array}{cccc}0\\\mathbf{I} \end{array}\right]\mathbf{y}^{\prime}_{0},\ \end{array} $$

(14)

$$ \left[\begin{array}{cccc} \mathbf{N} & \mathbf{C}\\\mathbf{D}&\mathbf{V} \end{array}\right] \left[\begin{array}{cccc} \hat{\mathbf{w}}^{*}\\\hat{\mathbf{y}^{\prime}}^{*} \end{array}\right] =\theta_{2} \left[\begin{array}{cccc}0\\\mathbf{F}^{H} \end{array}\right]\mathbf{y}^{\prime}_{0},\ $$

(15)

where $\mathbf {N}=diag(\hat {\mathbf {x}}^{*}_{0}\odot \hat {\mathbf {x}}_{0}+\theta _{3}{\sum }^{k}_{i=1}\hat {\mathbf {x}}^{*}_{i}\odot \hat {\mathbf {x}}_{i}+\theta _{1})$ and V = diag(1 + 𝜃₂).

Thus, $\left [\begin {array}{cccc} \hat {\mathbf {w}}^{*}\\\hat {\mathbf {y}^{\prime }}^{*} \end {array}\right ]$ canbe rewritten as follows:

$$ \left[\begin{array}{cccc} \hat{\mathbf{w}}^{*}\\\hat{\mathbf{y}^{\prime}}^{*} \end{array}\right] =\theta_{2} \left[\begin{array}{cccc} \mathbf{N} & \mathbf{C}\\\mathbf{D}&\mathbf{V} \end{array}\right]^{-1} \left[\begin{array}{cccc}0\\\mathbf{F}^{H} \end{array}\right]\mathbf{y}^{\prime}_{0}.\ $$

(16)

In (16), $\left [\begin {array}{cccc} \mathbf {N} & \mathbf {C}\\\mathbf {D}&\mathbf {V} \end {array}\right ]^{-1}$ is as follows:

$$ \begin{array}{lllll} &\left[\begin{array}{cccc} \mathbf{N} & \mathbf{C}\\\mathbf{D}&\mathbf{V} \end{array}\right]^{-1}\\ &= \left[\begin{array}{cccc} (\mathbf{N}- \mathbf{C}\mathbf{V}^{-1}\mathbf{D})^{-1}& -(\mathbf{N}- \mathbf{C}\mathbf{V}^{-1}\mathbf{D})^{-1}\mathbf{C}\mathbf{V}^{-1} \\-\mathbf{V}^{-1}\mathbf{D}(\mathbf{N}- \mathbf{C}\mathbf{V}^{-1}\mathbf{D})^{-1} & \mathbf{V}^{-1}\mathbf{D}(\mathbf{N}- \mathbf{C}\mathbf{V}^{-1}\mathbf{D})^{-1}\mathbf{C}\mathbf{V}^{-1}+\mathbf{V}^{-1} \end{array}\right] .\ \end{array} $$

(17)

Then, the solution of $\hat {\mathbf {w}}^{*}$ according to (16) and (17):

$$ \hat{\mathbf{w}}^{*}=-\theta_{2}(\mathbf{N}- \mathbf{C}\mathbf{V}^{-1}\mathbf{D})^{-1}\mathbf{C}\mathbf{V}^{-1}\mathbf{F}^{H}\mathbf{y}^{\prime}_{0}, $$

(18)

where (N −CV^− 1D)^− 1 is:

$$ \begin{array}{@{}rcl@{}} (\mathbf{N}- \mathbf{C}\mathbf{V}^{-1}\mathbf{D})^{-1}&=&[diag(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{x}}_{0}+\theta_{3}{\sum}^{k}_{i=1}\hat{\mathbf{x}}^{*}_{i}\odot\hat{\mathbf{x}}_{i}+\theta_{1})\\ &&-\frac{diag(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{x}}_{0}+\theta_{3}{\sum}^{k}_{i=1}\hat{\mathbf{x}}^{*}_{i}\odot\hat{\mathbf{x}}_{i})}{diag(1+\theta_{2})}]^{-1}\\ &=&diag(\frac{1 + \theta_{2}}{\theta_{2}(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{x}}_{0} + \theta_{3}{\sum}^{k}_{i=1}\hat{\mathbf{x}}^{*}_{i}\odot\hat{\mathbf{x}}_{i})+\theta_{1}(1 + \theta_{2})}).\ \end{array} $$

(19)

Finally, we can obtain the solution of filter parameter w:

$$ \begin{array}{llllllll} &\hat{\mathbf{w}}^{*}=\frac{\theta_{2}(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{y}}^{*}_{0})}{\theta_{2}(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{x}}_{0}+\theta_{3}{\sum}^{k}_{i=1}\hat{\mathbf{x}}^{*}_{i}\odot\hat{\mathbf{x}}_{i})+\theta_{1}(1+\theta_{2})}\\ &\Rightarrow\hat{\mathbf{w}}=\frac{\theta_{2}(\hat{\mathbf{x}}_{0}\odot\hat{\mathbf{y}}_{0})}{\theta_{2}(\hat{\mathbf{x}}^{*}_{0}\odot\hat{\mathbf{x}}_{0}+\theta_{3}{\sum}^{k}_{i=1}\hat{\mathbf{x}}^{*}_{i}\odot\hat{\mathbf{x}}_{i})+\theta_{1}(1+\theta_{2})}.\ \end{array} $$

(20)