A Context Based Deep Temporal Embedding Network in Action Recognition

Abstract

Long term temporal representation methods demand high computational cost, restricting their practical use in real world applications. We propose a two-step deep residual method for efficiently learning long-term discriminative temporal representation, whilst significantly reducing computational cost. In the first step, a novel self-supervision deep temporal embedding method is presented to embed repetitive short-term motions at a cluster-friendly feature space. In the second step, an efficient temporal representation is made by leveraging the differences between the original data and its associated repetitive motion clusters as a novel deep residual method. Experimental results demonstrate that, the proposed method achieves competitive results on some challenging human action recognition datasets like UCF101, HMDB51, THUMOS14, and Kinetics-400.

Appendices

Appendix 1

In this section, we describe some basic concepts including LSTM network, encoder–decoder LSTM network, and Deep Embedding Clustering network. Afterward, the proposed model would be introduced.

LSTM Network

RNN models temporal dynamics of the input sequence to the sequence of hidden states, and then hidden states to the outputs. This capability is as a result of feedback connections and internal memory. RNNs perform poorly in learning long-term temporal dynamics. They are not appropriate to deal with the problem of vanishing gradient. LSTM overcomes this problem by considering recurrent forget gates in its architecture. Forget gates are memory units that allow LSTM to learn from which time step of previously hidden states must be considered in the network updating. So, LSTMs have proven successful in very deep learning models while they can perfectly remember some important events in long time intervals. Recent works have proposed many modified LSTM models such as Peephole LSTM, Peephole convolutional LSTM, and the Gated Recurrent Unit (GRU) [87]. In this paper, we use the standard LSTM structure in our model. The functionality of an LSTM unit is described by the following recurrence equations:

$$ \begin{aligned} i_{l} & = \sigma \left( {W_{xi} x_{l} + W_{hi} h_{l - 1} } \right) \\ f_{l} & = \sigma \left( {W_{xf} x_{l} + W_{hf} h_{l - 1} } \right) \\ o_{l} & = \sigma \left( {W_{xo} x_{l} + W_{ho} h_{l - 1} } \right) \\ c_{l} & = f_{l} \odot c_{l - 1} + i_{l} \odot\Phi \left( {W_{xc} x_{l} + W_{hc} h_{l - 1} } \right) \\ h_{l} & = o_{l} \odot\Phi \left( {c_{l} } \right) \\ \end{aligned} $$

(14)

where $\sigma$ is the sigmoidal non-linearity, $\Phi$ is hyperbolic tangent non-linearity, $\odot$ represents the product with the gate value, $x_{l}$ is the input, $h_{l}$ is the hidden state, $o_{l}$ is the output at time step l and the weight matrices denoted by $W_{ij}$ are the trained parameters. These hidden states constitute a representation of an input sequence learned over time. Conventional LSTM, however, fails to take into consideration the impact of salient temporal dynamics present in the sequential input data [88]. The attention mechanism helps the model to select informative words.

2.1 Encoder–Decoder LSTM Network

The encoder–decoder LSTM was initially introduced for natural language processing problems where it demonstrated state-of-the-art performance [89]. In this regard, the unsupervised representation learned by encoder–decoder LSTM naturally facilitates the learning of temporal representations with our proposed method. An encoder–decoder LSTM is a two-layer RNN that acts as an encoder and a decoder pair. The encoder maps a variable-length source sequence to a fixed-length vector, and the decoder maps the vector representation back to a variable-length target sequence [87]. encoder–decoder LSTM is designed specifically for seq 2seq problems defined as:

$$ p\left( {y_{1} , \ldots y_{T} |x_{1} , \ldots x_{L} } \right) = \pi_{t = 1}^{T} p\left( {y_{t} |c, y_{1} , \ldots y_{t - 1} } \right) $$

(15)

where $y_{1} , \ldots y_{T}$ is output sequence, $x_{1} , \ldots x_{L}$ is input sequence, c is context vector (hidden vector of last LSTM unit) that summarizes the input sequence. The goal of encoder–decoder LSTM is to build a neural network to model and to maximize this conditional probability. The encoder part maps the input sequence to context vector as:

$$ h_{l} = g_{1} \left( {h_{l - 1} , x_{l} } \right) , c = h_{L} $$

(16)

The decoder part predicts and constructs the next data of the input sequence as output sequence:

$$ \begin{aligned} & s_{t} = g_{2} \left( {s_{t - 1} ,\left[ {y_{t - 1} ,c} \right] } \right) \\ & p\left( {y_{t} |c, y_{1} , \ldots y_{t - 1} } \right) = \sigma \left( {Ws_{T} + b} \right) \\ \end{aligned} $$

(17)

After greedy layer-wise training, all encoder layers would be concatenated followed by all decoder layers, to form a deep network, and then fine-tune it to minimize the prediction loss.

Deep Embedding Clustering Network

Deep embedding clustering [10] jointly optimizes feature transformation and clustering. This algorithm leverages a pre-training auto encoder as an initial estimation of data representation and then removes the decoder. Then, the remaining encoder is fine-tuned by an effective clustering loss in a self-learning manner using high confidential samples. It updates parameters of transformation network and cluster centers, simultaneously. Let $z_{i} = f_{\theta } \left( {x_{i} } \right)$ be the mapping function of the pre-trained encoder, where $x_{i}$ is an input data. Using $f_{\theta }$ we obtain all embedded points $\left\{ {z_{i} } \right\}$. Then, k-means is employed on $\left\{ {z_{i} } \right\}$ to get initial cluster centers $\left\{ {\mu_{i} } \right\}$. Afterward, the objective function is defined as follow:

$$ {\text{L}} = KL(P| |Q )= \mathop \sum \limits_{i} \mathop \sum \limits_{j} p_{ij} log\frac{{p_{ij} }}{{q_{ij} }} $$

(18)

$q_{ij}$ denotes the distance between embedded point $z_{i}$ and cluster center $\mu_{i}$ defined by Student’s t-distribution:

$$ q_{ij} = \frac{{\left( {1 + \left| {z_{i} - \mu_{j} } \right|^{2} /\alpha } \right)^{{ - \frac{\alpha + 1}{2}}} }}{{\mathop \sum \nolimits_{j} \left( {1 + \left| {z_{i} - \mu_{j} } \right|^{2} /\alpha } \right)^{{ - \frac{\alpha + 1}{2}}} }} $$

(19)

$p_{ij}$ is the target distribution that considers high confidential samples as supervision that makes clusters more densely, defined as:

$$ p_{ij} = \frac{{{{q_{ij}^{2} } \mathord{\left/ {\vphantom {{q_{ij}^{2} } {\sum\nolimits_{i} {q_{ij} } }}} \right. \kern-0pt} {\sum\nolimits_{i} {q_{ij} } }}}}{{{{\sum\nolimits_{\text{j}} {q_{{i{\text{j}}}}^{2} } } \mathord{\left/ {\vphantom {{\sum\nolimits_{\text{j}} {q_{{i{\text{j}}}}^{2} } } {\sum\nolimits_{i} {q_{ij} } }}} \right. \kern-0pt} {\sum\nolimits_{i} {q_{ij} } }}}} $$

(20)

As $p_{ij}$ is defined by $q_{ij}$, minimizing the loss function (L) becomes a form of self-training. The cluster assignment of sample $x_{i}$ is $argmax_{j } q_{ij}$.