Nonlinear gated channels networks for action recognition

Abstract

Video-based convolutional neural networks (CNNs) involve a large amount of training parameters, leading to the enormous computational complexity, which thereby delays the network convergence. Therefore, training successful CNN for action recognition rapidly is non-trivial. In this paper, a novel encoding called nonlinear gated channels unit (NGCU) is proposed to facilitate network training by encoding the global channel-level relationship. Based on this, nonlinear gated channels network (NGCN) is constructed for the end-to-end encoding, and the corresponding convergence performance is evaluated on the standard benchmarks UCF101 and HMDB51. Experimental results demonstrate that the proposed method is conducive to the convergence process of CNN based action recognition models.

Introduction

With modern learnable representations such as deep convolutional neural networks (CNNs) matured in many image understanding tasks [1], [2], [3], action recognition task has attracted wide attentions [6,7,16,17,20,29,[40], [41], [42]. Recent theoretical and empirical works have demonstrated that training effective CNNs rapidly is vital to various applications. Current methods for this goal can be roughly categorized into two factors, the external and internal algorithms. In the external pipeline, a large majority of optimization algorithms are proposed for faster convergence on the basis of stochastic gradient descend (sgd). On the one hand, with the historical gradient accumulated into the SGD momentum is proposed to dampen oscillations [23]. Additionally, nesterov accelerated gradient (NAG) first makes an anticipatory update towards the previous accumulated gradient, then a correction is conducted to avoid large fluctuation in the initial training stage [24]. On the other hand, each dimension of the CNN parameters should be updated by the various learning rates. In view of this, other improvement on the training optimizers is to design adaptive learning rates for the training parameters [25,26]. Another internal pipeline focuses on embedding hand-crafted rules inside of CNNs, which aim to guide models to learn towards the direction facilitating networks training. Glorot et al. have demonstrated that sparsity and neurons operating mostly in a linear regime can be brought together in biologically plausible neural networks, and Relu can help to bridge the gap between unsupervised pre-training and no pre-training [21]. Owing to the downside that margin changes in the parameters distribution of the lower layer will lead to large changes in the high-level parameters, it is time-consuming for CNN to adapt to the covariate shift. Therefore, batch normalization (BN) is proposed for dramatically accelerating the training process of deep networks by keeping the distribution of model's parameters consistent with Gaussian [22].

Despite that the internal and external pipelines mentioned above can facilitate network converging, training CNNs for action recognition is still cost-ineffective and time-consuming [[8], [9], [10], [11], [12], [19]], which is mainly obstructed by exponentially-growing temporal parameters [4,5,22]. This empowers the solutions to move one important step towards techniques for training network. Our current work reconsiders network training on the video sequences and intends to embed the hand-crafted rule into CNNs for the fast convergence, which is also complementary to both pipelines mentioned above. Given an image with r, g and b channels, there seems to be a certain relationship across r, g and b channels intuitively, co-constructing the final image. That is to say, r, g and b channels are indispensable to a complete image. Similarly, multiple convolutional channels responses of CNN can be explained as various local features, each of which is not completely independent of others. With the ability of capturing the global relationship between multiple local features, CNN can perceive the regional association, further rapidly recognizing the action in the training. Additionally, constructing the nonlinear relationship among the channel responses explicitly can increase the receptive field of multiple body parts. More specifically, three main contributions are provided in this work.

Our first contribution is to propose a novel unit called nonlinear gated channels unit, which can construct the nonlinear global relationship across the convolutional channel-level responses. With the channel-level relationship constructed, CNN can ascertain the appearance and motion of the whole body, thus accelerating the training process of video based CNNs.

Our second contribution is to simplify the proposed nonlinear gated channels unit (NGCU) and make a fast yet accurate implementation. It should be noted that, due to the setting that the responses of NGCU are kept consistent with that of the input, NGCU can thus be embodied inside of any standard CNN for end-to-end training.

Our third contribution is to construct the nonlinear gated channels networks (NGCNs) depicted in Fig. 1. The NGCN has been shown to result in a fast decline in the training loss while achieves a marginal performance improvement of actions classification. Additionally, the nonlinear gated channels network is complementary to the training tricks and optimizers.

The rest of the paper is organized as follows: Section 2 represents the related works. Section 3 makes a detailed description of the nonlinear gated channels networks (NGCN) and the backward propagation of NGCU. This is followed by the experiments in Section 4. Finally, we conclude this paper in Section 5.