Dynamic self-attention with vision synchronization networks for video question answering

Highlights

• A novel token selection mechanism based on the dynamic self-attention network is proposed to automatically extract important video features.

• A vision synchronization network is proposed to align appearance and motion features at the time slice level.

• Extensive experiments and analysis confirm the superiority of the proposed model DSAVS.

Abstract

Video Question Answering (VideoQA) has gained increasing attention as an important task in understanding the rich spatio-temporal contents, i.e., the appearance and motion in the video. However, existing approaches mainly use the question to learn attentions over all the sampled appearance and motion features separately, which neglect two properties of VideoQA: (1) the answer to the question is often reflected on a few frames and video clips, and most video contents are superfluous; (2) appearance and motion features are usually concomitant and complementary to each other in time series. In this paper, we propose a novel VideoQA model, i.e., Dynamic Self-Attention with Vision Synchronization Networks (DSAVS), to address these problems. Specifically, a gated token selection mechanism is proposed to dynamically select the important tokens from appearance and motion sequences. These chosen tokens are fed into a self-attention mechanism to model the internal dependencies for more effective representation learning. To capture the correlation between the appearance and motion features, a vision synchronization block is proposed to synchronize the two types of vision features at the time slice level. Then, the visual objects can be correlated with their corresponding activities and the performance is further improved. Extensive experiments conducted on three public VideoQA data sets confirm the effectivity and superiority of our model compared with state-of-the-art methods.

Introduction

With the rapid development of computer vision and natural language processing, tasks involving vision and language have inspired considerable interests. A new problem named Visual Question Answering (VQA), including Image Question Answering (ImageQA) [1], [2], [3], [4] and Video Question Answering (VideoQA) [5], [6], [7], [8], has emerged as a promising but intractable research point. It requires the agent to provide the correct answer to a question about the contents of a given image/video. Compared with ImageQA where image just contains appearance features, VideoQA is more challenging due to three reasons: (1) it deals with a long sequence of images that contain not only appearance but also motion features; (2) substantial redundant information exists in the video; (3) different questions require different parts of the video to infer the answer. However, the achievement of VideoQA is beneficial for various real-life applications, including tourist assistance, automatic customer service, and human-machine interaction [9].

A number of methods have sprung up in recent years to tackle the difficulties faced by VideoQA. These methods can be broadly divided into two categories depending on the way in which visual features are processed. The first type of method [9], [10], [11], [12] only uses the appearance features in the video. The work presented in [13] introduces a learnable aggregating network with only the appearance features. It mainly learns the cross-modal correlations using diversity learning schemes, which neglects the motion information of the video and thus is hard to capture fine-grained features in time series. The second type of method [14], [15], [16], [17] incorporates both appearance and motion features as the input. The model proposed in [15] gradually refines its attentions over both appearance and motion features using the question as guidance. However, since appearance and motion features obtained from a video have redundant and interference information, directly inferring the answer from them will result in a negative impact on the performance. Besides, this group of methods deals with the appearance and motion features separately, which neglects the synchrony relation between the two types of features.

Meanwhile, the unique characteristics of visual appearance and motion features in videos provide important clues for VideoQA. First, the answer to a question is usually reflected on a few key frames or video clips, and most video information is superfluous. Inferring the answer directly from all the appearance and motion information is not only computationally expensive but also introduces a lot of noise. As the examples shown in Fig. 1(a), in the left instance, the question “what color is the phone in the hand of the woman?” can be answered from the frames that contain a phone in the hand of a woman. However, the object phone does not appear in most frames of the video. It means that most of the information in the video is useless to this question. In the right instance, a little boy is calling in the video. The action “press the phone” only exists in a short video clip. Therefore, the information existing outside this short clip in the video is useless to answer the question “How many times did the boy press the phone?” Second, the visual objects have their own activities in a video, and thus the appearance and motion features are often concomitant and complementary to each other at the time slice level. Dealing with the appearance and motion features independently may result in mistakes. Take the example shown in Fig. 1(b) as an instance. The video mainly contains two parts of information, i.e., a boy is jumping and a dog is running. The appearance features are extracted as a sequence of “boy-boy- $\cdots$ -dog-dog”, while the motion features are extracted as “jump-jump- $\cdots$ -running-running”. Most of the existing approaches concentrate on the appearance of the “boy” for the question “what is the boy in the white vest doing?” with an attention mechanism. They also prefer “running” with the guidance of “boy” from the video, as the motion “running” has higher probability than “jumping” when it is combined with “boy”, i.e., ($\mathbf{\text{p}}(boy,running)>\mathbf{\text{p}}(boy,jumping)$) in the training set. Then, a wrong answer “running” is inferred. The reason is that existing methods employ the attention mechanism to appearance and motion features separately, which neglect the synchrony relation between the two types of visual features. Obviously, the appearance and motion features have different roles in visual reasoning, and they are concomitant and complementary in time series. Therefore, these latent correlations between the appearance and motion features should be exploited, so that the action of the boy can be matched to jumping and the action of the dog to running.

However, it is quite difficult to capture the answer-related key information and the correlation between appearance and motion features. First, there is no explicit knowledge about which video frames and clips are relevant to the answer. Second, the appearance and motion features are represented in heterogeneous space, and it is difficult to find the motion features for a given visual object. To tackle these challenges, we propose to take advantage of the latent correlation between different types of video features and the answer for VideoQA. In particular, we investigate: (1) how to mine the answer-related frames and clips in the video; (2) how to learn the correlation between the appearance and motion features. Our solutions to these questions result in a novel VideoQA model, i.e., Dynamic Self-Attention with Vision Synchronization Networks (DSAVS). To automatically find out the key video frames and clips used for answer inference, a gated token selection mechanism with a dynamic self-attention block is proposed to select the supportive tokens from the input sequence. To effectively learn the correlation between appearance and motion features, we synchronize them at the time slice level with a vision synchronization block, where the two types of features are fused in the guidance of the question sentence. Then, both the dynamic self-attention and vision synchronization blocks are integrated into an end-to-end framework to infer the answer. The main contributions are summarized as follows:

• We propose a dynamic self-attention method to automatically select important video information to learn internal dependencies, avoiding a lot of redundant and noisy information.

• Unlike the existing VideoQA methods, we introduce a vision synchronization model to synchronize the appearance and motion features, which can avoid the misalignment of appearance and motion in time series.

• Extensive experiments conducted on three public VideoQA data sets confirm the effectivity and superiority of the proposed DSAVS compared with state-of-the-art methods.

The remainder of the paper is organized as follows. We first survey the related works. Then, the problem statement and the details of our model are presented. Next, we describe the experiments and the analysis of the results, and the paper is concluded in the end.