Video summarization with a graph convolutional attention network

Abstract

Video summarization has established itself as a fundamental technique for generating compact and concise video, which alleviates managing and browsing large-scale video data. Existing methods fail to fully consider the local and global relations among frames of video, leading to a deteriorated summarization performance. To address the above problem, we propose a graph convolutional attention network (GCAN) for video summarization. GCAN consists of two parts, embedding learning and context fusion, where embedding learning includes the temporal branch and graph branch. In particular, GCAN uses dilated temporal convolution to model local cues and temporal self-attention to exploit global cues for video frames. It learns graph embedding via a multi-layer graph convolutional network to reveal the intrinsic structure of frame samples. The context fusion part combines the output streams from the temporal branch and graph branch to create the context-aware representation of frames, on which the importance scores are evaluated for selecting representative frames to generate video summary. Experiments are carried out on two benchmark databases, SumMe and TVSum, showing that the proposed GCAN approach enjoys superior performance compared to several state-of-the-art alternatives in three evaluation settings.

摘要

视频摘要已成为生成浓缩简洁视频的一种基础技术, 该技术有利于管理和浏览大规模视频数据. 已有方法并未充分考虑各视频帧之间的局部和全局关系, 导致摘要性能下降. 提出一种基于图卷积注意力网络 (graph convolutional attention network, GCAN) 的视频摘要方法. GCAN由嵌入学习和上下文融合两部分组成, 其中嵌入学习包括时序分支和图分支. 具体而言, GCAN使用空洞时序卷积对局部线索和时序自注意力建模, 能有效利用各视频帧的全局线索; 同时利用多层图卷积网络学习图嵌入, 反映视频帧样本的本征结构. 上下文融合部分将时序分支和图分支的输出信息流合并形成视频帧的上下文表示, 然后计算其重要性得分, 据此选择具有代表性的帧, 生成视频摘要. 在两个基准数据集SumMe和TVSum上的实验结果表明, 相比其他多种先进方法, GCAN方法在3种不同评测环境下取得更优越的性能.