Multi-video summarization with query-dependent weighted archetypal analysis

Abstract

Given the tremendous growth of web videos, video summarization is becoming increasingly important to improve user’s browsing experience. Since most existing methods focus on generating an informative summarization from a single video and often fail to generate satisfying results for multiple videos, we propose an unsupervised framework for summarizing a set of topic-related videos. We develop a Multi-Video Summarization via Multi-modal Weighted Archetypal Analysis (MVS-MWAA) method to extract a concise summarization that is both representative and informative. To ensure the summarization query-dependent, we design a multi-modal graph to guide the generation of the weight in WAA, which we call query-dependent weight. Specifically, the multi-modal graph fuses information of video frames, tags, and query-dependent web images. Furthermore, we present a Ranking from Bottom to Top (RBT) approach to make it understandable. Extensive experimental results demonstrate that our approach clearly outperforms the state-of-the-art methods.

Introduction

With the development of multimedia technology and the popularity of handheld devices, there is an urgent need for an efficient technique to index and manage the increasing volume of unstructured videos. For example, people always desire to capture the main story in a video or several videos, especially news videos, as quick as possible. As one of the promising techniques, video summarization [1], [2], [3], [4] aims at condensing a long video or lots of short videos into a compact form [5], [6], which has drawn much attention in recent years.

Video summarization can be static or dynamic. Typically, a static summarization is formed with a number of keyframes, while a dynamic one is composed of a successive of video clips. In this paper, we focus on static summarization. In addition, according to the number of videos to be summarized, video summarization can be categorized into Single-Video Summarization (SVS) and Multi-Video Summarization (MVS). Most existing methods focus on SVS, whose purpose is to summarize a long video into a compact form [7], [8], [9], [10]. Recently, with the popularity of online news videos and personal videos, MVS receives increasing attention [11], [12], [13]. MVS in this paper refers to the query-dependent summarization, whose aim is to condense a large number of searched videos into a concise summarization. It enables users to quickly browse and comprehend the main idea of massive videos from the same query, thus is able to appeal more potential users. Generally, MVS is more challenging than SVS mainly because of the following three sides. (1) Since these videos are from the same query, they usually have high content redundancy. (2) These videos have plenty of irrelevant content, which demands MVS should be query-aware to narrow down the search intention gap. (3) SVS is usually displayed by the chronological sequence in the original video. However, MVS needs to deal with and analyze a large number of short videos with a few minutes. Accordingly, it is difficult to set an easy to understand representation order since the keyframes are from different videos.

In recent years, some important progress has been made in the research of MVS. However, generating a summarization from a series of topic-related videos is still a challenging problem. Some studies summarize videos of specific genres by utilizing some genre specific information [14], [15]. For example, [14] proposes to apply meta-data sensor information related to geographical area to summarize multiple sensor-rich topic-related videos. However, the utilization of this genre specific information in turn restrict this type of method in a wider range of application. Therefore, some recent attempts try to exploit the user search intents to narrow down the search intention gap for query-dependent summarization. For instance, Wang et al. [16] propose a method for event driven web videos summarization by tag localization and key-shot mining to generate satisfactory results, where the searched images are used to estimate the similarities between them and the keyframe of each shot in key-shot identification. In [17], the authors propose a multi-task deep visual-semantic embedding modal, where the query-dependent video thumbnails are generated based on both visual and side information (e.g., title, description, and query). Besides, Yao et al. [18] apply supervised learning method on video summarization. They propose a novel pairwise deep ranking model to learn the relationship between highlight and non-highlight video segments. And a two-stream network structure by representing video segments from complementary information on appearance of video frames and temporal dynamics across frames is developed for video highlight detection. Then the video summarization is extracted by highlight scores which are obtained by training the highlight detection model. However, how to efficiently reflect the search intend in MVS is still an open and challenging problem.

Recently, we have seen a proliferation of Archetypal Analysis (AA) algorithm in different fields, such as in economics [19], pattern recognition [20], document summarization [21], and computer vision [22]. It represents each individual in a dataset as a mixture of individuals of pure type or archetypes [23]. Recently, it is also used in video summarization [24]. Specifically, the authors propose a novel Co-Archetypal Analysis (CAA) algorithm, which learns canonical visual concepts shared between video and web images by finding a joint-factorial representation of two datasets. The frame-level importance is measured based on the learned factorial representation of the video and then combined into shot-level scores, by which the summarization is generated with a fix length.

In this paper, we present an alternate way to utilize AA to MVS. We propose a query-dependent MVS method by using weighted AA algorithm. Different from the idea of CAA in [24], we explore the web images searched by the same query and the tags around the videos as the query-dependent information to guide the AA algorithm in video summarization to be query-dependent. Fig. 1 depicts the framework of this paper.

The main contributions of this paper lie in the following three aspects:

• A novel MVS method with Weighted Archetypal Analysis (WAA) is proposed, which is called Multi-Video Summarization via Multi-modal Weighted AA (MVS-MWAA).

• To ensure the summarization query-dependent, we design a multi-modal graph to guide the generation of the weight in WAA, which we call query-dependent weight. Specifically, the multi-modal graph algorithm exploits not only video data, but its tags and query-independent web images.

• To make the summarization logical and readable, a novel Ranking from Bottom to Top (RBT) method is developed.

The rest of this paper is organized as follows. Section 2 reviews the related work. The brief concept of WAA is introduced in Section 3. Section 4 describes the details of the proposed MVS-MWAA method. Section 5 introduces the proposed RBT method for MVS presentation. Experiments are then presented and analyzed in Section 6. Section 7 concludes the paper.