Video fingerprinting using Latent Dirichlet Allocation and facial images

Abstract

This paper investigates the possibility of extracting latent aspects of a video in order to develop a video fingerprinting framework. Semantic visual information about humans, more specifically face occurrences in video frames, along with a generative probabilistic model, namely the Latent Dirichlet Allocation (LDA), are used for this purpose. The latent variables, namely the video topics are modeled as a mixture of distributions of faces in each video. The method also involves a clustering approach based on Scale Invariant Features Transform (SIFT) for clustering the detected faces and adapts the bag-of-words concept into a bag-of-faces one, in order to ensure exchangeability between topics distributions. Experimental results, on three different data sets, provide low misclassification rates of the order of 2% and false rejection rates of 0%. These rates provide evidence that the proposed method performs very efficiently for video fingerprinting.

Introduction

Video fingerprinting [1], also known as content-based copy detection, or robust perceptual hashing [2], or near replica detection [3], refers to methods that try to identify whether a given video is a replica or a near replica of one of the videos existing in a video database. The need for efficient video fingerprinting algorithms is due to the enormous amount of video content and the scale of illegal video copying and distribution. Video sharing web sites such as YouTube need such algorithms in order to automatically check the intellectual property rights for videos that are uploaded in their database. Video fingerprinting is used in many applications such as copyright protection, multimedia databases management, broadcast monitoring, etc. Most methods, calculate a feature vector (i.e. the video fingerprint or perceptual hash) for the video in question, and compare it, by means of a similarity or distance function, against a set of vectors stored in a database.

Many methods exist for image perceptual hashing. In [4] Monga et al. have used non-negative matrix factorization and random projections to create a perceptual hash. In [5] a detailed theoretical work on non-negative matrix factorization in use with perceptual hashing is reported. In [6], a virtual watermark detection based method is used for perceptual hashing under the assumption that this virtual watermark detector responds similarly in images of similar content. Finally, in [7] the radon transform is used to create the image perceptual hashes.

However, their extension of image fingerprinting methods to video data (e.g. on a frame-by-frame basis) is not straightforward and efficient due to the temporal dimension. A limited number of video fingerprinting or replica detection techniques have been proposed in the literature so far. In [8], Lee et al. have used a novel binary fingerprint obtained by using a feature selection algorithm called the symmetric pairwise boosting. The binary fingerprints are obtained by filtering and quantizing perceptually significant features extracted from an input video clip. Oostveen et al. have proposed a spatio-temporal fingerprint based on the luminance difference in spatio-temporal blocks [9]. Coskun et al. have proposed two robust video hashing algorithms for copy identification that are based on the Discrete Cosine Transform (DCT) [10]. Hampapur and Bolle, compare various global video descriptors based on motion, color and spatio-temporal intensity distribution [11]. Law-To et al. propose a technique for video copy tracking which is based on labels of local descriptor behavior computed along the video [12]. Their aim was to distinguish copies within a collection of highly similar videos, as well as to link similar videos, in order to reduce redundancy in video collections or gather the associated metadata. Changick and Vasudev have proposed a copy detection scheme, where each video frame is partitioned into 2×2 blocks by intensity averaging [13]. Their spatio-temporal approach combines spatial matching of ordinal signatures obtained from the partitions of each video frame and temporal matching of signatures from the temporal partitions trails. Lee and Yoo present a video fingerprinting method based on 2-D Oriented Principal Component Analysis (2D-OPCA) of affine covariant regions [14]. According to this method, in order to achieve robustness against geometric transformations, the fingerprints are extracted from local regions, covariant with a class of affine transformations. For reliable local fingerprint matching, only spatio-temporally consistent matches are taken into account. Finally, in [15], the same authors, propose a novel video fingerprinting method based on the centroid of a pixel neighborhood gradient orientations. By these means, they achieve robustness against common video processing tasks such as frame rate change, lossy compression and others.

Latent Dirichlet Allocation (LDA) is a generative probabilistic model introduced in [16]. It is a powerful method for capturing statistical properties of a collection of conditionally independent and identically distributed random variables. The main idea behind LDA is the fact that such a set of random variables can be represented by a mixture of probability distributions. The latter is known as the de Finetti theorem [17]. This approach has already been applied in text modeling with good results [18]. It has been proven that LDA performs better than the pLSI (probabilistic Latent Semantic Indexing) algorithm, in the context of text modeling [16]. Moreover, this framework has been recently used in the context of image processing [19], [20], [21], [22], [23], [24], [25], where LDA is used for image classification, image retrieval and other image related analysis tasks. Moreover, LDA has been used in video analysis and description [26], [27], [28], [29], [30] for video summarization, scenes categorization and other tasks.

In this paper, we investigate the possibility of extracting latent aspects of a video in order to develop a video fingerprinting framework. Semantic visual information about humans, more specifically face occurrences in video frames, along with a generative probabilistic model (LDA), are used for this purpose. The latent variables, namely the video topics, are modeled as a mixture of distributions of faces in each video. The method also involves a clustering approach based on Scale Invariant Features Transform (SIFT) for clustering the detected faces and adapts the bag-of-words concept into a bag-of-faces one, in order to ensure exchangeability between topics distributions.

The novelty of this paper lies mainly in the use of latent aspects of the video content, aiming at extracting the underlying video topics and using them in video fingerprinting. In more detail, this paper includes the following novelties:

• The use of face occurrences in a video, to be called “facewords”, for describing this video. However, this framework can be easily extended to cases without humans, since “facewords” can be replaced by animals, even objects and scene artifacts, provided that an appropriate detector is used.

• The use of latent semantic analysis for video fingerprinting. Although many papers are using probabilistic latent semantic analysis (pLSA) for a number of image and video processing tasks, only recent publications like [19], [28] have utilized the LDA algorithm. However, none did use this framework for video fingerprinting, to the best of our knowledge.

• The face clustering technique which makes use of SIFT features evaluated on facial images.

The paper is organized as follows: in Section 2, an outline of the proposed framework is presented and a step-by-step presentation of how the main parts of the framework integrate. In Section 3, we introduce the face detection, facial feature extraction and face clustering methodology, the creation of the universal vocabulary and a procedure for incrementing the video database with new videos. Section 4 reviews the LDA framework. In Section 5, we explain the training phase, as well as the query mechanism of the proposed fingerprinting framework. Experimental results and complexity analysis are presented in Section 6. Finally, conclusions are drawn in Section 7.