Large scale continuous visual event recognition using max-margin Hough transformation framework

Abstract

In this paper we propose a novel method for continuous visual event recognition (CVER) on a large scale video dataset using max-margin Hough transformation framework. Due to high scalability, diverse real environmental state and wide scene variability direct application of action recognition/detection methods such as spatio-temporal interest point (STIP)-local feature based technique, on the whole dataset is practically infeasible. To address this problem, we apply a motion region extraction technique which is based on motion segmentation and region clustering to identify possible candidate “event of interest” as a preprocessing step. On these candidate regions a STIP detector is applied and local motion features are computed. For activity representation we use generalized Hough transform framework where each feature point casts a weighted vote for possible activity class centre. A max-margin frame work is applied to learn the feature codebook weight. For activity detection, peaks in the Hough voting space are taken into account and initial event hypothesis is generated using the spatio-temporal information of the participating STIPs. For event recognition a verification Support Vector Machine is used. An extensive evaluation on benchmark large scale video surveillance dataset (VIRAT) and as well on a small scale benchmark dataset (MSR) shows that the proposed method is applicable on a wide range of continuous visual event recognition applications having extremely challenging conditions.

Introduction

Visual event recognition i.e. recognition of semantic spatio-temporal visual patterns such as “waving”, “boxing”, “getting into vehicle” and “running” etc. is a fundamental Computer Vision problem. An enormous amount of work on this topic can be found in literature survey [1], [2], [3], [4]. Recently, research in this field is directing towards continuous visual event recognition (CVER) where the goal is to both recognize an event and to localize the corresponding space–time volume from large continuous video [5] like in object detection in images where only spatial location is important. This area is more closely related to the real world video surveillance analytics need than the current research which aims to classify a prerecorded video clip of a single event. Accurate CVER would have direct and far reaching impact in surveillance, video-guided human behaviour analysis, assistive technology and video archive analysis.

The task of CVER, i.e. the activity detection on large scale real world video surveillance dataset, is an extremely challenging task and current state-of-the methods for 2D small scale action recognition become infeasible to apply. One of the main challenges for CVER is the scalability, e.g. a CVER dataset like VIRAT dataset [5] contains 23 event types distributed throughout 29 h of video. The other difficulties are due to (i) natural appearance since the events are recorded in a real world scenario, (ii) huge spatial and temporal coverage which affects the video resolution, e.g. the human heights within videos range 25–200 pixels constituting 2.4–20% of the heights of the recorded videos with an average being about 7%, (iii) diverse event types and (iv) huge variability in view-points, scenes and subjects (see Fig. 1) [5].

Among all these above mentioned difficulties, action detection in video (both small and large scale) is a challenging problem mainly due to the scalability of its search space. Without knowing the location, temporal duration and the spatial scale (spatial resolution of the activity) of the action, an exhaustive search is a NP-hard problem. For example, a 1 min video sequence of size 160 × 20 × 1800 contains more than 10¹⁴ spatial sub-volumes of various sizes and locations [6]. To solve this issue there are methods like discriminative sub-volume search [6], unsupervised random forest indexing [7]. Although promising, these works always use small scale video datasets like KTH¹ and MSR action datasets [6] where the challenges present in CVER, mentioned above, are absent.

To solve the search space complexity of CVER, it is necessary to apply a motion region identifier to roughly detect the motion region of interest where the events to be searched may appear. Oh et al. [5] apply a multi-object tracking using frame difference, and the obtained tracks are divided into detection units resulting over 20 K units, as a preprocessing step. This division of detection unit by a fixed amount always misses some events that are having different duration. On the other hand, in our approach first a motion segmentation method similar to [8] is applied to obtain the primary candidate region set. The obtained regions are further joined using a region clustering technique based on action heuristics. Finally, we obtain on an average 3 K candidate regions as opposed to 20 K by [5] with a higher recall rate. This has a major impact towards the search space reduction and on achieving faster event detection in large scale.

Our method for event detection is related to several ideas recurring in the literature. Firstly, we use STIP detector which is successfully applied in 2D action recognition problems [9], [10], [11], [12], [13]. Several local features are computed such as histogram of oriented optical flow (HOF) [14], histogram of oriented gradient in 3D (HOG3D) [15] and extended SURF (ESURF) [16] at the detected STIPs. We use the idea of local appearance codebook [17] including bag-of-word approach [18], [19] to group the detected features into a set of visual words that represent an event class.

The next idea is to use the generalized Hough transformation (GHT) framework for object detection in images into event detection in videos. Originally developed for detecting straight line [20], Hough transforms are generalized to use for detecting generic parametric shapes [21]. Recently, GHT scheme is successfully used for detecting object class instances tracking and action recognition [22], [23], [24], [25], [26], [27].

The concept of GHT usually refers to any detection process based on an additive aggregation of evidence, Hough votes, coming from local image/video elements. Such aggregation is performed in a parametric space called as Hough space, where each point corresponds to the existence of an instance in a particular configuration. The Hough space may be a product set of different locations, scales and aspects, etc. The detection process is then reduced to finding maxima peaks in the sum of all Hough votes in the Hough space domain, where the location of each peak gives the configuration of a particular detected object/event instance.

The implicit shape model of Leibe et al. [23] and the max-margin hough transformation of Maji and Malik [25] serve a baseline for our work. These works mainly focus on object detection. During training, they augment each visual words in the codebook with the spatial distribution of the displacements between the object centre and the respective visual word location. Using max-margin setup the weights of each visual words are learned. At the detection time, these spatial distributions are converted into Hough votes within the Hough transformation framework. The weights of the visual words are also used for extra information to the Hough votes.

To incorporate this idea into CVER, we need to extend the dimensionality of the voting space since now each STIP will vote for a parallelepiped centre i.e. the event centre. To make it easier to understand, we scale each candidate event into a normalized cube and during training the interest point (feature) distributions along the cube centre is learned for each event class. The scale information is also saved so that by using a simple reverse conversion the normalized cube can be transformed into the actual event parallelepiped. After obtaining a set of visual words from detected event features, a max-margin frame work similar to [25] is applied for learning weights of each visual words for each event class. For a test candidate region, the detected interest points (features) are matched with the event class visual words and weighted votes for the possible event centre are obtained in the Hough voting space. The votes corresponding to the peaks of Hough space reveal the possible hypothesis of the detected events in the actual video. Finally, a verification Support Vector Machine (SVM) designed for the particular event class is used to obtain the recognition score.

The main advantage of using a GHT framework is, it avoids the need of exhaustive search like in sliding window technique, which is infeasible to apply in CVER. GHT directly works on the STIPs and the local features that are extracted from the candidate motion regions. An instant probabilistic score can be obtained on the activity centre and based on which an activity hypothesis can be generated. By using a verification SVM a more robust recognition is obtained, once the activity hypothesis is generated by GHT.

To test our approach we use large scale CVER dataset, VIRAT, proposed by [5]. Our result shows state-of-the-art performance on this dataset. To show the wide applicability of our method, we choose small scale video search dataset MSR [6] and also obtain above state-of-the-art result.