Feature fusion for basic behavior unit segmentation from video sequences

Abstract

It has become increasingly popular to study animal behaviors with the assistance of video recordings. An automated video processing and behavior analysis system is desired to replace the traditional manual annotation. We propose a framework for automatic video based behavior analysis systems, which consists of four major modules: behavior modeling, feature extraction from video sequences, basic behavior unit (BBU) discovery and complex behavior recognition. BBU discovery is performed based on features extracted from video sequences, hence the fusion of multiple dimensional features is very important. In this paper, we explore the application of feature fusion techniques to BBU discovery with one and multiple cameras. We applied the vector fusion (SBP) method, a multi-variate vector visualization technique, in fusing the features obtained from a single camera. This technique reduces the multiple dimensional data into two dimensional (SBP) space, and the spatial and temporal analysis in SBP space can help discover the underlying data groups. Then we present a simple feature fusion technique for BBU discovery from multiple cameras with the affinity graph method. Finally, we present encouraging results on a physical system and a synthetic mouse-in-a-cage scenario from one, two, and three cameras. The feature fusion methods in this paper are simple yet effective.

Introduction

It has become an increasingly important research area to automatically analyze object behaviors from visually (e.g., motion) captured data or video recordings. The major tasks are to automatically detect and track objects from video sequences and analyze its high level activities or behaviors. Humans and vehicles have been mostly the focus of the visual surveillance and behavior understanding research [1], [2], [3], [4] for security purposes, e.g., access control in certain area, anomaly detection in crowded mass transportation area, etc.

In areas of biology, pharmacology, toxicology, entomology and animal welfare, video recordings are popularly used to analyze the behaviors of animals (e.g. lab mice, rodents, poultry, wild animals, etc.) The traditional human annotation approach is time consuming and results may vary from one observer to another. Hence the automatic animal behavior analysis from visual data is drawing more and more attention in both the research and industrial community [5], [6].

In the area of visual robot control, it is also desired for robots to automatically learn and recognize behaviors from motion capture or visual data [7], [8], [9], which would enable the intelligent robots to respond according to the visual information captured by cameras.

Among all the efforts made in an automated behavior analysis system, the basic behavior unit (BBU) classification (or segmentation) is one important task [10]. Usually the sequences of visual data from images need first to be grouped into BBUs [11], or primitive (atomic) behaviors [7], and then complex behaviors are analyzed based upon the relationship between the BBUs and context. Prior to the BBU segmentation step, spatiotemporal features are usually extracted. In the literature, interest points, shape properties of the detected object blobs, contours, or features derived thereby are used to perform BBU classification. Feature extraction itself is an important task.

In the literature, researchers has been trying to solve the BBU classification and feature extraction tasks separately. In this paper, we take a integrated approach and propose a framework for such an automatic behavior analysis system. We first present the framework, and then focus on investigating feature fusion techniques in BBU discovery: we will present the exploration of the vector fusion method [12] in feature dimension reduction, and the fusion of features from multiple cameras using the affinity graph method.

Our research is motivated by the need of a professor in medicine, who is interested in the automatic video analysis of behavior changes before and after injecting certain medicine to the lab mouse, as shown in Fig. 1. The behaviors interested includes resting, eating, exploring, and mostly importantly, grooming. In this paper, we use behaviors of the mouse-in-cage scenario for our experiments and analysis.