Conceptual descriptions from monitoring and watching image sequences

Abstract

This paper contrasts two ways of forming conceptual descriptions from images. The first, called “monitoring”, just follows the flow of data from images to interpretation, having little need for top-level control. The second, called “watching”, emphasizes the use of top-level control and actively selects evidence for task-based descriptions of the dynamic scenes. Here we look at the effect this has on forming conceptual descriptions. First, we look at how motion verbs and the perception of events contribute to an effective representational scheme. Then we go on to discuss illustrated examples of computing conceptual descriptions from images in our implementations of the monitoring and watching systems. Finally, we discuss future plans and related work.

Introduction

This paper concerns the flow of information in the extraction of conceptual descriptions from image sequences. The questions we want to ask here are: is it just a bottom–up process? Or is there some top–down control? Do you start with small fragments of data (say, events or activity fragments) and use these to construct larger forms (say, episodes)? Or do you start with the larger form and try to find the data that fits what you want/expect to find?

We consider two contrasting approaches in some detail here: (1) passively going from images to conceptual descriptions which we call “monitoring”; and (2) a more active, task oriented version called “watching”. These instances are illustrated by the two computer programs hivis-monitor and hivis-watcher.

The passive systems used in “monitoring” require very little top-level control making them simple to implement. Basically the control level is data independent and unchanging having the single task of detecting events from the image data, such as, when an object changes motion by starting or stopping. Once the events are detected a monitoring system, such as hivis-monitor, then composes related, temporally ordered events to produce appropriate episodes which might, depending on the application environment and events observed, be closing a door or entering a roundabout. This type of system is useful for off-line analysis and collecting statistics or learning about different behaviours. However, it is not suitable for on-line use, as each episode needs to be completed before it can be used for higher level interpretation. For on-line analysis and evaluation we need immediate feedback concerning the likely emerging behaviour in the scene which entails a more active control of the processing focussed on the current task. We propose that attentional control in “watching” should depend on a set of preattentive operators that we know are related to the task. In hivis-watcher these operators dynamically allocate markers to objects of interest. For example, in this paper we show how preattentive selection using mutual-proximity can identify vehicles that may be involved in overtaking. Attentional processing then establishes indexed (deictic) state changes for the events that confirm overtaking behaviour. hivis-watcher‘s performance is completely dependent upon the given data and surveillance task it is asked to perform. Making it a much more flexible, real-time option.

As shown in Fig. 1 this implementation uses preprocessed visual data in the form of 3D poseboxes from a model-based tracker (details are given by Sullivan [76], Tan et al. [77] and Worrall et al. [82], [83]). This means that the preattentive stage is simulated in the hivis-programs—something that is being corrected as described in Section 5.

Our surveillance problem has the following simplifications that make visual understanding more tractable: we use a fixed camera that observes the activity of rigid objects in a structured domain. Examples include: a road traffic scene where the main interest is the road vehicles, and airport holding areas where we are interested in the activities of the various special vehicles that unload and service the passenger aeroplanes. We call this single viewpoint of the fixed camera the “official-observer”. From this camera input we wish to obtain a description of the activity, taking place in the dynamic wide-area scene, and then an understanding of the dynamic and improvised interactions of the scene objects. There are constraints on the official-observer's interpretation of the objects in the scene: we only see the objects that are in the camera's field-of-view; we do not know each participant's goal (typically something like “go to place X”); and what we see is mostly reactive behaviour (rather than deeply planned).

To provide a background to the work described here we first consider the nature of conceptual descriptions, before describing hivis-monitor and hivis-watcher. Two examples are used in this paper: a road-traffic one that presents scenarios such as overtaking; and an office example that is used to demonstrate how these two programs could be used. This second application area is the subject of future work, but is presented to show the range of applications being considered.