Interactive reconstruction of virtual environments from video sequences

Abstract

There are many real-world applications of Virtual Reality requiring the construction of complex and accurate three-dimensional models that represent real environments. In this paper, we describe a rapid and robust semi-automatic system that allows such environments to be quickly and easily built from video sequences captured with standard consumer-level digital cameras. The system combines an automatic camera calibration algorithm with an interactive model-building phase, followed by automatic extraction and synthesis of surface textures from frames of the video sequence. The capabilities of the system are illustrated using a variety of example reconstructions.

Introduction

Building three-dimensional models that resemble real environments has always been a difficult problem. Generally, this requires modelling accurate three-dimensional geometry, as well as surface materials or textures covering each surface. In addition to the appearance of the environment, modelling the behaviour of objects is also very important if a Virtual Environment (VE) is to allow any kind of user interaction. Generally, a scene hierarchy is constructed by specifying the relationships between objects in the scene. These relationships can then be used to assist the user when interacting with the environment.

Traditional methods of constructing models have involved a skilled user and a three-dimensional computer-aided design (CAD) program. Accurately modelling a real environment in such a way can only be done if the user has obtained maps or blueprints of the scene, or has access to the scene in order to take precise physical measurements. Either way, the process is slow and laborious for anything but the simplest of scenes. Obtaining surface texture information has also traditionally been very difficult, requiring manual warping and mapping of photographics images onto each surface.

In the field of computer vision, automatic techniques have recently been developed that allow three-dimensional information to be constructed directly from photographs of the scene [1], [2], [3], [4]. Typically, these algorithms analyse multiple images of an environment in order to infer the position and attributes of each camera, as well as the three-dimensional location of a dense set of points corresponding to important features in the images. In order to build more useful polygonal models, these points must be triangulated and subsequently segmented into separate objects. Similar triangulation and segmentation algorithms are required when expensive laser-range scanners are used to sample the scene geometry [5], [6].

Automatic reconstruction techniques are, however, not yet robust enough to build useful VEs, which must have a simple enough form to allow them to be rendered in real-time and must contain enough structure so that a user can interact with important objects. Additionally, automatic algorithms are able only to reconstruct geometry that is seen explicitly in the images. This causes problems such as the appearance of holes and gaps in the model in regions that are not visible in any image, but which might well become visible when the model is being examined from different viewpoints.

In order to overcome problems such as these, semi-automatic approaches have been proposed that effectively combine user-guided segmentation of objects with automatic calculation of the position of those objects and the camera parameters [7], [8], [9], [10], [11], [12]. The benefit of using semi-automatic, rather than fully automatic algorithms, is that we can employ user knowledge when modelling the environment: the walls of a room may be identified by the user and modelled as single large polygons, thereby overcoming problems caused by object occlusion. An object hierarchy may also be easily maintained during the construction of the scene, and environments may be constructed in an incremental fashion, with large features specified at the start of the construction process and extra details added as necessary depending upon the envisaged use of the model.

The approach described in this paper falls between these automatic and semi-automatic categories. One significant disadvantage of current semi-automatic techniques is that they are limited to small numbers of input images due to the large amount of user interaction required to identify enough common features for camera calibration. In contrast, the main source of input data for our system is video sequences. Because of this, we can combine automated feature tracking [13], robust structure-from-motion [14] and self-calibration algorithms to automatically determine the camera parameters for each frame of the sequence. An overview of the calibration process is given in Fig. 1.

Once these calibration data have been obtained, we use semi-automatic techniques to build geometric descriptions of the objects in the scene. This is achieved by interactively manipulating the position, orientation and size of simple objects such as polygons, boxes and cylinders so that their projections into the frames of the video sequence match the projections of real objects. This manipulation is achieved using hierarchical parent–child constraints and image-based constraints that indicate the preferred projections of object vertices into frames of the sequence. We will describe a non-linear optimization algorithm that is capable of manipulating the position of these objects in real-time, so that the constraints specified by the user are best satisfied. Combining automatic camera calibration algorithms with interactive geometry reconstruction allows the user to spend more time modelling important features of the scene, rather than preparing the system for camera calibration.

We are interested in applying these techniques and algorithms to real-world problems, where the ability to quickly construct an accurate VE corresponding to a real scene can be an extremely powerful tool. In particular, we are studying the problem of constructing VEs of crime scenes, using forensic photographs [15] and video images and are cooperating with the Greater Manchester Police force (UK) and the UK's National Training Centre for Scientific Support to Crime Investigation (NTCSSCI) [16].

An earlier pilot project [17], [18] examined the feasibility of using VE reconstructions for police work. In the pilot project we undertook an entirely manual creation of a VE corresponding to a real crime scene, using architectural drawings and forensic photographs. The construction was extremely labour intensive, however, discussions with police officers, forensic scientists and trainers who had evaluated the pilot project demonstrated that such reconstructions can be of great benefit, offering new possibilities for analysis, training and briefing presentations. For such applications, the accuracy and fidelity of the VE are clearly very important, although different applications will be satisfied with various degrees of fidelity. The techniques described in this paper are currently being evaluated with reference to this area, although the algorithms described here are general enough to build reconstructions for a variety of different applications. Due to confidentiality requirements, the example reconstructions we show at the end of the paper are not from real crime scenes. Instead, we show examples of the system in use for typical indoor and outdoor reconstructions.

The remainder of this paper is structured as follows: in the next section we describe our approach to automatic calibration of video sequences. Following that, Section 3 describes how this calibration data may be used in an interactive setting to reconstruct geometric descriptions of objects in the scene. Once geometry has been reconstructed, we describe our automatic algorithm for extracing surface texture data in Section 4. Finally, different examples of the system in use are given in Section 5, and the paper closes with a discussion of current and future work in Section 6.