Multiple consumer-grade depth camera registration using everyday objects

Highlights

• Utilizing pre-scanned shapes for joint calibration of multiple static depth cameras

• Joint extrinsic calibration and depth distortion correction

• A dataset for comparing different methods for multi-depth camera registration

Abstract

The registration of multiple consumer-grade depth sensors is a challenging task due to noisy and systematic distortions in depth measurements. Most of the existing works heavily rely on large number of checkerboard observations for calibration and registration of multiple depth cameras, which is tedious and not flexible. In this paper, we propose a more practical method for conducting and maintaining registration of multi-depth sensors, via replacing checkerboards with everyday objects found in the scene, such as regular furniture. Particularly, high quality pre-scanned 3D shapes of standard furniture are used as calibration targets. We propose a unified framework that jointly computes the optimal extrinsic calibration and depth correction parameters. Experimental results show that our proposed method significantly outperforms state-of-the-art depth camera registration methods.

Introduction

Consumer-grade depth cameras have become widely used as a low-cost means for real-time 3D sensing. However, such existing depth cameras have two major limitations: they only have a restricted field of view, and are able to only recover a depth field rather than true 3D data. One approach to overcome these problems would be to move the camera around [1] under the assumption of a static scene; but if real-time wide-coverage 3D data is needed, such as in 3D telepresence [2] and real-time novel view synthesis [3], then multiple depth cameras are required. The challenge then becomes that of registering and calibrating multiple depth cameras, given known problems that such cameras have systematic distortions in depth measurements, and are in general very noisy.

Extrinsic calibration may be conducted on a multi-camera rig, in order to determine the relative poses of the depth cameras with respect to a common reference frame. One approach is to fall back on classical image-based (i.e. optical) calibration techniques utilizing a checkerboard and color or raw IR imagery. Doing so would effectively calibrate the cameras through passive stereo triangulation, while completely ignoring the depth sensing mechanisms. However, 3D data subsequently obtained from the cameras will register poorly as the systematic depth distortions present are not accounted for. An alternative approach is to compute the rigid transform that will geometrically align 3D data separately sensed in the multiple cameras. Although doing so does not correct for the depth distortions, this optimization procedure will typically distribute 3D registration errors more evenly.

While depth distortion correction methods have been proposed for a single depth camera [2], [4], [5], [6], [7], combining such methods with extrinsic calibration in a sequential or piecemeal manner remains unsatisfactory and will still lead to significant misalignment, as the depth correction for individual devices tend to be sensitive to errors. Instead, we propose a unified framework in which extrinsic calibration and depth distortion correction are determined together.

For short experiments, an initial one-off calibration may be satisfactory. However if accurate registration has to be retained for an extended period of time, then in practice recalibration must be conducted at multiple intervals, due to mechanical displacement of the cameras (intentional, accidental, or environmental temperature variations, etc.) or simply sensor drift. This can pose a major practical problem if the calibration process is tedious, such as requiring tens or even hundreds of checkerboard observations for each recalibration. We thus seek to find a more practical method of maintaining registration, such as replacing checkerboards with everyday objects found in the scene, such as regular furniture.

In this work, we propose a new approach for registering depth maps obtained from multiple fixed depth cameras using a non-rigid alignment strategy. High quality pre-scanned 3D shapes of standard furniture are used as calibration targets, rather than checkerboards; these scans may be obtained offline using a moving depth camera together with a structure from motion algorithm such as KinectFusion [1]. The key idea of the proposed approach is to utilize the high quality geometry of the pre-scanned shape as both a reference and a conduit for registering the dynamic 3D data obtained from the multiple static depth cameras.

The contributions of this paper are as follows:

• To the best of our knowledge, this is the first work to primarily utilize pre-scanned shapes for joint calibration of multiple static depth cameras;

• A unified framework is presented that jointly computes the optimal extrinsic calibration and depth correction parameters; and

• A dataset is constructed for comparing different methods for multi-depth camera registration.