Automatic calibration of camera sensor networks based on 3D texture map information

Highlights

• Automatic calibration of complete 6DOF camera parameters using only 3D texture map information.

• A novel image descriptor based on Quantized Line parameters in the Hough space (QLH) is proposed for 2D–3D matching-based automatic calibration.

• Global estimation of the 6DOF camera parameters without initial conditions can be performed by applying particle filter-based optimization scheme.

Abstract

To construct an intelligent space with a distributed camera sensor network, pre-calibration of all cameras (i.e., determining the absolute poses of each camera) is an essential task that is extremely tedious. This paper considers the automatic calibration method for camera sensor networks based on 3D texture map information of a given environment. In other words, this paper solves a global localization problem for the poses of the camera sensor networks given the 3D texture map information. To manage the complete calibration problem, we propose a novel image descriptor based on quantized line parameters in the Hough space (QLH) to perform a particle filter-based matching process between line features extracted from both a distributed camera image and the 3D texture map information. We evaluate the proposed method in a simulation environment with a virtual camera network and in a real environment with a wireless camera sensor network. The results demonstrate that the proposed system can calibrate complete external camera parameters successfully.

Introduction

Distributed sensor networks installed in external environments can recognize various events that occur in the space, so that such space can be of much service in human–robot coexistence environments, as shown in Fig. 1. In recent years, many studies on an intelligent space, have been performed [1], [2]. Distributed camera sensor networks with multi-camera systems provide the most general infrastructure for constructing such intelligent space. In order to obtain reliable information from such a system, pre-calibration of all the cameras in the environment (i.e., determining the absolute positions and orientations of each camera) is an essential task that is extremely tedious. In this respect, several studies that provide Bayesian filter-based probabilistic estimates of optimal sensor parameters and target tracks have been conducted. Foxlin proposed the simultaneous localization and auto-calibration (SLAC) concept, which is a very general architectural framework for navigation and tracking systems with environment sensors [3]. Taylor et al. also implemented a simultaneous localization, calibration, and tracking (SLAT) system using radio and ultrasound pulse-based range sensors as environment sensors [4]. However, these methods can only be applied with range and bearing sensors and cannot make use of information from the camera sensor network, which is a popular network system for the intelligent space.

Chen et al. employed an approach that optimizes robot motion to minimize camera calibration errors; however, their approach assumes that the robot motion has no uncertainty, and rough parameters of the camera should be initialized by human observation [5]. Proposals by Rahimi et al. and Funiak et al. recovered the most likely camera poses and the target trajectory given in a sequence of observations from the camera network [6], [7]. However, these approaches have limitations in that they can only estimate 3 degree of freedoms (DOF) poses $(x,y,\varphi )$ with a restrictive assumption that requires aligning each camera’s ground-plane coordinate system with a global ground-plane coordinate system. The optimization problem of including orientation parameters for all axes $(\psi ,\theta ,\varphi )$ could have myriad local minimum solutions without additional constraints because many indistinguishable observations can exist even if the camera poses are different. For the reasons mentioned above, there has been no previous research that tried to estimate complete 6DOF camera parameters.

To overcome this limitation, our research group proposed a novel calibration method that estimates the complete 6DOF poses $(x,y,z,\psi ,\theta ,\varphi )$ for camera sensor networks by applying additional constraints [8]. The additional constraints consist of terms related to grid map information and a two-way observation model based on an assumption that the camera and target can observe each other. However, for this approach, a mobile agent is essential for implementing the calibration methods, so that these methods cannot be applied where the mobile agent cannot be used. In this respect, we propose another approach to achieve a complete calibration scheme for 6DOF external parameters that does not use any mobile agent, but instead, uses only the environmental map information to accommodate situations in which the mobile agent cannot be used. Therefore, the proposed complete 6DOF calibration system is able to construct a camera network system in arbitrary poses in the environment and easily calibrate its parameters under the assumption that there are no large structural alterations to a building (i.e., no wide discrepancies between the real environment and the map information).

In this approach, OctoMap, which is widely used to manage dense 3D environment models with texture information is utilized [9], [10]. The OctoMap divides the environment into irregular voxels that are managed in an Octree structure, so that it leads to very efficient memory management compared with a point-based structure. As shown in Fig. 2, the 3D texture map information can be utilized to generate virtual 2D images from arbitrary viewpoints (i.e., arbitrary 6DOF camera poses) using 3D projective geometry when the internal camera parameters are known; thus, the external camera parameters (i.e., 6DOF pose) can be determined by matching the virtual images generated at every viewpoint with real images from the camera sensor networks.

The problem addressed in this study can be considered to be similar to image-based self-localization problems of mobile robots, a dilemma that has been extensively studied over the past few years [11], [12], [13], [14], [15]. However, the solutions to these problems cannot be applied to complete 6DOF estimation problems but to only position estimation (or 3DOF estimation) problems, because the motion of mobile robots is expressed by 3DOF in 2D space. Furthermore, these methods can only be applied using omni-directional cameras to efficiently acquire surrounding information of a large environment and cannot make use of information from normal cameras. In particular, the method proposed by Ishizuka et al. achieved self-localization of mobile robots by matching 2D edge points observed from an omni-directional camera to 3D edge points obtained from the 3D environment model [15]. The equivalent method used in this study is called a 2D–3D edge matching scheme and several 6DOF registration techniques for a 3D geometric model (i.e., 3D map information) and 2D image data also have been previously proposed [16], [17], [18], [19], [20]. Most of these registration methods are based on the correspondence of 2D photometric edges and projected 3D geometrical edges on the 2D image plane. However, it is difficult to find corresponding edges correctly because robustly extracted edges are limited. Moreover, the initial pose should be manually set close to the correct pose to avoid being stuck in local minimums. To overcome the limitations of setting initial registration, Hara et al. proposed a new registration algorithm that can estimate an optimal pose robustly against initial registration errors [21]. However, it was not completely free from the initial set. The allowed maximum error of the initial registration is 2 m and 20 degrees for each axis. In conclusion, these registration schemes can be applied to matching 3D texture map information with 2D image data; however, initial registration by human observation should be performed and a global search algorithm (i.e., seeking a 6DOF solution in a global space) is yet to be established. Realizing the global search of the 6DOF solution with no strong constraints is impossible because of myriad local minimum solutions; thus, there has been no previous study that attempted this kind of approach. Here, the map information is very useful for reducing the solution space (i.e., the searching space for the 6DOF camera poses) given that the cameras are generally installed on the occupied region, such as interior walls, because of space limitations.

The contributions of this paper are as follows. The limitations of the early approaches are that they can estimate only 3DOF parameters $(x,y,\varphi )$ with restrictive assumptions and a mobile agent is needed for similar calibration patterns, as mentioned above. On the other hand, the proposed complete 6DOF calibration system in this paper only uses the environment map information; therefore, the proposed scheme easily calibrates its parameters without any mobile agent. Moreover, because we apply a novel matching scheme with a line feature-based descriptor that can manage some of the occlusions and clutter, the proposed calibration framework is relatively robust to illumination changes and also manages a few changes in the environment (i.e., discrepancies between the 3D map information and the camera image data) compared with the color information-based approach. In addition, the proposed calibration system requires no initial conditions because the particle filter-based approach, which is adapted for main paradigm for the proposed calibration task, has the ability to solve the global estimation problem, in comparison with previous local estimation scheme [21].

The remainder of this paper is organized as follows. Section 2 presents overview of the proposed calibration process based on 3D texture map information. Section 3 describes the parameterization step which converts 3D texture map information to simpler representation in detail. A novel image descriptor for a fast and accurate line-based matching process is presented in Section 4. Section 5 presents the particle filter-based parameter calibration step. The effectiveness of the proposed calibration scheme is evaluated with the experiment results in Section 6. Finally, Section 7 gives conclusions of this paper.