Camera calibration from very few images based on soft constraint optimization

Abstract

Camera calibration is a basic and crucial problem in photogrammetry and computer vision. Although existing calibration techniques exhibit excellent precision and flexibility in classical cases, most of them need from 5 to 10 calibration images. Unfortunately, only a limited number of calibration images and control points can be available in many application fields such as criminal investigation, industrial robot and augmented reality. For these cases, this paper presented a two-step calibration based on soft constraint optimization, which is motivated by "no free lunch" theorem and error analysis. The key steps include (1) homography estimation with weighting function, (2) Initialization based on a simplified model, and (3) soft constraint optimization in terms of reprojection error. The proposed method provides direct access to geometric information of the object from very few images. After extensive experiments, the results demonstrate that the proposed algorithm outperforms Zhang's algorithms from the point of view of the success ratio, accuracy and precision.

Introduction

Camera calibration is the estimation process of the intrinsic and extrinsic camera parameters [1,2]. The extrinsic parameters represent the 3D position and orientation of the camera frame with respect to the world reference frame [3]. While the intrinsic parameters reflect the internal camera geometric and optical characteristics, which are invariant if the focal length is fixed [4].

With the development of theory and technique, camera calibration has been being widely applied in various fields such as biology, materials, photon imaging, physical measurement and object detection. At the same time, many problems emerged in actual conditions. One of them is calibration from a few images, which has received significant attention due to the fact that very small observed data can be available for many cases such as criminal investigation, industrial robot [5] and augmented reality [6], and that once the parameters are known, it is easy to obtain the position and geometry of the object from its image, or the relative position of the camera. Unfortunately, for the case that no enough data are available, most of conventional calibrations, which tend to focus on a general calibration purpose, do not perform well.

According to "no free lunch" theorem [7], matching algorithms to problems gives higher average performance than applying a fixed algorithm to all. Following this idea, one of the solutions to calibration from very few images is to introduce constraints into the calibration process. To this end, some researchers brought geometric constraints into camera calibration through the scene that satisfies a Manhattan world assumption, that is, the scene contains three orthogonal, dominant directions [8]. For example, by using cuboids, de la Fraga et al. [9] proposed direct calibration method based on differential evolution. Similarly, Wilczkowiak et al. [10] took a parallelepiped as a calibration object. The results demonstrated that the relative errors of intrinsic parameters are no bigger than 7%. Using cylindrical object, Winkler and Zagar [11] fitted extrinsic parameters. The testing results showed that this method is valid. Chen [12] realized camera calibration from four coplanar corresponding points and a non-coplanar one, of which the performance is better than that of Miyagawa's approach [13].

On the other hand, many people exploited coplanar calibration object to impose geometric constraints. Avinash and Murali [14] employed a rectangular prism to generate two vanishing points and then to determine focal length. The performance of this method compares favorably to the Zhang's algorithm [15]. Similarly, Dan Liu et al. [16] presented a calibration method for a camera with lens distortion from a single image including vanishing points and ellipses. However, for such methods, the accuracy is dependent upon the vanishing points [16]. To resolve this dilemma, without vanishing points, Zhou et al. [17] achieved comparable measurement accuracy relative to the traditional method by adopting a single image including at least three squares with unknown length. Using two arbitrary coplanar circles, Zheng and Liu [18] proposed a closed-form solution of the focal length and the extrinsic parameters. The result can serve as the initial value of camera parameters for iterative optimization. Furthermore, Miyagawa et al. [13] proposed a technique to estimate parameters from five points on two orthogonal 1-D objects. The calibration results are close to those yielded by existing methods.

In addition, many other constraints have been suggested, which can bring considerable benefits for camera calibration. For instance, a radial alignment constraint was enforced on camera calibration by Tsai [4]. A planarity constraint was employed by Zhang [15]. Miraldo and Araujo [19] introduced the smoothness constraint into camera calibration. Based on the epipolar constraint, Heller et al. [20] proposed a novel solution to the hand-eye calibration problem. To address the same issue, Liu et al. [21] presented a method based on rigidity constraints.

In a word, despite a large amount of research appearing in the literature, accurate and flexible calibration from very few images has remained a difficult problem. This is essentially due to non-convex optimization arising from perspective distortions and insufficient data available to eliminate noise [22]. To solve this problem, constraint optimization is a promising way to achieve the global optimal solution. Based on the above idea, a novel calibration method is designed here especially for very few calibration images. This method involves three major steps: (1) homography matrix is estimated by the non-linear optimization with weighting function; (2) intrinsic and extrinsic camera parameters are initially computed based on the simplified model by decomposing homography matrix; (3) they are refined via a soft constraint optimization in terms of reprojection error. Comparative experiments demonstrate that the effectiveness of the proposed algorithm from the point of view of success ratio, precision and accuracy.

In addition, the impact of shooting angle and control point number on parameter estimation was explored in this work. A pictorial display of error changes is presented with respect to the above factors. That can be employed as an action guide to tradeoff accuracy and cost for researchers and practitioners involved.

The remainder of the paper is structured as follows: Section 2 introduces preliminary of camera calibration, Section 3 analyzes the main resource of calibration errors, Section 4 describes the camera calibration from very few images, Section 5 shows the experimental results, Section 6 discusses how the shooting angle of calibration objects and the number of control points impact the parameter estimation, and finally, Section 7 states the conclusions.