Scene point constraints in camera auto-calibration: an implementational perspective
Introduction
The structure and motion recovery problem concerns the recovery of both the scene structure and motion of the camera(s) from a set of images. Methods for solving this problem vary largely depending on the camera model chosen, these being, notably, the perspective [1], [2] and the affine [3], [4] camera models. The type of image features being used, e.g. image lines, points, or conics, on the other hand, determines the minimum number of images required for shape recovery [5], [6], [7]. If no special information about the camera or the scene is available then only a projective reconstruction of the scene can be obtained, cf. [1], [8], [2]. Since projective structures are not suitable for visualization, it is often desirable to carry out camera auto-calibration (see, e.g. [1], [2]) to obtain a Euclidean reconstruction up to an unknown similarity transformation.
Traditionally, the Euclidean structure of a scene has been obtained via two different types of methods. The first is often referred to as stratification, since one starts with a projective reconstruction and then finds an affine ‘stratum’ and finally a Euclidean ‘stratum’ to give the desired reconstruction. In the upgrade from projective to Euclidean reconstruction, some scene constraints, e.g. some distance or angular measurements, cf. [9] may be incorporated. The second type of method, often referred to as auto-calibration, takes into account some a priori information about the intrinsic parameters, e.g. known skew and/or aspect ratio (see [10], [11], [12], [13]) or constant intrinsic parameters (see [1]). The main focus of this latter type of method is on finding the intrinsic parameters, i.e. auto-calibrating the cameras.
Auto-calibration which involves the recovery of the absolute conic or the absolute dual quadric has been attempted by various researchers. The earliest work on recovery of the absolute conic is the algorithm by Faugeras et al. [14], where one camera is involved and its intrinsic parameters are assumed fixed. By assuming also fixed camera intrinsic parameters, Heyden and Åström [15] later retrieved Euclidean structures via the computation of the absolute dual quadric. With the same assumption, Triggs [11] proposed the use of quasi-linear constraints for the absolute dual quadric. Pollefeys et al. [16], on the other hand, incorporated the so-called modulus constraint into the stratification approach to upgrade projective structures to affine and finally recover the absolute conic and upgrade the structures to Euclidean. In their other work [13], they first assumed the principal points in all images vanish to linearize the equations for computing the absolute dual quadric, a method that we adopt in this paper. They then fine tuned the absolute dual quadric via a nonlinear optimization process in which the intrinsic parameters of the images are allowed to vary independently. The possibility of recovering camera intrinsic parameters under various conditions (e.g. a certain parameter is known or when they are known to be fixed) was proven by Heyden and Åström [17].
Although Pollefeys et al. [13] have indicated that scene constraints can be incorporated into auto-calibration, they did not carry out further research along this line. Liebowitz and Zisserman [18] later reported their use of the image projections of parallel and orthogonal scene lines to estimate the vanishing points and as constraints for estimating the absolute conic. However, their method requires the computation of the fundamental matrix, and that limits it to working with two images. Now it is recognized that scene constraints can be easily incorporated into various computer vision problems. Triggs et al. [19] report a detailed literature survey on bundle adjustment and the incorporation of scene constraints into the process. Gong and Xu [20], [21] have also attempted to incorporate constraints into surface recovery problems for calibrated cameras.
This paper is an extension of our previous work reported in [22], [23]. As before, we will use the natural camera model, i.e. a model that has zero skew and unit aspect ratio, and orthogonal scene planes as constraints for the estimation of the absolute dual quadric and bundle adjustment. The use of the natural camera model is justified by the high quality digital and video cameras available today. Even if the skew is non-zero and the aspect ratio of a camera is not unity, these two entities are known to be invariant under change of focus and so they can be pre-calibrated and treated as constant. The use of orthogonal scene planes is justified by the presence of many such planes in environments that contain man-made objects, e.g. indoor scenes, buildings. The scheme of incorporating scene constraints into auto-calibration covers a number of steps, namely, projective structure retrieval, absolute dual quadric estimation, Euclidean structure upgrade, and bundle adjustment. We will discuss each step in detail in the rest of the paper.
The paper is organized as follows. Section 2 recapitulates the background on projective reconstruction and factorization. In this section, the iterative projective factorization will be reviewed and the gauge freedom that we found essential to the subsequent auto-calibration step will be discussed. Section 3 then covers the linear estimation of the absolute dual quadric, the incorporation of orthogonal scene plane constraints, and the upgrade of projective structures to Euclidean. Section 4 discusses bundle adjustment without and with scene constraints imposed and the various issues we have studied during our investigation. Section 5 reports our experiments testing all the steps involved with both simulated and real video images. Section 6 lists other scene constraints that can be incorporated and a few issues on parameter setting. Finally, Section 7 concludes the paper.
Section snippets
Background on projective reconstruction
Given a scene point its projection onto an image plane is governed by:where the superscript j denotes the jth scene (or image) point from a list of scene (or image) points and ξj an unknown scalar known as the relative depth (also referred to as the projective depth). The first matrix is the camera matrix K which embodies the unknown focal length f and principal point (u0, v0) of the camera. The second matrix is the motion
Linear estimation of the absolute dual quadric
The structure contained in the shape matrix X is projective only, since for any joint projection matrix and joint shape matrix that satisfy (2), and also form a solution, where A is any non-singular 4×4 matrix.
To upgrade the projective structure to Euclidean, we must seek a matrix A for an appropriate change of coordinates in the estimated scene points. That is, we estimate A such thatwhere ∼ denotes equality up to an unknown scale. Let A be of the form
Bundle adjustment
The parameters obtained from the auto-calibration and Euclidean upgrade processes above are not optimal and an iterative refinement on the scene point coordinates and intrinsic and extrinsic parameters is required. This process of iteratively refining all these parameters simultaneously with the objective of minimizing the reprojection errors of feature points is known as bundle adjustment. In this section, we will discuss in detail how scene constraints are incorporated in our bundle
Results
We tested the entire procedure described in the previous three sections on synthetic image data as well as real image data.
For all the experiments on both the synthetic and real image data, the iterative projective factorization method described in Section 2.2 was first applied. Two independent processes were then carried out and their outputs were compared:
- •
Process (i): The absolute dual quadric was estimated using the linear method with no scene constraints added to the linear system. After
Discussion
We found that the final Euclidean structure depended significantly on the initial estimate of the absolute dual quadric obtained by the linear method and that the gauge of the system also played an important part on the estimation of absolute dual quadric and the initial Euclidean structure. Other factors, such as the distribution of corresponding points in the images and the depth variation of objects in the scene, all make a contribution to the initial estimation of the absolute dual quadric.
Conclusion
We have described a scheme for incorporating scene constraints into the auto-calibration process for the structure and motion recovery problem. The scheme involves a number of steps ranging from projective factorization, computation of the projective structure of the scene, estimation of the absolute quadric for Euclidean upgrade, to bundle adjustment for statistically optimizing the initial Euclidean reconstruction. In both the camera auto-calibration and bundle adjustment steps, we have shown
References (45)
- et al.
It can be done without camera calibration
Pattern Recognition Letters
(1991) - et al.
An Iterative Factorization Method for Projective Structure and Motion from Image Sequences
Image and Vision Computing
(1999) What Can be Seen in Three Dimensions with an Uncalibrated Stereo Rig?
Estimation of Relative Camera Positions for Uncalibrated Cameras
- et al.
Shape and Motion from Image Streams under Orthography: a Factorization Method
International Journal of Computer Vision
(1992) Conic Reconstruction and Correspondence from Two Views
IEEE Transactions on Pattern Analysis and Machine Intelligence
(1996)- R.I. Hartley, A Linear Method for Reconstruction from Lines and Points, in: Proc. International Conference on Computer...
- F. Kahl, A. Heyden, Using Conic Correspondences in Two Images to Estimate the Epipolar Geometry, in: Proc....
- K. Åström, A. Heyden, F. Kahl, M. Oskarsson, Structure and Motion from Lines under Affine Projections, in: Proc....
Stratification of Three-Dimensional Projective, Affine and Metric Representations
, Journal of the Optical Society of America
(1995)
Self-calibration of a moving camera from point correspondances and fundamental matrices
International Journal of Computer Vision
Camera Self-Calibration: Theory and Experiments
Cited by (6)
A flexible architecture for multi-view 3DTV based on uncalibrated cameras
2014, Journal of Visual Communication and Image RepresentationCitation Excerpt :Projective reconstruction is then obtained by robustly fitting the planes to each of the geometric classes [15]. Additional orthogonal scene plane constraints (e.g. left-ground, right-ground, center-ground planes) as well as camera intrinsic constraints [19,20] are imposed on absolute quadric to upgrade it to Euclidean one. Further refinement is done by bundle adjustment [22].
Plane-based camera self-calibration by metric rectification of images
2008, Image and Vision ComputingAutomatic Calibration and Reconstruction for Active Vision Systems
2012, Intelligent Systems, Control and Automation: Science and EngineeringCombining Mendonça-Cipolla self-calibration and scene constraints
2011, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Metrics for 3D rotations: Comparison and analysis
2009, Journal of Mathematical Imaging and VisionThree-dimensional reconstruction using the perpendicularity constraint
2007, 3DIM 2007 - Proceedings 6th International Conference on 3-D Digital Imaging and Modeling