Elsevier

Image and Vision Computing

Volume 23, Issue 8, 1 August 2005, Pages 747-760
Image and Vision Computing

Scene point constraints in camera auto-calibration: an implementational perspective

https://doi.org/10.1016/j.imavis.2005.05.003Get rights and content

Abstract

We present a scheme for incorporating scene constraints into the auto-calibration process for the structure and motion recovery problem. The steps covered by the scheme include projective factorization of the joint image measurement matrix, recovery of the absolute dual quadric, the upgrade from projective structure to its Euclidean counterpart, and incorporation of constraints from orthogonal scene planes into bundle adjustment. The focus of the paper is on the implementation details of all these steps and discussion of the various issues that arose. We have tested the scheme on both synthetic and real image data and found that it is more advantageous to incorporate into camera auto-calibration and bundle adjustment as many scene constraints as are available rather than performing auto-calibration and bundle adjustment alone.

Introduction

The structure and motion recovery problem concerns the recovery of both the scene structure and motion of the camera(s) from a set of images. Methods for solving this problem vary largely depending on the camera model chosen, these being, notably, the perspective [1], [2] and the affine [3], [4] camera models. The type of image features being used, e.g. image lines, points, or conics, on the other hand, determines the minimum number of images required for shape recovery [5], [6], [7]. If no special information about the camera or the scene is available then only a projective reconstruction of the scene can be obtained, cf. [1], [8], [2]. Since projective structures are not suitable for visualization, it is often desirable to carry out camera auto-calibration (see, e.g. [1], [2]) to obtain a Euclidean reconstruction up to an unknown similarity transformation.

Traditionally, the Euclidean structure of a scene has been obtained via two different types of methods. The first is often referred to as stratification, since one starts with a projective reconstruction and then finds an affine ‘stratum’ and finally a Euclidean ‘stratum’ to give the desired reconstruction. In the upgrade from projective to Euclidean reconstruction, some scene constraints, e.g. some distance or angular measurements, cf. [9] may be incorporated. The second type of method, often referred to as auto-calibration, takes into account some a priori information about the intrinsic parameters, e.g. known skew and/or aspect ratio (see [10], [11], [12], [13]) or constant intrinsic parameters (see [1]). The main focus of this latter type of method is on finding the intrinsic parameters, i.e. auto-calibrating the cameras.

Auto-calibration which involves the recovery of the absolute conic or the absolute dual quadric has been attempted by various researchers. The earliest work on recovery of the absolute conic is the algorithm by Faugeras et al. [14], where one camera is involved and its intrinsic parameters are assumed fixed. By assuming also fixed camera intrinsic parameters, Heyden and Åström [15] later retrieved Euclidean structures via the computation of the absolute dual quadric. With the same assumption, Triggs [11] proposed the use of quasi-linear constraints for the absolute dual quadric. Pollefeys et al. [16], on the other hand, incorporated the so-called modulus constraint into the stratification approach to upgrade projective structures to affine and finally recover the absolute conic and upgrade the structures to Euclidean. In their other work [13], they first assumed the principal points in all images vanish to linearize the equations for computing the absolute dual quadric, a method that we adopt in this paper. They then fine tuned the absolute dual quadric via a nonlinear optimization process in which the intrinsic parameters of the images are allowed to vary independently. The possibility of recovering camera intrinsic parameters under various conditions (e.g. a certain parameter is known or when they are known to be fixed) was proven by Heyden and Åström [17].

Although Pollefeys et al. [13] have indicated that scene constraints can be incorporated into auto-calibration, they did not carry out further research along this line. Liebowitz and Zisserman [18] later reported their use of the image projections of parallel and orthogonal scene lines to estimate the vanishing points and as constraints for estimating the absolute conic. However, their method requires the computation of the fundamental matrix, and that limits it to working with two images. Now it is recognized that scene constraints can be easily incorporated into various computer vision problems. Triggs et al. [19] report a detailed literature survey on bundle adjustment and the incorporation of scene constraints into the process. Gong and Xu [20], [21] have also attempted to incorporate constraints into surface recovery problems for calibrated cameras.

This paper is an extension of our previous work reported in [22], [23]. As before, we will use the natural camera model, i.e. a model that has zero skew and unit aspect ratio, and orthogonal scene planes as constraints for the estimation of the absolute dual quadric and bundle adjustment. The use of the natural camera model is justified by the high quality digital and video cameras available today. Even if the skew is non-zero and the aspect ratio of a camera is not unity, these two entities are known to be invariant under change of focus and so they can be pre-calibrated and treated as constant. The use of orthogonal scene planes is justified by the presence of many such planes in environments that contain man-made objects, e.g. indoor scenes, buildings. The scheme of incorporating scene constraints into auto-calibration covers a number of steps, namely, projective structure retrieval, absolute dual quadric estimation, Euclidean structure upgrade, and bundle adjustment. We will discuss each step in detail in the rest of the paper.

The paper is organized as follows. Section 2 recapitulates the background on projective reconstruction and factorization. In this section, the iterative projective factorization will be reviewed and the gauge freedom that we found essential to the subsequent auto-calibration step will be discussed. Section 3 then covers the linear estimation of the absolute dual quadric, the incorporation of orthogonal scene plane constraints, and the upgrade of projective structures to Euclidean. Section 4 discusses bundle adjustment without and with scene constraints imposed and the various issues we have studied during our investigation. Section 5 reports our experiments testing all the steps involved with both simulated and real video images. Section 6 lists other scene constraints that can be incorporated and a few issues on parameter setting. Finally, Section 7 concludes the paper.

Section snippets

Background on projective reconstruction

Given a scene point Xj=[XjYjZj1]T, its projection uj=[ujvj1]T onto an image plane is governed by:ξjuj=[f0u00fv0001][RRt]Xji.e.ξjuj=K[RRt]Xj,where the superscript j denotes the jth scene (or image) point from a list of scene (or image) points and ξj an unknown scalar known as the relative depth (also referred to as the projective depth). The first matrix is the camera matrix K which embodies the unknown focal length f and principal point (u0, v0) of the camera. The second matrix is the motion

Linear estimation of the absolute dual quadric

The structure contained in the shape matrix X is projective only, since for any joint projection matrix P˜ and joint shape matrix X˜ that satisfy (2), P˜A and A1X˜ also form a solution, where A is any non-singular 4×4 matrix.

To upgrade the projective structure to Euclidean, we must seek a matrix A for an appropriate change of coordinates in the estimated scene points. That is, we estimate A such thatPiAKi[RiRiti],fori=1,,m,where ∼ denotes equality up to an unknown scale. Let A be of the form

Bundle adjustment

The parameters obtained from the auto-calibration and Euclidean upgrade processes above are not optimal and an iterative refinement on the scene point coordinates and intrinsic and extrinsic parameters is required. This process of iteratively refining all these parameters simultaneously with the objective of minimizing the reprojection errors of feature points is known as bundle adjustment. In this section, we will discuss in detail how scene constraints are incorporated in our bundle

Results

We tested the entire procedure described in the previous three sections on synthetic image data as well as real image data.

For all the experiments on both the synthetic and real image data, the iterative projective factorization method described in Section 2.2 was first applied. Two independent processes were then carried out and their outputs were compared:

  • Process (i): The absolute dual quadric was estimated using the linear method with no scene constraints added to the linear system. After

Discussion

We found that the final Euclidean structure depended significantly on the initial estimate of the absolute dual quadric obtained by the linear method and that the gauge of the system also played an important part on the estimation of absolute dual quadric and the initial Euclidean structure. Other factors, such as the distribution of corresponding points in the images and the depth variation of objects in the scene, all make a contribution to the initial estimation of the absolute dual quadric.

Conclusion

We have described a scheme for incorporating scene constraints into the auto-calibration process for the structure and motion recovery problem. The scheme involves a number of steps ranging from projective factorization, computation of the projective structure of the scene, estimation of the absolute quadric for Euclidean upgrade, to bundle adjustment for statistically optimizing the initial Euclidean reconstruction. In both the camera auto-calibration and bundle adjustment steps, we have shown

References (45)

  • R. Mohr et al.

    It can be done without camera calibration

    Pattern Recognition Letters

    (1991)
  • A. Heyden et al.

    An Iterative Factorization Method for Projective Structure and Motion from Image Sequences

    Image and Vision Computing

    (1999)
  • O.D. Faugeras

    What Can be Seen in Three Dimensions with an Uncalibrated Stereo Rig?

  • R.I. Hartley

    Estimation of Relative Camera Positions for Uncalibrated Cameras

  • C. Tomasi et al.

    Shape and Motion from Image Streams under Orthography: a Factorization Method

    International Journal of Computer Vision

    (1992)
  • L. Quan

    Conic Reconstruction and Correspondence from Two Views

    IEEE Transactions on Pattern Analysis and Machine Intelligence

    (1996)
  • R.I. Hartley, A Linear Method for Reconstruction from Lines and Points, in: Proc. International Conference on Computer...
  • F. Kahl, A. Heyden, Using Conic Correspondences in Two Images to Estimate the Epipolar Geometry, in: Proc....
  • K. Åström, A. Heyden, F. Kahl, M. Oskarsson, Structure and Motion from Lines under Affine Projections, in: Proc....
  • O. Faugeras

    Stratification of Three-Dimensional Projective, Affine and Metric Representations

    , Journal of the Optical Society of America

    (1995)
  • Q.-T. Luong et al.

    Self-calibration of a moving camera from point correspondances and fundamental matrices

    International Journal of Computer Vision

    (1997)
  • B. Triggs, Autocalibration and the Absolute Quadric, in: Proc. IEEE Conf. on Computer Vision and Pattern Recognition,...
  • A. Heyden, K. Åström, Euclidean Reconstruction from Image Sequences with Varying and Unknown Focal Length and Principal...
  • M. Pollefeys, R. Koch, L.V. Gool, Self-Calibration and Metric Reconstruction in spite of Varying and Unknown Internal...
  • O. Faugeras et al.

    Camera Self-Calibration: Theory and Experiments

  • A. Heyden, K. Åström, Euclidean Reconstruction from Constant Intrinsic Parameters, in: Proc. International Conference...
  • M. Pollefeys, L.V. Gool, M. Oosterlinck, The Modulus Constraint: A New Constraint for Self-Calibration, in: Proc. Int....
  • A. Heyden, K. Åström, Flexible Calibraton: Minimal Cases for Auto-Calibration, in: Proc. International Conference on...
  • D. Liebowitz, A. Zisserman, Combining Scene and Auto-calibration Constraints, in: Proc. Int. Conf. on Computer Vision,...
  • B. Triggs, P.F. McLauchlan, R.I. Hartley, A.W. Fitzgibbon, Bundle Adjustment - A Modern Synthesis, in: Vision...
  • R. Gong, G. Xu, Quadratic Surface Reconstruction from Multiple Views Using SQP, in: Proc. Asian Conf. on Computer...
  • R. Gong, G. Xu, 3D Structure from a Single Calibrated View Using Distance Constraints, in: Proc. Asian Conf. on...
  • Cited by (6)

    • A flexible architecture for multi-view 3DTV based on uncalibrated cameras

      2014, Journal of Visual Communication and Image Representation
      Citation Excerpt :

      Projective reconstruction is then obtained by robustly fitting the planes to each of the geometric classes [15]. Additional orthogonal scene plane constraints (e.g. left-ground, right-ground, center-ground planes) as well as camera intrinsic constraints [19,20] are imposed on absolute quadric to upgrade it to Euclidean one. Further refinement is done by bundle adjustment [22].

    • Automatic Calibration and Reconstruction for Active Vision Systems

      2012, Intelligent Systems, Control and Automation: Science and Engineering
    • Combining Mendonça-Cipolla self-calibration and scene constraints

      2011, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
    • Metrics for 3D rotations: Comparison and analysis

      2009, Journal of Mathematical Imaging and Vision
    • Three-dimensional reconstruction using the perpendicularity constraint

      2007, 3DIM 2007 - Proceedings 6th International Conference on 3-D Digital Imaging and Modeling
    View full text