A subspace method for projective reconstruction from multiple images with missing data

https://doi.org/10.1016/j.imavis.2006.02.003Get rights and content

Abstract

In this paper, we consider the problem of projective reconstruction based on the subspace method. Unlike existing subspace methods which require that all the points are visible in all views, we propose an algorithm to estimate projective shape, projective depths and missing data iteratively. All these estimation problems are formulated within a subspace framework in terms of the minimization of a single consistent objective function, hence ensuring the convergence of the iterative solution. Experimental results using both synthetic data and real images are provided to illustrate the performance of the proposed method.

Introduction

The reconstruction 3D Euclidean structure from multiple uncalibrated 2D images has been a long standing difficult problem in computer vision that has many important applications such as geometric modeling and virtual scene synthesis. Many different approaches have been proposed for 3D reconstruction from multiple 2D images, including direct methods [1] that impose metric constraints from the outset to estimate cameras and 3D scene directly in an Euclidean frame, and stratified methods [2] that perform the recovery of 3D structure in stages, first in a projective frame followed by an upgrade to an affine and then the Euclidean frame. The stratified approach has the advantage that the 3D Euclidean reconstruction is decomposed into independent and simpler steps. Furthermore, it is shown in [3] that there is less chance for optimization algorithms to be trapped in local minima in a projective frame than a Euclidean frame, which is not surprising because optimization for the projective reconstruction is free of metric constraints. In the stratified approach, projective reconstruction is a necessary step prior to Euclidean reconstruction.

The factorization approach to projective reconstruction has received considerable attention in recent years. Factorization-based methods have an inherent advantage of being able to handle any number of images simultaneously without special treatment for any subgroup of views. In the factorization-based approach, the projective reconstruction problem is formulated as one of factorizing a scaled measurement matrix containing unknown depth parameters into a product of the structure and shape matrices. A key issue in the factorization approach is the determination of the unknown projective depths. In [4], the depths are determined by means of epipolar constraints in a non-iterative manner, but the method requires the estimation of the fundamental matrices between pairs of views and is sensitive to noise. Most of the other recent approaches use iterative methods to estimate the projective depths by minimizing an algebraic error (e.g. [5], [6], [7], [8]) or a subspace proximity measure (e.g. [9], [10], [11]). Alternatives to factorization methods include iterative eigen algorithm of [12], [13] and bundle adjustment techniques [14], [15] which performs reconstruction by minimizing the 2D reprojection error. However, bundle adjustment [14], being based on non-linear optimization algorithms, requires a good starting point to yield an acceptable solution. Simulation studies based on synthetic and real images sequences suggest that subspace-based methods are able to converge to a solution with reprojection errors close to that obtained by bundle adjustment, but do not require prior knowledge of a good initial solution. In particular, the method of [10] has the advantage of being independent of the coordinate system chosen for the image planes. However, there are two issues that have not been addressed in [10].

First, the algorithm of [10] is not guaranteed to converge. In the subspace method of [10], the iterative algorithm alternates between (i) performing an SVD of a measurement matrix to determine the best 4D subspace approximating that spanned by the image points scaled by some yet unknown depths, and (ii) determining the depths to re-scale the image points so that the subspace for each image is as close as possible to the 4D subspace obtained in (i). Both steps (i) and (ii) are posed as minimization problems. However, different measures are minimized in the two different steps as if they are distinct minimization problems. Because of this, the convergence of the algorithm cannot be established. Second, the method of [10] assumes that all object points are visible on all images. In practice, it is unlikely that this condition is satisfied due to occlusion. For the factorization method to be practically applicable, it is important that the method caters for object points, which are visible only on some of the images but are missing from the other images. Despite the merits of the subspace approach, we are not aware of any existing subspace-based factorization method that can handle missing points while ensuring convergence.

In this paper, we will use the subspace method of [10] as a basis for the factorization method. In order to establish convergence, we will formulate the factorization problem as a minimization problem with a single consistent objective function that is optimized for distinct purposes (with respect to different sets of parameters) throughout an iterative algorithm. Missing points will be estimated as part of the algorithm. With convergence of the algorithm in mind, the difficulty is that the estimation of the missing points must be performed in a way consistent with the measure being minimized in the solution of the factorization problem. A key point underlying our approach is that subspaces will always be represented by an orthonormal basis, and this condition is enforced as a constraint at all stages of optimization. The constraint that subspace basis should always be orthonormal may render a minimization problem non-linear. Necessary solutions to such problems will be developed in this paper.

The paper is organized as follows. A measure for subspace inclusion is introduced in Section 2 before we formulate the factorization problem. An algorithmic solution to the factorization problem incorporating missing point estimation is developed in Section 3. Experimental results using both synthetic data and real images are provided in Section 4 to illustrate the performance of the proposed method. Section 5 contains some concluding remarks.

We shall use the following notation: span(M) denotes the subspace spanned by the rows of a matrix M; ‖MF denotes the Frobenius norm of a matrix M.

Section snippets

The subspace method

Consider a set of 3D points Xj=[xjyjzj1]T (j=1,…,n) viewed by m cameras with projection matrices PiR3×4 (i=1,…,m). Let the projection of Xj on the ith view be the image point wij=[uijvij1]T (i=1,…,m; j=1,…,n) in normalized homogeneous coordinates. The 2D image points wij on the ith view can be assembled into an unscaled measurement matrix Wi (for the ith view) given by:Wi=[wi1wi2win]R3×n.

The image point wij is related to Xj byλijwij=PiXjwhere λij is the depth of the object point Xj seen by

Algorithmic solution

In view of the form of the minimization problem (15), it is natural to consider solving the variables of the three nested minimization problems by estimating X, λij and w˜ij (missing points only) successively in an iterative loop, giving rise to the following algorithm (where superscript k denotes the variables in the kth iteration). In the algorithm, each of X, λij and w˜ij is solved in turn as a free parameter of an optimization problem while the other two variables are fixed at their latest

Experimental results

In this section, the proposed method is firstly evaluated using synthetic data and compared with Sturm–Triggs' method [4] and Heyden's subspace method [10]. Then, we consider real images with missing data, in which case only the proposed method is evaluated as the methods of [4], [10] are not applicable.

Conclusion

In this paper, we have used the subspace approach to consider the factorization problem for projective reconstruction from multiple images. The novelty of our method lies in a formulation of the factorization problem in terms of the optimization of a single objective function that is minimized in a consistent manner with respect to three distinct sets of parameters for different purposes. As a result, we are able to estimate missing data as part of the algorithm while preserving guaranteed

Acknowledgement

The work described in this paper was supported by grants from the Research Grants Council of Hong Kong Special Administrative Region, China (Project Nos. HKU7058/02E and HKU7135/05E).

References (18)

  • W.K. Tang et al.

    A column-space approach to projective reconstruction

    Computer Vision and Image Understanding

    (2006)
  • A. Heyden et al.

    An iterative factorization method for projective structure and motion from image sequences

    Image and Vision Computing

    (1999)
  • G.Q. Chen et al.

    Practical algorithms for stratified structure-from-motion

    Image and Vision Computing

    (2002)
  • R.I. Hartley, Euclidean reconstruction from uncalibrated views, in J. Mundy, A. Zisserman (Eds.), Applications of...
  • O.D. Faugeras

    Stratification of three-dimensional vision: projective, affine, and metric representations

    Journal of the Optical Society of America-A

    (1995)
  • J. Oliensis et al.

    An experimental study of projective structure from motion

    IEEE Transactions Pattern Analysis and Machine Intelligence

    (1999)
  • P. Sturm et al.

    A factorization based algorithm for multi-image projective structure and motion

    European Conference on Computer Vision, Cambridge, UK

    (1996)
  • B. Triggs

    Factorization methods for projective structure and motion

    IEEE International Conference on Computer Vision and Pattern Recognition, IEEE, San Francisco

    (1996)
  • S. Mahamud et al.

    Provably-convergent iterative methods for projective structure from motion

    IEEE International Conference on Computer Vision and Pattern Recognition, Kauai, Hawaii

    (2001)
There are more references available in the full text version of this article.

Cited by (0)

View full text