Elsevier

Pattern Recognition Letters

Volume 25, Issue 12, September 2004, Pages 1399-1410
Pattern Recognition Letters

A novel 3D planar object reconstruction from multiple uncalibrated images using the plane-induced homographies

https://doi.org/10.1016/j.patrec.2004.05.018Get rights and content

Abstract

A computer vision method is proposed to determine all the visible 3D planar surfaces in a scene from uncalibrated images and locate them in a single 3D projective space. Most of the existing methods for reconstructing planar objects use point correspondences to estimate the fundamental matrix and derive the compatible projection equations, before they apply the standard triangulation technique to find the 3D points and fit the planes to the 3D points. This type of approaches is generally slow and less accurate because the 3D points are estimated separately, making them vulnerable to image error. We present a plane based reconstruction method to estimate the 3D projective structure using the planar homographies estimated from the plane features in the images. First, we estimate the homography for each visible plane, and then we use the homographies of two primary planes to compute an epipole. We proceed to represent the epipolar geometry for each image pair using the estimated homography and epipole, together with a specified reference plane coefficient vector. Next, we show that the 3D plane coefficient vector of any plane visible in each image pair can be determined with respect to the reference plane coefficient vector once its planar homography is found. Finally, the reconstruction results obtained in individual projective spaces are integrated within a common projective space. To this end, we use the homography and plane equation information of two planes and the epipole associated to derive the coordinate transformation matrix between two involved projective spaces. To evaluate the performance of our method, we apply our method to the synthetic images and real images. All the results indicate the method works successfully.

Introduction

In the physical world (especially the man-made world) planar surfaces such as walls, windows, table, roof, road, and terrace can be found in the indoor as well as the outdoor scenes. Our task is to reconstruct the 3D planar surfaces in a scene from multiple uncalibrated images taken by a camera placed at different viewpoints. In general, the methods for 3D projective or uncalibrated reconstruction (Mohr and Arbogast, 1991; Faugeras, 1992, Faugeras, 1993; Hartley et al., 1992, Hartley, 1994; Beardsley et al., 1997) are point-based. They estimate the fundamental matrix from a sufficient number of corresponding point pairs first, and then derive the epipole and the canonical geometric representation for projective views using the fundamental matrix. Then, for each pair of corresponding points, they use a triangulation technique or bundle adjustment technique to compute the 3D point coordinates in the projective space. Finally, for the determination of the uncalibrated planar scene structure (Luong and Faugeras, 1993; Sawhney, 1994; Criminisi and Zisserman, 1998; Irani et al., 1998; Szeliski and Torr, 1998; Fradkin et al., 1999; Johansson, 1999; Zelnik-Manor and Irani, 2000), the 3D points found are fitted by planes. However, it is desirable to derive the 3D planar scene structure in terms of plane features in the images directly, for these features are more reliable than the point or line features (Luong and Faugeras, 1993). The estimation of the 3D projective planar structure based on the projected plane feature information exclusively has not yet received much attention, although it is known that the corresponding projected plane regions in a pair of stereo images induce a homography. It is also known that homographies are useful to many other practical applications including:

  • (a)

    Fundamental matrix estimation or canonical projective geometry representation (Luong and Vieville, 1996; Luong and Faugeras, 1993).

  • (b)

    2D image mosaicing or view synthesis (Szeliski, 1996).

  • (c)

    Plane + parallax analysis (Irani et al., 1998; Criminisi and Zisserman, 1998; Sawhney, 1994).

  • (d)

    Planar motion estimation and ego-motion (Irani et al., 1997; Szeliski and Torr, 1998; Zelnik-Manor and Irani, 2000).


Recently, two methods have been proposed for the 3D projective reconstruction of planes and cameras. The first method assumes all planes are visible in all images and the second method assumes a reference plane is visible in all images (Rother et al., 2002, Rother, 2003). In practice, it is not realistic to have all planes or even one plane visible in all images unless a very large ground plane is available. When there is no reference plane visible in all images, the reconstruction problem cannot be formulated within a common projective space and the reconstruction results will be inevitably obtained in different projective spaces.

We shall recover the 3D scene planar structure from the uncalibrated images using the plane-induced homographies without assuming that all planes or one plane must be seen in all images. To obtain the homographies, we must locate the projected regions of planar surfaces in the images. There are methods for detecting regions corresponding to planar surfaces in the image (Sinclair and Blake, 1996; Hamid and Cipolla, 1997; Theiler and Chabbi, 1999). After the image regions of planar surfaces have been extracted, we use the Gabor filtering technique (Sun et al., 2002) to identify at least four point correspondences for every plane in the stereo images in order to obtain the initial value of the homography. Then we iteratively refine the homography based on a nonlinear minimization method given in (Szeliski, 1996). Next, we use two homographies to compute the epipole and to find the compatible projection equations in terms of the estimated homography and an assigned plane coefficient vector of a reference plane, together with the estimated epipole. With the projection equations thus derived we then prove that the 3D equation of any other plane visible in the stereo images can be computed with respect to the reference plane equation as long as its homography is determined. Finally, we merge or integrate all reconstructed plane equations found in individual projective spaces within a common space through the coordinate (or space) transformations. Again, each required coordinate transformation matrix is expressed by the homography and plane coefficient vector information of two planes visible in the involved image pairs. Fig. 1 shows the flow diagram of our method.

The remaining sections of the paper are organized as follows. Section 2 is the preliminaries and mathematical notations for the projective reconstruction. Section 3 shows how the 3D equations of all planar surfaces visible in the stereo images can be determined from their homographies. Section 4 presents the integration of the reconstruction results obtained in different projective spaces through the coordinate transformations. Section 5 shows the estimation of the plane-induced homographies and the related epipole. Section 6 reports the experimental results on both the synthetic and real images. Section 7 is the concluding remarks.

Section snippets

Preliminaries and mathematical notations for projective reconstruction

Consider any two consecutive images (Ii,Ij) in an image sequence for reconstructing the visible planar surfaces. Let Ri, ti be the extrinsic parameters and Mi be the 3 × 3 upper triangular intrinsic camera matrix of the ith camera. Then the coordinates of a 3D point pE=[xEyEzE]T and its 2D projection point ui=[uivi]T in image Ii are related by a pinhole camera model (Hartley et al., 1992; Faugeras, 1993; Hartley and Zisserman, 2000):uivi1Mi[Ri|ti]xEyEzE1.

To represent the point in the

Reconstruction of all visible planes from a given image pair

In the new projective space the projection equations become

Similarly, for any other plane ΠB visible in (Ii, Ij) the induced homography between the plane regions in image pair (Ii, Ij) is expressed byBijBMjRjMi−1−λeẽjbETMi−1with the plane equation of ΠB being bETpE+1=0.

Next, we shall prove the fact that the relation between plane coefficient vectors of planes ΠB and ΠA is determined once their homographies Aij and Bij are found. From above we haveMjRjMi−1=1λAAijeẽjaETMi−1=1λBBijeẽjb

Integration of planes reconstructed from different image pairs

Next, we consider the integration of reconstructed planes obtained from different image pairs (Ii, Ij) and (Ij, Ik), which contain the projections of two commonly visible planes. We shall use the plane-based coordinate transformation method for integrating the reconstruction results defined in different spaces.

Let the 4 × 4 coordinate transformation matrix Hijk, mapping the points in the projective space {p̃ij} to the points in the projective space {p̃jk}, be defined byp̃jk=Hijkp̃ij.Then, the

Computation of homographies

We need to estimate Aij from the image data associated with the planar surface ΠA. We shall use the region-based matching, instead of point-based matching, to find the homography. First of all, we use the Gabor filtering technique (Sun et al., 2002) to identify at least four point correspondences in order to obtain the initial solution of the homography. We then use the Levenberg–Marquardt iterative nonlinear minimization algorithm (Szeliski, 1996) to minimize the sum of the squared intensity

Experiment 1

In the first experiment we use a synthetic tower whose feature points and schematic diagram are given in Table 1 and Fig. 2. We take a sequence of six pictures to cover all aspects of the tower using a virtual camera looking down from the upper positions. The image resolution is 640 × 480 in pixel. Three consecutive images of the sequence, I1, I2, and I3, are shown in Fig. 3. We apply the reconstruction process to this data set. We employ a linear least-squares method based on eight corresponding

Conclusions

An uncalibrated planar object reconstruction method has been described in which we rely on the plane information. We first estimate the homography for all planar surfaces using the region features of planar surfaces, and then we use the homographies induced by two planes to compute the epipole. We represent explicitly the compatible projection equations for the stereo images using the information of planar homographies and an assigned reference plane coefficient vector. We continue to derive

References (29)

  • K. Daniilidis et al.

    Active intrinsic calibration using vanishing points

    Pattern Recogn. Lett.

    (1996)
  • Q.T. Luong et al.

    Canonic representations for the geometries of multiple projective views

    Comput. Vis. Image Und.

    (1996)
  • R. Mohr et al.

    It can be done without camera calibration

    Pattern Recogn. Lett.

    (1991)
  • P.A. Beardsley et al.

    Sequential updating of projective and affine structure from motion

    Int. J. of Comput. Vis.

    (1997)
  • Criminisi, R., Zisserman, A., 1998. Duality, rigidity and planar parallax. In: Proceedings of European Conference on...
  • Faugeras, O.D., 1992. What can be seen in three dimensions with an uncalibrated stereo rig? In: Proceedings of European...
  • O.D. Faugeras

    Three-Dimensional Computer Vision: A Geometric Viewpoint

    (1993)
  • Fitzgibbon, A.W., Zisserman, A., 1998. Automatic camera recovery for closed or open image sequences. In: Proceedings of...
  • Fradkin, M., Roux, M., Maitre, H., Lelogu, U.M., 1999. Surface reconstruction from multiple aerial images in dense...
  • Hartley, R.I., Gupta, R., Chang, T., 1992. Stereo from uncalibrated cameras. In: Proceedings of International...
  • R.I. Hartley

    Projective reconstruction and invariants from multiple images

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1994)
  • R.I. Hartley

    In defense of the eight-point algorithm

    IEEE Trans. Pattern Anal. Mach. Intell.

    (1997)
  • R.I. Hartley et al.

    Multiple View Geometry in Computer Vision

    (2000)
  • Hamid, N.H.G., Cipolla, R., 1997. Identifying planar regions in a scene using uncalibrated stereo vision. In:...
  • View full text