A novel 3D planar object reconstruction from multiple uncalibrated images using the plane-induced homographies
Introduction
In the physical world (especially the man-made world) planar surfaces such as walls, windows, table, roof, road, and terrace can be found in the indoor as well as the outdoor scenes. Our task is to reconstruct the 3D planar surfaces in a scene from multiple uncalibrated images taken by a camera placed at different viewpoints. In general, the methods for 3D projective or uncalibrated reconstruction (Mohr and Arbogast, 1991; Faugeras, 1992, Faugeras, 1993; Hartley et al., 1992, Hartley, 1994; Beardsley et al., 1997) are point-based. They estimate the fundamental matrix from a sufficient number of corresponding point pairs first, and then derive the epipole and the canonical geometric representation for projective views using the fundamental matrix. Then, for each pair of corresponding points, they use a triangulation technique or bundle adjustment technique to compute the 3D point coordinates in the projective space. Finally, for the determination of the uncalibrated planar scene structure (Luong and Faugeras, 1993; Sawhney, 1994; Criminisi and Zisserman, 1998; Irani et al., 1998; Szeliski and Torr, 1998; Fradkin et al., 1999; Johansson, 1999; Zelnik-Manor and Irani, 2000), the 3D points found are fitted by planes. However, it is desirable to derive the 3D planar scene structure in terms of plane features in the images directly, for these features are more reliable than the point or line features (Luong and Faugeras, 1993). The estimation of the 3D projective planar structure based on the projected plane feature information exclusively has not yet received much attention, although it is known that the corresponding projected plane regions in a pair of stereo images induce a homography. It is also known that homographies are useful to many other practical applications including:
- (a)
Fundamental matrix estimation or canonical projective geometry representation (Luong and Vieville, 1996; Luong and Faugeras, 1993).
- (b)
2D image mosaicing or view synthesis (Szeliski, 1996).
- (c)
Plane + parallax analysis (Irani et al., 1998; Criminisi and Zisserman, 1998; Sawhney, 1994).
- (d)
Planar motion estimation and ego-motion (Irani et al., 1997; Szeliski and Torr, 1998; Zelnik-Manor and Irani, 2000).
Recently, two methods have been proposed for the 3D projective reconstruction of planes and cameras. The first method assumes all planes are visible in all images and the second method assumes a reference plane is visible in all images (Rother et al., 2002, Rother, 2003). In practice, it is not realistic to have all planes or even one plane visible in all images unless a very large ground plane is available. When there is no reference plane visible in all images, the reconstruction problem cannot be formulated within a common projective space and the reconstruction results will be inevitably obtained in different projective spaces.
We shall recover the 3D scene planar structure from the uncalibrated images using the plane-induced homographies without assuming that all planes or one plane must be seen in all images. To obtain the homographies, we must locate the projected regions of planar surfaces in the images. There are methods for detecting regions corresponding to planar surfaces in the image (Sinclair and Blake, 1996; Hamid and Cipolla, 1997; Theiler and Chabbi, 1999). After the image regions of planar surfaces have been extracted, we use the Gabor filtering technique (Sun et al., 2002) to identify at least four point correspondences for every plane in the stereo images in order to obtain the initial value of the homography. Then we iteratively refine the homography based on a nonlinear minimization method given in (Szeliski, 1996). Next, we use two homographies to compute the epipole and to find the compatible projection equations in terms of the estimated homography and an assigned plane coefficient vector of a reference plane, together with the estimated epipole. With the projection equations thus derived we then prove that the 3D equation of any other plane visible in the stereo images can be computed with respect to the reference plane equation as long as its homography is determined. Finally, we merge or integrate all reconstructed plane equations found in individual projective spaces within a common space through the coordinate (or space) transformations. Again, each required coordinate transformation matrix is expressed by the homography and plane coefficient vector information of two planes visible in the involved image pairs. Fig. 1 shows the flow diagram of our method.
The remaining sections of the paper are organized as follows. Section 2 is the preliminaries and mathematical notations for the projective reconstruction. Section 3 shows how the 3D equations of all planar surfaces visible in the stereo images can be determined from their homographies. Section 4 presents the integration of the reconstruction results obtained in different projective spaces through the coordinate transformations. Section 5 shows the estimation of the plane-induced homographies and the related epipole. Section 6 reports the experimental results on both the synthetic and real images. Section 7 is the concluding remarks.
Section snippets
Preliminaries and mathematical notations for projective reconstruction
Consider any two consecutive images (Ii,Ij) in an image sequence for reconstructing the visible planar surfaces. Let Ri, be the extrinsic parameters and Mi be the 3 × 3 upper triangular intrinsic camera matrix of the ith camera. Then the coordinates of a 3D point and its 2D projection point in image Ii are related by a pinhole camera model (Hartley et al., 1992; Faugeras, 1993; Hartley and Zisserman, 2000):
To represent the point in the
Reconstruction of all visible planes from a given image pair
In the new projective space the projection equations become
Similarly, for any other plane ΠB visible in (Ii, Ij) the induced homography between the plane regions in image pair (Ii, Ij) is expressed bywith the plane equation of ΠB being .
Next, we shall prove the fact that the relation between plane coefficient vectors of planes ΠB and ΠA is determined once their homographies Aij and Bij are found. From above we have
Integration of planes reconstructed from different image pairs
Next, we consider the integration of reconstructed planes obtained from different image pairs (Ii, Ij) and (Ij, Ik), which contain the projections of two commonly visible planes. We shall use the plane-based coordinate transformation method for integrating the reconstruction results defined in different spaces.
Let the 4 × 4 coordinate transformation matrix Hijk, mapping the points in the projective space to the points in the projective space , be defined byThen, the
Computation of homographies
We need to estimate Aij from the image data associated with the planar surface ΠA. We shall use the region-based matching, instead of point-based matching, to find the homography. First of all, we use the Gabor filtering technique (Sun et al., 2002) to identify at least four point correspondences in order to obtain the initial solution of the homography. We then use the Levenberg–Marquardt iterative nonlinear minimization algorithm (Szeliski, 1996) to minimize the sum of the squared intensity
Experiment 1
In the first experiment we use a synthetic tower whose feature points and schematic diagram are given in Table 1 and Fig. 2. We take a sequence of six pictures to cover all aspects of the tower using a virtual camera looking down from the upper positions. The image resolution is 640 × 480 in pixel. Three consecutive images of the sequence, I1, I2, and I3, are shown in Fig. 3. We apply the reconstruction process to this data set. We employ a linear least-squares method based on eight corresponding
Conclusions
An uncalibrated planar object reconstruction method has been described in which we rely on the plane information. We first estimate the homography for all planar surfaces using the region features of planar surfaces, and then we use the homographies induced by two planes to compute the epipole. We represent explicitly the compatible projection equations for the stereo images using the information of planar homographies and an assigned reference plane coefficient vector. We continue to derive
References (29)
- et al.
Active intrinsic calibration using vanishing points
Pattern Recogn. Lett.
(1996) - et al.
Canonic representations for the geometries of multiple projective views
Comput. Vis. Image Und.
(1996) - et al.
It can be done without camera calibration
Pattern Recogn. Lett.
(1991) - et al.
Sequential updating of projective and affine structure from motion
Int. J. of Comput. Vis.
(1997) - Criminisi, R., Zisserman, A., 1998. Duality, rigidity and planar parallax. In: Proceedings of European Conference on...
- Faugeras, O.D., 1992. What can be seen in three dimensions with an uncalibrated stereo rig? In: Proceedings of European...
Three-Dimensional Computer Vision: A Geometric Viewpoint
(1993)- Fitzgibbon, A.W., Zisserman, A., 1998. Automatic camera recovery for closed or open image sequences. In: Proceedings of...
- Fradkin, M., Roux, M., Maitre, H., Lelogu, U.M., 1999. Surface reconstruction from multiple aerial images in dense...
- Hartley, R.I., Gupta, R., Chang, T., 1992. Stereo from uncalibrated cameras. In: Proceedings of International...
Projective reconstruction and invariants from multiple images
IEEE Trans. Pattern Anal. Mach. Intell.
In defense of the eight-point algorithm
IEEE Trans. Pattern Anal. Mach. Intell.
Multiple View Geometry in Computer Vision
Cited by (2)
Evolutionary algorithms for a mixed stereovision uncalibrated 3D reconstruction
2015, Multimedia Tools and ApplicationsA zernike moment phase-based descriptor for local image representation and matching
2010, IEEE Transactions on Image Processing