Elsevier

Pattern Recognition

Volume 43, Issue 3, March 2010, Pages 545-556
Pattern Recognition

Projective reconstruction of ellipses from multiple images

https://doi.org/10.1016/j.patcog.2009.07.003Get rights and content

Abstract

This paper presents a new approach for reconstructing 3D ellipses (including circles) from a sequence of 2D images taken by uncalibrated cameras. Our strategy is to estimate an ellipse in 3D space by reconstructing N(5) 3D points (called representative points) on it, where the representative points are reconstructed by minimizing the distances from their projections to the measured 2D ellipses on different images (i.e., 2D reprojection error). This minimization problem is transformed into a sequence of minimization sub-problems that can be readily solved by an algorithm which is guaranteed to converge to a (local) minimum of the 2D reprojection error. Our method can reconstruct multiple 3D ellipses simultaneously from multiple images and it readily handles images with missing and/or partially occluded ellipses. The proposed method is evaluated using both synthetic and real data.

Introduction

In computer vision, the recovery of 3D scene from a sequence of images is a fundamental problem. Existing research mainly concentrates on 3D reconstruction of point features [1], [2], [3] and line segments [4], [5], [6]. However, an ellipse (a circle is included as a special case), is one of the most common features that have been used for 3D location estimation for the following reasons: (1) objects with circular and elliptical contours are frequently seen in the real world and (2) the image-location accuracy of an ellipse is higher than that of a point or a line segment [7], [8], which is very attractive for applications. For these reasons, ellipses have been used to solve various machine-vision-related problems. For example, they are used to accurately locate a mobile robot's position using circular landmarks [9], [10], to estimate the 3D orientation of objects with circular contours [11], and to recognize 3D pre-marked objects using circular markers [12].

Therefore, the 3D reconstruction of ellipses is of fundamental importance. Recently, the problem of pose estimation, stereo and motion estimation based on ellipses has attracted some attention [13], [14], [15], [16], [17], but there are still much fewer articles dealing with ellipses than those devoted to points and line segments. Although there are a few papers on conics reconstruction (of which ellipses can be considered as a special case), there is little work specifically on ellipse reconstruction. The 3D reconstruction of ellipses is therefore still an open problem and needs further exploring.

In [13], [14], algorithms are proposed to reconstruct the positions and orientations of conics in 3D space, where the cameras are assumed to be calibrated. By deriving two polynomial conditions, Long [15] proposes a unified closed-form mathematical solution for conic reconstruction from two uncalibrated images. The method is simple and stable, but it assumes that the pair of projection matrices is known. Instead of directly recovering the parametric description of a quadratic curve in 3D space as the above mentioned methods, Xie [16] emphasizes on the 3D reconstruction of points belonging to the 3D conics. By imposing a planarity constraint, 3D conics are reconstructed once the 3D points on them are known. The experiments show that the reconstruction precision is much better than direct 3D recovery of the conic parameters. This method assumes that the relative geometry between two perspective views is known, and thus the point correspondences on the curves are known.

Up to now, all the methods we know for 3D conic reconstruction share two limitations. The first one is that they require prior information of the cameras or the geometric relations between images, which is not always available in real applications. The second one is that the reconstructions are based on a pair of images only. It is known that 3D conic reconstruction over two views is ambiguous up to two solutions [13], [14], [15], defined by any one of the plane pair, as shown in Fig. 1.

For higher-order general algebraic curves, Kaminski et al. proposed algorithms to reconstruct general algebraic curves (both for planar [18] and [19] for non-planar cases) from multiple views, assuming known camera matrices. In [20], Berthillson et al. introduce the concept of affine shape, and an algorithm for reconstructing general curves from uncalibrated images is proposed in [21], which relies on aligning the parameterizations of matched curves so that the points traced on each curve become corresponding points. This algorithm works with any number of images taken by uncalibrated cameras, but a final step of bundle adjustment is required to refine the reconstruction, and it has difficulty to reconstruct multiple curves from multiple views with missing curves and/or partial occlusion of curves in some of the views.

To solve the existing problems mentioned above, we propose a point-based ellipse reconstruction method in this paper. The proposed method recovers an ellipse in 3D space by reconstructing N(5) 3D points (referred to as representative points) on it, by minimizing the geometric distance from the projections of the N representative points to the corresponding measured ellipses in images.

The proposed approach has the following advantages:

  • The method reconstructs the 3D ellipses over a sequence of images, which provides more information and help to obtain a unique solution.

  • No prior knowledge about the cameras or the relative geometry of the images is required.

  • The reconstruction is realized by iteratively minimizing a cost function which represents a geometric measure of the goodness of reconstruction without the need of further refinement by bundle adjustment.

  • The iterative optimization algorithm is guaranteed to converge and is robust to noise in images.

  • The method can recover multiple ellipses simultaneously and it readily handles missing and/or partially occluded ellipses in images.

  • The method recovers the projection matrices simultaneously with the 3D ellipses. By doing so, our algorithm achieves higher reconstruction accuracies, since the ellipse data also contributes to the estimation.

This paper is organized as follows. In Section 2, we briefly summarize the point-based projective reconstruction method of [2], which forms the basis of our proposed ellipse reconstruction method. In Section 3, the ellipse reconstruction problem is reformulated as a minimization problem. The proposed algorithm is presented in Section 4. Experimental results are given in Section 5 to demonstrate the performance of the proposed method. Concluding remarks are given in Section 6.

Section snippets

Projective reconstruction of 3D points

Suppose we are given m views of t 3D points. The l-th (l=1,2,,t) point is projected onto the i-th (i=1,2,,m) view and measured as xil=[uilvil1]T in homogeneous coordinates. Since it is allowed that some points be missing in some views, the available measurements are defined by an index set:B={(i,l)|thelthfeaturepointisobservedontheithview}.The goal of projective point reconstruction is to estimate the t 3D points from the available measurements on the images. Let us denote the l-th

Given data

As shown in Fig. 2, suppose we are given m views of a 3D scene composed of n ellipses and t feature points (not necessarily on the ellipses). It is known that an ellipse in 3D space projects to an ellipse on a 2D image [22]. Since the ellipses may not be visible in all the views, we define an index set for the measured 2D ellipses asA={(i,j)|thejthellipseisobservedontheithview}.The measured 2D ellipses will be denoted by cij((i,j)A) (the ellipses on the images in Fig. 2) and the 2D points by

Ellipse reconstruction

In this section, we will show how (5) can be reformulated as a multi-linear least-squares minimization problem that can be solved by means of an iterative procedure.

Experimental results

The proposed method is evaluated by both synthetic and real data. The algorithm is implemented using Matlab 7.0 running under Windows XP on a Pentium 4 2.40 GHz machine with 512 MB RAM.

Conclusions

In this paper, a new approach is developed for reconstructing 3D ellipses from m(3) images taken by uncalibrated cameras. We reconstruct a 3D ellipse (or a circle) by minimizing the distance of N(5) reprojected representative points on the ellipse (or circle) from the measured 2D ellipses (or circles) on different images, except for a reference ellipse where the distance is measured from some pre-defined representative points to their reprojections. By solving a sequence of weighted

Acknowledgments

The work in this paper was supported by the Research Grants Council of Hong Kong Special Administrative Region, China (GRF Projects HKU 712808E and HKU 711208E) and CRCG of the University of Hong Kong.

About the Author—F. MAI received her B.Eng. and M.Eng. degrees in Control Theory and Control Engineering in 2000 and 2003 respectively from Tianjin University, China, and her Ph.D. degree from the University of Hong Kong in 2008. She is now a research associate in the Department of Electrical and Electronic Engineering at the University of Hong Kong. Her research interests include feature extraction, feature matching and 3D reconstruction.

References (25)

  • H.C. Lee et al.

    3-D shape from contour and selective confirmation

    Computer Vision, Graphics, and Image Processing

    (1983)
  • P. Sturm, B. Triggs, A factorization based algorithm for multiple-image projective structure and motion, in:...
  • Y.S. Hung et al.

    Projective reconstruction from multiple views with minimization of 2D reprojection error

    International Journal of Computer Vision

    (2006)
  • C. Tomasi et al.

    Shape and motion from image streams under orthography: a factorization method

    International Journal of Computer Vision

    (1992)
  • R. I. Hartley, Projective reconstruction from line correspondences, in: Proceedings of IEEE Computer Society Conference...
  • R.I. Hartley

    Lines and points in three views and the trifocal tensor

    International Journal of Computer Vision

    (1997)
  • W.K. Tang et al.

    Projective reconstruction from line-correspondences in multiple uncalibrated images

    Pattern Recognition

    (2006)
  • E.S. Mcvey et al.

    Ranking of pattern for use in automation

    IEEE Transactions on Industrial Electronics and Control Instrumentation

    (1977)
  • R.A. Young, Locating industrial parts with subpixel accuracies, in: SPIE Proceedings of Optics, Illumination, Image...
  • M.J. Magee, J.K. Aggarwal, Determining the position of a robot using a single calibration object, in: Proceedings of...
  • M.R. Kabuka et al.

    Position verification of a mobile robot using standard pattern

    IEEE Journal of Robotics and Automation

    (1987)
  • R. Safaee-Rad, B. Benhabib, K.C. Smith, Z. Zhou, Pre-marking methods, for 3D object recognition, in: Proceedings of...
  • Cited by (0)

    About the Author—F. MAI received her B.Eng. and M.Eng. degrees in Control Theory and Control Engineering in 2000 and 2003 respectively from Tianjin University, China, and her Ph.D. degree from the University of Hong Kong in 2008. She is now a research associate in the Department of Electrical and Electronic Engineering at the University of Hong Kong. Her research interests include feature extraction, feature matching and 3D reconstruction.

    About the Author—Y.S. HUNG received his B.Sc. (Eng) degree in Electrical Engineering and B.Sc. degree in Mathematics, both from the University of Hong Kong, and his M.Phil. and Ph.D. degrees from the University of Cambridge. He has worked as a research associate at the University of Cambridge, and as a lecturer at the University of Surrey before he joined the University of Hong Kong in 1989, where he is now a professor and the head of the department. Dr Hung is a recipient of the Best Teaching Awards in 1991 from the Hong Kong University Students’ Union. He is a chartered engineer, a fellow of IEE and a senior member of IEEE. He has authored and co-authored over 100 publications in books, international journals and conferences.

    About the Author—G. CHESI received the Laurea degree in Information Engineering from the University of Firenze in 1997 and the Ph.D. in Systems Engineering from the University of Bologna in 2001. He was a visiting scientist at the Department of Engineering of the University of Cambridge (1999–2000) and Department of Information Physics and Computing of the University of Tokyo (2001–2004). He was with the Department of Information Engineering of the University of Siena (2000–2006) and he joined the Department of Electrical and Electronic Engineering of the University of Hong Kong in 2006. Dr. Chesi was the recipient of the Best Student Award of the Faculty of Engineering of the University of Firenze in 1997. He is an Associate Editor of Automatica, an Associate Editor of IEEE Transactions on Automatic Control, and a Guest Editor of the Special Issue on ‘Positive Polynomials in Control’ of IEEE Transactions on Automatic Control. His research interests include computer vision, nonlinear systems, robotics, robust control, and systems biology. He is author of the book “Homogeneous Polynomial Forms for Robustness Analysis of Uncertain Systems” (Springer, 2009).

    View full text