Bayesian perspective-plane (BPP) with maximum likelihood searching for visual localization

Abstract

The proposed “Perspective-Plane” in this paper is similar to the well-known “Perspective-n-Point (PnP)” or “Perspective-n-Line (PnL)” problems in computer vision. However, it has broader applications and potentials, because planar scenes are more widely available than control points or lines in daily life. We address this problem in the Bayesian framework and propose the “Bayesian Perspective-Plane (BPP)” algorithm, which can deal with more generalized constraints rather than type-specific ones. The BPP algorithm consists of three steps: 1) plane normal computation by maximum likelihood searching from Bayesian formulation; 2) plane distance computation; and 3) visual localization. In the first step, computation of the plane normal is formulated within the Bayesian framework, and is solved by using the proposed Maximum Likelihood Searching Model (MLS-M). Two searching modes of 2D and 1D are discussed. MLS-M can incorporate generalized planar and out-of-plane deterministic constraints. With the computed normal, the plane distance is recovered from a reference length or distance. The positions of the object or the camera can be determined afterwards. Extensions of the proposed BPP algorithm to deal with un-calibrated images and for camera calibration are discussed. The BPP algorithm has been tested with both simulation and real image data. In the experiments, the algorithm was applied to recover planar structure and localize objects by using different types of constraints. The 2D and 1D searching modes were illustrated for plane normal computation. The results demonstrate that the algorithm is accurate and generalized for object localization. Extensions of the proposed model for camera calibration were also illustrated in the experiment. The potential of the proposed algorithm was further demonstrated to solve the classic Perspective-Three-Point (P3P) problem and classify the solutions in the experiment. The proposed BPP algorithm suggests a practical and effective approach for visual localization.

Appendixes

• A. Uniformly sampling on the circle in 3D space

  We can uniform sample on the circle in Eq. (19) by first sampling on a standard circle, and then mapping the sampled points via a rotation transform. This is implemented by following the three steps:

  • Step 1 Uniformly sample the standard circle, which has the following equation:

    $$ \left\{\begin{array}{l}{X}^2+{Y}^2=1\\ {}Z=0\end{array}\right. $$

    (29)

    This can be done by uniformly sampling within the angle space, and a sampled point is represented by ${\left[\begin{array}{ccc}\hfill \cos \left({\theta}_i\right)\hfill & \hfill \sin \left({\theta}_i\right)\hfill & \hfill 0\hfill \end{array}\right]}^T$, with θ _i  ∈ [0 2π).

  • Step 2 Compute the rotation transform. The two circles defined by Eq. (19) and (29) are mapped via a rotation matrix, which satisfies

    $$ R{\left[\begin{array}{ccc}\hfill 0\hfill & \hfill 0\hfill & \hfill 1\hfill \end{array}\right]}^T=N\kern0.5em $$

    (30)

    The rotation matrix is not uniquely determined, because a circle is invariant to the rotation around the normal. We practically can use two arbitrary orthogonal vectors that satisfy Eq. (3), as the first and second column vectors.

  • Step 3 Transform the sampled points by rotation. Each sampled normal on the standard circle is transformed with the rotation matrix

    $$ R\left[\begin{array}{c}\hfill \cos \left({\theta}_i\right)\hfill \\ {}\hfill \sin \left({\theta}_i\right)\hfill \\ {}\hfill 0\hfill \end{array}\right]=\left[\begin{array}{c}\hfill {r}_1 \cos \left({\theta}_i\right)+{r}_4 \sin \left({\theta}_i\right)\hfill \\ {}\hfill {r}_2 \cos \left({\theta}_i\right)+{r}_5 \sin \left({\theta}_i\right)\hfill \\ {}\hfill {r}_3 \cos \left({\theta}_i\right)+{r}_6 \sin \left({\theta}_i\right)\hfill \end{array}\right] $$

    (31)

    Hence, a uniform sampling on the circle from Eq. (19) is accomplished. We can prove that the angles between two neighboring sampled points before and after transformation are identical, because

    $$ {\left(R\left[\begin{array}{c}\hfill \cos \left({\theta}_{\mathrm{i}}\right)\hfill \\ {}\hfill \sin \left({\theta}_{\mathrm{i}}\right)\hfill \\ {}\hfill 0\hfill \end{array}\right]\right)}^T\left(R\left[\begin{array}{c}\hfill \cos \left({\theta}_{\mathrm{i}\hbox{-} 1}\right)\hfill \\ {}\hfill \sin \left({\theta}_{\mathrm{i}\hbox{-} 1}\right)\hfill \\ {}\hfill 0\hfill \end{array}\right]\right)={\left[\begin{array}{c}\hfill \cos \left({\theta}_{\mathrm{i}}\right)\hfill \\ {}\hfill \sin \left({\theta}_{\mathrm{i}}\right)\hfill \\ {}\hfill 0\hfill \end{array}\right]}^T\left[\begin{array}{c}\hfill \cos \left({\theta}_{\mathrm{i}\hbox{-} 1}\right)\hfill \\ {}\hfill \sin \left({\theta}_{\mathrm{i}\hbox{-} 1}\right)\hfill \\ {}\hfill 0\hfill \end{array}\right] $$

    (32)

• B. Define the searching space for focal length

  It is difficult to define the searching space of the focal length directly, since images are in different resolutions in practice. Instead, we define the searching space from the camera viewing angles. There are three view angles defined for a camera: horizontal, vertical, and diagonal. In this paper, we use the horizontal one. It is defined as

  $$ {\theta}_h=2{ \tan}^{-1}\left(\frac{h}{2f}\right) $$

  (33)

  where h is the horizontal resolution of the image, and f is the camera focal length. As a result, we can estimate the focal length from the view angle and the image resolution:

  $$ f=\frac{h}{2 \tan \left(\frac{\theta_h}{2}\right)} $$

  (34)

  Usually, we have some prior knowledge of the common used lens. We can thus define the searching space of the focal length from the range of the view angle and the image resolution.

• C. Lemma: Determining the support plane is the necessary and sufficient condition to solve the P3P problem

  • 1. Proof: 1) Necessary condition

    The distances of the three control points to the camera are known from a solved P3P. Hence, we can determine the 3D coordinates of each control points by the following equations

    $$ \left\{\begin{array}{l}{X}_i=\lambda {K}^{-1}{x}_i\\ {}\left\Vert {X}_i\right\Vert ={d}_i\end{array}\right. $$

    (35)

    With the recovered control points, the support plane is uniquely determined then.

  • 1. 2) Sufficient condition

    The plane normal and distance are known from a determined support plane. As a result, we can use Eq. (4) to compute the 3D positions of each control points. The distances of the three points are computed readily from the 3D coordinates. Hence, the P3P problem is solved.