Accurate structure from motion using consistent cluster merging

Abstract

The incremental Structure-from-Motion approach is widely used for scene reconstruction as it is robust to outliers. However, the method suffers from two major limitations: error accumulation and heavy time consumption. To alleviate these problems, we propose a redundant cluster merge approach which is effective and efficient. Different from the previous clustering methods, each cluster has only one overlapping adjacent cluster. each of the sub clusters divided by our approach has several adjacent cluster candidates with overlapping. In addition, these cluster candidates are verified whether they are suitable for merging. By selecting the correct estimated clusters, cluster merging achieves more accurate results. The cluster verification is implemented based on the fact that the correctly estimated clusters have consistent point cloud and extrinsic camera parameters in each image of the same scene will be formulated as two constraints. In addition, we introduce a feature matching consistency constraint to eliminate the falsely matched feature pairs. The gain in accuracy of feature matching leads to better estimated results in each cluster. Experiments were performed on three public datasets. The reconstruction results show that our method outperformed state-of-the-art SfM approaches in terms of both efficiency and accuracy.

Appendix A: Coordinate transformation

1.1 A.1 Method 1: Coordinate transform between two clusters

Given two adjacent clusters with the same images, the images in the first cluster and in the second cluster are set as ${I_{f}^{k}}$ and ${I_{s}^{k}}$, respectively. The estimated 3D points from the first cluster transformed into the image ${I_{f}^{k}}$ local coordinate system satisfy

$$ {\textbf{X}}_{cam{\text{1}}} = {\textbf{R}}_{1} {\textbf{X}} + {\textbf{t}}_{1} , $$

(7)

where X is the 3D global coordinate of a point obtained in the first cluster, and R₁,t₁ are the estimated relative extrinsic camera parameters corresponding to image ${I_{f}^{k}}$. X_cam1 is the transformed coordinate in the image ${I_{f}^{k}}$ local coordinate system. Equation (7) can be rewritten as

$$ {\textbf{X}} = {\textbf{R}}_{1}^{- 1} ({\textbf{X}}_{cam{\text{1}}} - {\textbf{t}}_{1} ) = {\textbf{R}}_{1}^{- 1} {\textbf{X}}_{cam{\text{1}}} - {\textbf{R}}_{1}^{- 1} {\textbf{t}}_{1}. $$

(8)

Assuming the local coordinate system of image ${I_{s}^{k}}$ is set as the global coordinate system in the second cluster, the 3D coordinates of points estimated in the second cluster are denoted as X₂. Since image ${I_{f}^{k}}$ and image ${I_{s}^{k}}$ are the same image, therefore, X₂ = X_cam1. From equation (8), we obtain

$$ {\textbf{X}} = {\textbf{R}}_{1}^{- 1} {\textbf{X}}_{2} - {\textbf{R}}_{1}^{- 1} {\textbf{t}}_{1} . $$

(9)

Given one image in the second cluster, ${I_{s}^{i}}$, we explain how to estimate the parameters R,t that transforms the extrinsic camera parameters corresponding to ${I_{s}^{i}}$ in the second cluster coordinate system into the first cluster coordinate system as follows:

The 3D global coordinates of a point obtained in the first cluster, X, and the 3D coordinate of the point transformed into ${I_{s}^{i}}$ local coordinate system, X_cam2, satisfy

$$ {\textbf{X}}_{cam2} {\text{ = }}{\textbf{RX}} + {\textbf{t}}, $$

(10)

where X is the 3D global coordinate of the point defined as (A.3), and R,t are the extrinsic camera parameters corresponding to image ${I_{s}^{i}}$ in the first cluster coordinate system.

With the substitution of (9), Formula (10) can be rewritten as

$$ {\textbf{X}}_{cam2} = {\textbf{R}}({\textbf{R}}_{1}^{- 1} {\textbf{X}}_{\text{2}} - {\textbf{R}}_{1}^{- 1} {\textbf{t}}_{1} ) + {\textbf{t}} = {\textbf{RR}}_{1}^{- 1} {\textbf{X}}_{\text{2}} - {\textbf{RR}}_{1}^{- 1} {\textbf{t}}_{1} + {\textbf{t}}. $$

(11)

The 3D coordinate of this point transformed into ${I_{s}^{i}}$ local coordinate system can also be defined as

$$ {\textbf{X}}_{cam2} = {\textbf{R}}_{2} {\textbf{X}}_{2} + {\textbf{t}}_{2} , $$

(12)

where R₂,t₂ are the estimated extrinsic camera parameters corresponding to ${I_{s}^{i}}$ in the second cluster, and X₂ is the estimated 3D global coordinate of the point in the second cluster.

Since the coordinates of the point transformed into the ${I_{s}^{i}}$ local coordinate system are unchanged, (12) equals to (11), and can be formulated as

$$ {\textbf{RR}}_{1}^{- 1} {\textbf{X}}_{\text{2}} - {\textbf{RR}}_{1}^{- 1} {\textbf{t}}_{1} + {\textbf{t}} = {\textbf{R}}_{2} {\textbf{X}}_{2} + {\textbf{t}}_{2} . $$

(13)

From (13), we obtain

$$ {\textbf{RR}}_{1}^{- 1} = {\textbf{R}}_{2} \Rightarrow {\textbf{R}} = {\textbf{R}}_{2} {\textbf{R}}_{1}. $$

(14)

$$ - {\textbf{RR}}_{1}^{- 1} {\textbf{t}}_{1} + {\textbf{t}} = {\textbf{t}}_{2} \Rightarrow {\textbf{t}} = {\textbf{RR}}_{1}^{- 1} {\textbf{t}}_{1} + {\textbf{t}}_{2} = {\textbf{R}}_{2} {\textbf{t}}_{1} + {\textbf{t}}_{2}. $$

(15)

1.2 A.2 Method 2: Coordinate transformation in the new global coordinate system

According to the previous derivation, if the ${I_{f}^{k}}$ local coordinate system is set to the global coordinate system in the first cluster, we have R₁ = I; t₁ = 0. From equations (14) and (15), R and t are calculated as R = R₂; t = t₂, which means that the estimated extrinsic parameters of a camera in the second cluster can be estimated as the extrinsic parameters of this camera in the first cluster global coordinate system if the global coordinate system in each cluster is set as the same image local coordinate system.

Given an image in a cluster, I_g, we describe how the estimated extrinsic parameters of one camera in the cluster are transformed into the coordinate system with the local coordinate system of I_g as the cluster global coordinate system as follows.

We denote the extrinsic camera parameters corresponding to image I_g as R_g and t_g. The estimated coordinates of a point, X, transformed into the I_g image local coordinate system as X_cam, can be formulated as:

$$ {\textbf{X}}_{cam} = {\textbf{R}}_{g} {\textbf{X}} + {\textbf{t}}_{g}. $$

(16)

Let R,t be the extrinsic camera parameters of the N^th image after transformation. Since the local coordinate system of image I_g is set as the global coordinate system after transformation, after transformation, the coordinate of the point in the N^th image local coordinate system is subject to

$$ {\textbf{RX}}_{cam} + {\textbf{t}} = {\textbf{R}}\left( {{\textbf{R}}_{g} {\textbf{X}} + {\textbf{t}}_{g} } \right) + {\textbf{t}} = {\textbf{RR}}_{g} {\textbf{X}}{\text{ + }}{\textbf{Rt}}_{g} + {\textbf{t}}. $$

(17)

Since the coordinate of the point transformed into the N^th image local coordinate system is unchanged, therefore, we have

$$ {\textbf{RR}}_{g} {\textbf{X}}{\text{ + }}{\textbf{Rt}}_{g} + {\textbf{t}} = {\textbf{R}}_{n} {\textbf{X}} + {\textbf{t}}_{n} , $$

(18)

where R_n and t_n are the estimated extrinsic camera parameters corresponding to the N^th image before transformation.

According to (18), we have

$$ {\textbf{RR}}_{g} = {\textbf{R}}_{n} ,{\textbf{Rt}}_{g} + {\textbf{t}} = {\textbf{t}}_{n}. $$

(19)

From (19), we conclude that the extrinsic camera parameters correspond to the N^th image transformed into the coordinate system with the local coordinate system of I_g as the cluster global coordinate system satisfy

$$ {\textbf{R}} = {\textbf{R}}_{n} {\textbf{R}}_{g}^{- 1} ,{\textbf{t}} = {\textbf{t}}_{n} - {\textbf{Rt}}_{g}. $$

(20)

Thus, the coordinates of a point are transformed into the coordinate in the new global coordinate system are estimated as

$$ {\textbf{X}}^{\prime} = {\textbf{R}}_{g} {\textbf{X}} + {\textbf{t}}_{g}. $$

(21)