Robust and efficient object reconstructions from closed loop sequences

Abstract

We propose a new hierarchical structure from motion system for close loop sequences. Our system includes a novel approach for clustering cameras into multiple sets, whose camera poses are initially reconstructed separately and later globally registered w.r.t a single coordinate frame. Each of the multiple sets is robustly reconstructed by a novel guarded least median square protocol. Our method is accelerated by reducing the parameter space of bundle adjustment in the local reconstruction optimizations. We also propose a new synthetic dataset that could be useful in 3D object reconstruction problems. Extensive experiments with both synthetic and real data were carried out to validate our method. Our system presented better results than ACTS and GPE in terms of rotation and translation errors in the camera pose estimations, and the accuracy is quite close to COLMAP while our method is much faster.

A Comparing against ground truth

Let $\mathbf {G}$ be the ground truth camera network poses $\mathbf {G}=\left[ {{I},\bar{{H}}_{2}\bar{{H}}_{3},\cdots ,\bar{{H}}_{{n}}}\right]$ of a dataset, and $\mathbf {Q}$ be the estimated camera network poses $\mathbf {Q}={\left[ {I},\hat{{H}}_{2},\hat{{H}}_{3},\cdots ,\hat{{H}}_{{n}}\right] }$ from the same dataset. Here, I is $4\times 4$ identity matrix, ${H}_{k}$ is $4\times 4$ homogeneous matrix for kth camera pose. It is straightforward to notice that $\mathbf {G}$ is equal to $\mathbf {Q}$ up to similarity transform. That is,

$$\begin{aligned} \mathbf {G}={H_{s}}\cdot \mathbf {Q}. \end{aligned}$$

(10)

Therefore, we need to find a similarity transformation ${H_{s}}$ that minimizes

$$\begin{aligned} \underset{H_{s}}{\mathrm{argmin}}\left\| \mathbf {G}-({H_{s}}\mathbf {\cdot Q})\right\| ^{2}= & {} 0. \end{aligned}$$

(11)

An approximate solution to (11) is under relaxed orthonormality and determinant constraints of rotation parts of the camera matrices, computed by its least square sense approximation—i.e. multiplying $\mathbf {G}$ with pseudo inverse of $\mathbf {Q}$. Let us consider the linear least square approximation of (11) is ${H_{s,\mathrm{init}}}$.

Then, we let ${\bar{{H}}}_{k}=\{\bar{{q}}_{k},{\bar{{t}}}{}_{k}\}$ and ${{H}_{{s,\mathrm{init}}}}\cdot \hat{{H}}_{k}\equiv \{{q}_{k},{t}_{k}\}$, where ${\bar{{t}}}_{k}$ and ${t}_{k}$ are translation vectors of ${\bar{{H}}}_{k}$ and ${{H}_{s}}\hat{{H}_{k}}$, respectively, and ${\bar{{q}}}_{k}$ and ${q}_{k}$ are unit quaternion vectors representing the orientations of ${\bar{{H}}}_{k}$ and ${{H}_{s}}\hat{{H}_{k}}$, respectively. The nonlinear optimization of ${H}_{s}$ finalizes the alignment of two camera networks: $\mathbf {G}$ and ${{H}_{{s,\mathrm{init}}}}\cdot \mathbf {Q}$ . This optimization is done by

$$\begin{aligned} \underset{H_{s}}{\mathrm{argmin}}\,\sum (\left\| {t}_{k}-{\bar{t}}_{k}\right\| _{2}+\min (\left\| {q}_{k}-{\bar{{q}}}_{k}\right\| _{2},\,\left\| {q}_{k}+\bar{{q}}_{k}\right\| _{2})), \end{aligned}$$

(12)

We employed the Levenberg–Marquardt algorithm to minimize the translation and orientation residuals simultaneously. This procedure optimizes the similarity transform that aligns two camera networks on top of each other. Then, the camera poses are compared individually in terms of their orientations and locations.