Refractive Pose Refinement

Abstract

In this paper, we investigate absolute and relative pose estimation under refraction, which are essential problems for refractive structure from motion. To cope with refraction effects, we first formulate geometric constraints for establishing iterative algorithms to optimize absolute and relative pose. By classifying two scenarios according to the geometric relationship between the camera and refractive interface, we derive the corresponding solutions to solve the optimization problems efficiently. In the scenario where the geometry between the camera and refractive interface is fixed (e.g., underwater imaging), we also show that the refractive epipolar constraint for relative pose can be established as a summation of the classical essential matrix and two correction terms caused by refraction by using the virtual camera transformation. Thanks to its succinct form, the resulting refractive epipolar constraint can be efficiently optimized. We evaluate our proposed algorithms on synthetic data showing superior accuracy and computational efficiency compared to state-of-the-art (SOTA) methods. We further demonstrate the application of the proposed algorithms in refractive structure from motion on real data. Our datasets (Hu et al., RefractiveSfM, https://github.com/diku-dk/RefractiveSfM, 2022) and code (Hu et al., DIKU Refractive Scenes Dataset 2022, Data, 2022) are publicly available.

Appendices

A Jacobian Matrices related to Pose Estimation under Refraction

This appendix section gives the Jacobian matrices derived for pose estimation under refraction.

1.1 A.1 Scenario 1

1.1.1 A.1.1 Relative Pose Refiment

The residual of the epipolar cost and its analytical Jacobian are given as follows:

$$\begin{aligned}{} & {} {\mathcal E}= {\textbf{u}^\prime }^\top \textbf{R}^\top (\textbf{e})^\wedge \textbf{u}\ , \nonumber \\{} & {} \textbf{J}{{\mathcal E}}_\textbf{x} =\left[ \textbf{J}{{\mathcal E}}_\textbf{R}\quad ,\textbf{J}{{\mathcal E}}_\textbf{c}\quad ,\textbf{0}\right] \ , \nonumber \\{} & {} \textbf{J}{{\mathcal E}}_\textbf{R}={\textbf{u}\prime }^\top \textbf{R}^\top \textbf{u}^\wedge \textbf{R}\textbf{c}_{\text {V}_2}^\wedge -{\textbf{u}^\prime }^\top \left( \textbf{R}^\top \textbf{u}^\wedge \textbf{e} \right) ^\wedge \ , \nonumber \\{} & {} \textbf{J}{{\mathcal E}}_\textbf{c} =-{\textbf{u}^\prime }^\top \textbf{R}^\top \textbf{u}^\wedge \ , \end{aligned}$$

(A.1)

where $\textbf{e}=\textbf{R}\textbf{c}_{\text {V}_2}+\textbf{c}-\textbf{c}_{\text {V}_1}$, $\textbf{x}=[\varvec{\xi }^\top , \textbf{v}^\top ,{\textbf{w}^i}^\top ]^\top ,\ i=1,\cdots ,N$.

Next, we derive the residual function of the reprojection cost and the corresponding Jacobian matrix as follows:

$$\begin{aligned}{} & {} {\mathcal R}=\left[ \begin{array}{l}f_{\textrm{V}} \frac{\textbf{p}_{\textrm{V,1}}}{\textbf{p}_{\textrm{V,3}}} -f_{\textrm{V}}\frac{\textbf{r}_{\textrm{V,1}}}{\textbf{r}_{\textrm{V,3}}}\\ f_{\textrm{V}} \frac{\textbf{p}_{\textrm{V,2}}}{\textbf{p}_{\textrm{V,3}}} -f_{\textrm{V}}\frac{\textbf{r}_{\textrm{V,2}}}{\textbf{r}_{\textrm{V,3}}} \end{array} \right] ,\nonumber \\{} & {} \textbf{J}{{\mathcal R}}_\textbf{x} = \textbf{J}_{\textbf{p}_{\textrm{V}}} \textbf{J}^{\textbf{p}_{\textrm{V}}}_{\textbf{x}}\ ,\nonumber \\{} & {} \textbf{J}_{\textbf{p}_{\textrm{V}}}=\left[ \begin{array}{ccc} \frac{f_{\textrm{V}}}{\textbf{p}_{\textrm{V,3}}} &{} 0 &{} -f_{\textrm{V}} \frac{\textbf{p}_{\textrm{V,1}}}{\textbf{p}_{\textrm{V,3}}^2}\\ 0 &{} \frac{f_{\textrm{V}}}{\textbf{p}_{\textrm{V,3}}} &{} -f_{\textrm{V}} \frac{\textbf{p}_{\textrm{V,2}}}{\textbf{p}_{\textrm{V,3}}^2} \end{array} \right] , \nonumber \\{} & {} \textbf{J}^{\textbf{p}_{\textrm{V}}}_{\textbf{x}}= \left[ \begin{array}{ccc} \textbf{R}_{\textrm{V}}^\top \left( \textbf{R}^\top (\textbf{p}-\textbf{c})\right) ^\wedge&-\textbf{R}_{\textrm{V}}^\top \textbf{R}^\top&\textbf{R}_{\textrm{V}}^\top \textbf{R}^\top \end{array} \right] \ , \end{aligned}$$

(A.2)

where $\textbf{p}_{\textrm{V}}=\textbf{R}_{\textrm{V}}^\top \left( \textbf{R}^\top (\textbf{p}-\textbf{c})-\textbf{c}_{\textbf{V}_2}\right)$, $\textbf{r}_{\textrm{V}}=\textbf{R}_{\textrm{V}}^\top \textbf{r}$, $\textbf{r}$ denotes the refractive ray of feature $\textbf{u}$, $f_{\textrm{V}}$ denotes the focal length of the virtual camera, $\textbf{R}_{\textrm{V}}$ and $\textbf{c}_{\textrm{V}}$ are the rotation and location of the virtual camera coordinate frame with respect to the camera coordinate frame.

1.2 A.2 Scenario 2

1.2.1 A.2.1 Absolute Pose Refinement

By using the chain rule, we have the Jacobian matrix w.r.t $\textbf{R}$ and $\textbf{C}$ as

$$\begin{aligned} \textbf{J}{\mathcal L}_\textbf{R}= & {} \textbf{J}{\textbf{l}_{\text {W}_{1}}} \textbf{J}^{\textbf{l}_{\text {W}_{1}}}_\textbf{R} + \textbf{J}_{\textbf{l}_{\text {W}_{2}}} \textbf{J}^{\textbf{l}_{\text {W}_{2}}}_{\textbf{R}}\ ,\nonumber \\ \textbf{J}_\textbf{c}= & {} \textbf{J}_{\textbf{l}_{\text {W}_{1}}} \textbf{J}^{\textbf{l}_{\text {W}_{1}}}_\textbf{c} + \textbf{J}_{\textbf{l}_{\text {W}_{2}}} \textbf{J}^{\textbf{l}_{\text {W}_{2}}}_{\textbf{c}}\ . \end{aligned}$$

(A.3)

where $\textbf{J}_{\textbf{l}_{\text {W}_{1}}}=-\textbf{p}_\text {W}^\wedge$ and $\textbf{J}_{\textbf{l}_{\text {W}_{2}}}=\textbf{I}$. Since $\textbf{R}$ and $\textbf{c}$ are embedded in $\textbf{l}_\text {W}$, again, by the chain rule, we have:

$$\begin{aligned} \textbf{J}^{\mathbf {l_{W_{1}}}}_\textbf{R}= & {} \textbf{J}^{\mathbf {l_{W_{1}}}}_{\textbf{q}_\text {W}}\textbf{J}^{\textbf{q}_\text {W}}_{\textbf{i}_{\text {W}}}\textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}+ \textbf{J}^{\mathbf {l_{W_{1}}}}_{\textbf{r}_\text {W}}\textbf{J}^{\textbf{r}_\text {W}}_{\textbf{i}_{\text {W}}}\textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}, \nonumber \\ \textbf{J}^{\mathbf {l_{W_{1}}}}_\textbf{c}= & {} \textbf{J}^{\mathbf {l_{W_{1}}}}_{\textbf{q}_\text {W}}\textbf{J}^{\textbf{q}_\text {W}}_{^\text {W}\textbf{c}_\text {C}} \nonumber \\ \textbf{J}^{\mathbf {l_{W_{2}}}}_{\textbf{R}}= & {} \textbf{J}^{\mathbf {l_{W_{2}}}}_{\textbf{q}_\text {W}}\textbf{J}^{\textbf{q}_\text {W}}_{\textbf{i}_\text {W}}\textbf{J}^{\textbf{i}_\text {W}}_{\textbf{R}}+\textbf{J}^{\mathbf {l_{W_{2}}}}_{\textbf{r}_\text {W}}\textbf{J}^{\textbf{r}_\text {W}}_{\textbf{i}_\text {W}}\textbf{J}^{\textbf{i}_\text {W}}_{\textbf{R}}, \nonumber \\ \textbf{J}^{\mathbf {l_{W_{2}}}}_{\textbf{c}}= & {} \textbf{J}^{\mathbf {l_{W_{2}}}}_{\textbf{q}_\text {W}}\textbf{J}^{\textbf{q}_\text {W}}_{^\text {W}\textbf{c}_\text {C}}, \end{aligned}$$

(A.4)

where

(A.5)

1.2.2 A.2.2 Relative Pose Refinement

Recall the refractive epipolar constraint given as

$$\begin{aligned} {\mathcal E}= {\textbf{r}^2_\text {W}}^\top {(^\text {W}\textbf{c}^2_{\text {V}}- ^\text {W}\textbf{c}^1_{\text {V}})}^\wedge {\textbf{r}^1_\text {W}}\ , \end{aligned}$$

(A.6)

using the chain rule, we have the Jacobian matrices w.r.t the variation of $\textbf{R}$, $\textbf{c}$, and $\textbf{p}$ as

$$\begin{aligned} \textbf{J}{\mathcal E}_{\textbf{x}}= & {} [\textbf{J}{\mathcal E}_\textbf{R}, \textbf{J}{\mathcal E}_\textbf{c}, \textbf{0}]\ , \nonumber \\ \textbf{J}{\mathcal E}_\textbf{R}= & {} \textbf{J}{\mathcal E}_{\textbf{r}^2_\text {W}} \textbf{J}^{\textbf{r}^2_\text {W}}_\textbf{R} + \textbf{J}{\mathcal E}_{^\text {W}\textbf{c}^2_{\text {V}}} \textbf{J}^{^\text {W}\textbf{c}^2_{\text {V}}}_\textbf{R}\ , \nonumber \\ \textbf{J}{\mathcal E}_\textbf{c}= & {} \textbf{J}{\mathcal E}_{^\text {W}\textbf{c}^2_{\text {V}}} \textbf{J}^{^\text {w}\textbf{c}^2_{\text {V}}}_\textbf{c}\ . \end{aligned}$$

(A.7)

where

$$\begin{aligned} \textbf{J}{\mathcal E}_{\textbf{r}^2_\text {W}}= & {} \left( (^\text {W}\textbf{c}^2_{\text {V}}- ^\text {W}\textbf{c}^1_{\text {V}})^\wedge \textbf{r}^1_\text {W}\right) ^\top ,\nonumber \\ \textbf{J}{\mathcal E}_{^\text {W}\textbf{c}^2_{\text {V}}}= & {} -{\textbf{r}^2_\text {W}}^\top (\textbf{r}^1_\text {W})^\wedge \ , \nonumber \\ \textbf{J}^{\textbf{r}^2_\text {W}}_\textbf{R}= & {} \textbf{J}^{\textbf{r}^2_\text {W}}_{\textbf{i}_{\text {W}}} \textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}\ ,\nonumber \\ \textbf{J}^{^\text {W}\textbf{c}^2_{\text {V}}}_\textbf{R}= & {} \textbf{J}^{^\text {W}\textbf{c}^2_\text {V}}_{\textbf{i}_\text {W}} \textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}\ , \nonumber \\ \textbf{J}^{^\text {W}\textbf{c}^2_{\text {V}}}_\textbf{c}= & {} \textbf{J}^{^\text {W}\textbf{c}^2_\text {V}}_{^\text {W}\textbf{c}_\text {C}}\ . \end{aligned}$$

(A.8)

Similarly, for the reprojection error given as

$$\begin{aligned} {\mathcal R}=\left[ \begin{array}{l}f_{\textrm{V}} \frac{\textbf{p}_{\textrm{V,1}}}{\textbf{p}_{\textrm{V,3}}} -f_{\textrm{V}}\frac{\textbf{r}_{\textrm{V,1}}}{\textbf{r}_{\textrm{V,3}}}\\ f_{\textrm{V}}\frac{\textbf{p}_{\textrm{V,2}}}{\textbf{p}_{\textrm{V,3}}} -f_{\textrm{V}}\frac{\textbf{r}_{\textrm{V,2}}}{\textbf{r}_{\textrm{V,3}}} \end{array}\right] \ , \end{aligned}$$

(A.9)

using the chain rule, we have the Jacobian matrices w.r.t $\textbf{R}$, $\textbf{c}$, and $\textbf{p}$ as

$$\begin{aligned} \textbf{J}{\mathcal R}_{\textbf{x}}= & {} [\textbf{J}{\mathcal R}_\textbf{R}, \textbf{J}{\mathcal R}_\textbf{c}, \textbf{J}{\mathcal R}_\textbf{p}]\ , \nonumber \\ \textbf{J}{\mathcal R}_\textbf{R}= & {} \textbf{J}{\mathcal R}_{\textbf{p}_\text {V}}\textbf{J}^{\textbf{p}_\text {V}}_\textbf{R} + \textbf{J}{\mathcal R}_{\textbf{r}_\text {V}} \textbf{J}^{\textbf{r}_\text {V}}_\textbf{R}\ , \nonumber \\ \textbf{J}{\mathcal R}_\textbf{c}= & {} \textbf{J}{\mathcal R}_{\textbf{p}_\text {V}}\textbf{J}^{\textbf{p}_\text {V}}_\textbf{c} + \textbf{J}{\mathcal R}_{\textbf{r}_\text {V}} \textbf{J}^{\textbf{r}_\text {V}}_\textbf{c}\ , \nonumber \\ \textbf{J}{\mathcal R}_\textbf{p}= & {} \textbf{J}{\mathcal R}_{\textbf{p}_\text {V}}\textbf{J}^{\textbf{p}_\text {V}}_\textbf{p}\ , \end{aligned}$$

(A.10)

where

$$\begin{aligned} \textbf{J}{\mathcal R}_{\textbf{p}_\text {V}}= & {} \left[ \begin{array}{ccc} f_\text {V}\frac{1}{\textbf{p}_\text {V,3}} &{} 0 &{} -f_\text {V}\frac{\textbf{p}_\text {V,1}}{\textbf{p}_\text {V,3}^2} \\ 0 &{} f_\text {V}\frac{1}{\textbf{p}_\text {V,3}} &{} -f_\text {V}\frac{\textbf{p}_\text {V,2}}{\textbf{p}_\text {V,3}^2} \end{array} \right] \ ,\nonumber \\ \textbf{J}{\mathcal R}_{\textbf{r}_\text {V}}= & {} \left[ \begin{array}{ccc} -f_\text {V}\frac{1}{\textbf{r}_\text {V,3}} &{} 0 &{} f_\text {V}\frac{\textbf{r}_\text {V,1}}{\textbf{r}_\text {V,3}^2} \\ 0 &{} -f_\text {V}\frac{1}{\textbf{r}_\text {V}(3)} &{} f_\text {V}\frac{\textbf{r}_\text {V,2}}{\textbf{r}_\text {V,3}^2} \end{array} \right] \ ,\nonumber \\ \textbf{J}^{\textbf{p}_\text {V}}_\textbf{R}= & {} \textbf{J}^{\textbf{p}_\text {V}}_{^\text {W}\textbf{c}_\text {V}}\textbf{J}^{^\text {W}\textbf{c}^2_{\text {V}}}_\textbf{R}\ , \quad \textbf{J}^{\textbf{r}_\text {V}}_\textbf{R} = \textbf{J}^{\textbf{r}_\text {V}}_{\textbf{q}_\text {W}} \textbf{J}^{\textbf{q}_\text {W}}_{\textbf{i}_{\text {W}}} \textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}\ ,\quad \textbf{J}^{\textbf{p}_\text {V}}_\textbf{p} = \textbf{I}\ ,\nonumber \\ \textbf{J}^{\textbf{p}_\text {V}}_\textbf{c}= & {} \textbf{J}^{\textbf{p}_\text {V}}_{^\text {W}\textbf{c}_\text {V}} \textbf{J}^{^\text {W}\textbf{c}^i\text {V}}_{^\text {W}\textbf{c}_\text {C}} \ , \quad \textbf{J}^{\textbf{r}_\text {V}}_\textbf{c} = \textbf{J}^{\textbf{r}_\text {V}}_{\textbf{q}_\text {W}} \textbf{J}^{\textbf{q}_\text {W}}_{^\text {W}\textbf{c}_{\text {C}}}\ . \end{aligned}$$

(A.11)

Some intermediate Jacobian matrices in (A.8) and (A.11) are given as follows:

$$\begin{aligned} \textbf{J}^{\textbf{r}^2_\text {W}}_\textbf{R}= & {} \textbf{J}^{\textbf{r}^2_\text {W}}_{\textbf{i}_{\text {W}}} \textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}\ ,\nonumber \\ \textbf{J}^{^\text {W}\textbf{c}^2_{\text {V}}}_\textbf{R}= & {} \textbf{J}^{^\text {W}\textbf{c}^2_\text {V}}_{\textbf{i}_\text {W}} \textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}\ , \nonumber \\ \textbf{J}^{^\text {W}\textbf{c}^2_{\text {V}}}_\textbf{c}= & {} \textbf{J}^{^\text {W}\textbf{c}^2_\text {V}}_{^\text {W}\textbf{c}_\text {C}} \ , \end{aligned}$$

(A.12)

where

$$\begin{aligned} \textbf{J}^{\textbf{q}_\text {W}}_{^\text {W}\textbf{c}_{\text {C}}}= & {} \left[ \begin{array}{ccc} 1 &{} 0 &{} - \frac{\textbf{i}_{\text {W,1}}}{\textbf{i}_{\text {W,3}}} \\ 0 &{} 1 &{} - \frac{\textbf{i}_{\text {W,2}}}{\textbf{i}_{\text {W,3}}} \\ 0 &{} 0 &{} 0\\ \end{array}\right] \ , \nonumber \\ \textbf{J}^{\textbf{q}_\text {W}}_{\textbf{i}_{\text {W}}}= & {} \left[ \begin{array}{ccc} -\frac{^\text {W}\textbf{c}_{\text {C,3}}}{\textbf{i}_{\text {W,3}}} &{} 0 &{} ^\text {W}\textbf{c}_{\text {C,3}}\frac{\textbf{i}_{\text {W,1}}}{\textbf{i}_{\text {W,3}}^2}\\ 0 &{} -\frac{^\text {W}\textbf{c}_{\text {C,3}}}{\textbf{i}_{\text {W,3}}} &{} ^\text {W}\textbf{c}_{\text {C,3}}\frac{\textbf{i}_{\text {W,2}}}{\textbf{i}_{\text {W,3}}^2}\\ 0 &{} 0 &{} 0\\ \end{array}\right] \ , \nonumber \\ \textbf{J}^{\textbf{r}^i_\text {W}}_{\textbf{i}_{\text {W}}}= & {} \left[ \begin{array}{ccc} \lambda &{} 0 &{} 0 \\ 0 &{} \lambda &{} 0 \\ 0 &{} 0 &{} \frac{\lambda ^2\textbf{i}_{\text {W,3}}}{\gamma }\\ \end{array}\right] \ , \nonumber \\ \textbf{J}^{^\text {W}\textbf{c}^i\text {V}}_{\textbf{i}_\text {W}}= & {} \left[ \begin{array}{ccc} 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} 0 \\ 0 &{} 0 &{} \frac{-^\text {W}\textbf{c}_\text {C,3}(1-\lambda ^2)}{\lambda \textbf{i}_\text {W,3}^2\gamma } \\ \end{array}\right] \ , \nonumber \\ \textbf{J}^{^\text {W}\textbf{c}^i\text {V}}_{^\text {W}\textbf{c}_\text {C}}= & {} \left[ \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} 1 &{} 0 \\ 0 &{} 0 &{} \frac{1}{\lambda \textbf{i}_\text {W,3}}\gamma \\ \end{array}\right] \ , \nonumber \\ \textbf{J}^{\textbf{i}_{\text {W}}}_{\textbf{R}}= & {} \left( {\textbf{q}_\text {W}}\right) ^\wedge \ , \textbf{J}^{\textbf{p}_\text {V}}_{^\text {W}\textbf{c}_\text {V}} = -\textbf{I}\ ,\quad \textbf{J}^{\textbf{r}_\text {V}}_{\textbf{q}_\text {W}} = \textbf{I}\ . \end{aligned}$$

(A.13)

where $\gamma =\sqrt{\left( 1-\lambda ^{2}\right) +\lambda ^{2} \textbf{i}_\text {W,3}^{2}}$.

B Additional Experiments

1.1 B.1 Setup of Synthetic Simulation

Figure 33 show how the data are generated to evaluate the absolute pose solvers in the non planar and planar cases. Figure 34 show how the data are generated to evaluate the relative pose solvers.

Fig. 33

Simulation setup for absolute pose estimation: The left figure shows the setup for generating data (world points, image points, camera pose) under refraction for the non planar case. The right figure shows the setup for generating data for the planar case

Fig. 34

Simulation setup for relative pose estimation: The figure shows the setup for generating data (world points, image points, camera pose) under refraction

1.2 B.2 Scenario 1: Absolute Pose Solver

As mentioned in Sect. 5.1.1, we found that

• UPnP Kneip et al. (2014) unfortunately does not perform well even in non planar case without noise perturbation, which is shown in Fig. 35.

• gPnP Kneip et al. (2013), gP+s Ventura et al. (2014), and MMPPE Miraldo et al. (2018) unfortunately does not perform well even in planar case without noise perturbation, which is shown in Fig. 36.

Fig. 35

Comparison of absolute pose solvers in non planar case. Note that we set the standard deviation of the Gaussian noise to 0

Fig. 36

Comparison of absolute pose solvers in planar case. Note that we set the standard deviation of the Gaussian noise to 0

1.3 B.3 Scenario 1: Relative Pose Solver

As mentioned in Sect. 5.1.2, we intended to add other GEC (Generalized Epipolar Constraint Pless (2003)) such as ge Kneip and Li (2014) and 6pt Stewenius et al. (2005). However, ge and 6pt do not show satisfactory performance in noise-free condition, as shown in Fig. 37.

Fig. 37

Comparison of GEC solvers in noise-free case. Note that 17pt gives very accurate results so that its box is very tiny in the box plots