A Robust Monocular 3D Object Tracking Method Combining Statistical and Photometric Constraints

Abstract

Both region-based methods and direct methods have become popular in recent years for tracking the 6-dof pose of an object from monocular video sequences. Region-based methods estimate the pose of the object by maximizing the discrimination between statistical foreground and background appearance models, while direct methods aim to minimize the photometric error through direct image alignment. In practice, region-based methods only care about the pixels within a narrow band of the object contour due to the level-set-based probabilistic formulation, leaving the foreground pixels beyond the evaluation band unused. On the other hand, direct methods only utilize the raw pixel information of the object, but ignore the statistical properties of foreground and background regions. In this paper, we find it beneficial to combine these two kinds of methods together. We construct a new probabilistic formulation for 3D object tracking by combining statistical constraints from region-based methods and photometric constraints from direct methods. In this way, we take advantage of both statistical property and raw pixel values of the image in a complementary manner. Moreover, in order to achieve better performance when tracking heterogeneous objects in complex scenes, we propose to increase the distinctiveness of foreground and background statistical models by partitioning the global foreground and background regions into a small number of sub-regions around the object contour. We demonstrate the effectiveness of the proposed novel strategies on a newly constructed real-world dataset containing different types of objects with ground-truth poses. Further experiments on several challenging public datasets also show that our method obtains competitive or even superior tracking results compared to previous works. In comparison with the recent state-of-art region-based method, the proposed hybrid method is proved to be more stable under silhouette pose ambiguities with a slightly lower tracking accuracy.

Appendix A: The Hessian Approximation

Here we give a mathematical explanation of the Hessian approximation [Eq. (19)] used for the statistical term in our energy function:

$$\begin{aligned} {E_{fb}}= & {} - \sum \limits _{i = 1:n} \sum \limits _{{\mathbf{x}} \in {\varOmega _i}} \log \left( {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} \right. \nonumber \\&\left. + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}} \right) \end{aligned}$$

(24)

We first consider the statistical energy for a single pixel $\mathbf {x}$:

$$\begin{aligned} {E_{fb}}\left( {\mathbf{x}} \right)&= - \log \left( {{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}}} \right) \nonumber \\ \end{aligned}$$

(25)

The Jacobian for pixel $\mathbf {x}$:

$$\begin{aligned} {\mathbf{J}}_{fb}\left( {\mathbf{x}} \right)&= \frac{{\partial {E_{fb}}\left( {\mathbf{x}} \right) }}{{\partial {\mathbf{p}}}}\, \nonumber \\&\quad = \, -\, \frac{{{P_{{f_i}}} - {P_{{b_i}}}}}{{{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}}}} \nonumber \\&\quad \frac{{\partial {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) }}{{\partial {\mathbf{p}}}} \end{aligned}$$

(26)

Here ${{\mathbf{J}}_{fb}}\left( {\mathbf{x}} \right) \in {{\mathbf{R}}^6}$, corresponding to the 6-dof pose parameter $\mathbf {p}$. The m-th element can be written as:

$$\begin{aligned} {\left[ {{{\mathbf{J}}_{fb}} \left( {\mathbf{x}} \right) } \right] _m}&= \frac{{\partial {E_{fb}}\left( {\mathbf{x}} \right) }}{{\partial {p_m}}}\, \nonumber \\&\quad = \, - \,\frac{{{P_{{f_i}}} - {P_{{b_i}}}}}{{{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}}}} \nonumber \\&\quad \frac{{\partial {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) }}{{\partial {p_m}}} \end{aligned}$$

(27)

The Hessian for pixel $\mathbf {x}$: ${{\mathbf{H}}_{fb}}\left( {\mathbf{x}} \right) \in {{\mathbf{R}}^{6 \times 6}}$, and the (m, n)-th element can be written as:

$$\begin{aligned}&{\left[ {{{\mathbf{H}}_{fb}}\left( {\mathbf{x}} \right) } \right] _{m,n}} = \frac{{{\partial ^2}{E_{fb}}\left( {\mathbf{x}} \right) }}{{\partial {p_m}\partial {p_n}}}\, \nonumber \\ \qquad&= \frac{{\partial \left[ { - \frac{{{P_{{f_i}}} - {P_{{b_i}}}}}{{{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}}}}\frac{{\partial {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) }}{{\partial {p_m}}}} \right] }}{{\partial {p_n}}}\, \nonumber \\ \qquad&= \frac{{{{\left( {{P_{{f_i}}} - {P_{{b_i}}}} \right) }^2}\frac{{\partial {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) }}{{\partial {p_n}}}}}{{{{\left[ {{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}}} \right] }^2}}}\frac{{\partial {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) }}{{\partial {p_m}}} \nonumber \\&\qquad -\, \frac{{{P_{{f_i}}} - {P_{{b_i}}}}}{{{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) {P_{{f_i}}} + \left( {1 - {H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) } \right) {P_{{b_i}}}}}\frac{{{\partial ^2}{H_e}\left( {\varPhi \left( {\mathbf{x}} \right) } \right) }}{{\partial {p_m}\partial {p_n}}} \nonumber \\ \end{aligned}$$

(28)

We denote the first term as $h_1$ (which contains first order derivatives), and the second term as $h_2$ (which contains second order derivatives):

$$\begin{aligned} {\left[ {{{\mathbf{H}}_{fb}}\left( {\mathbf{x}} \right) } \right] _{m,n}} = {h_1} - {h_2} \end{aligned}$$

(29)

Comparing to Eq. (27), we have:

$$\begin{aligned} {h_1} = {\left[ {{{\mathbf{J}}_{fb}}\left( {\mathbf{x}} \right) } \right] _m}{\left[ {{{\mathbf{J}}_{fb}}\left( {\mathbf{x}} \right) } \right] _n} = {\left[ {{{\mathbf{J}}_{fb}}{{\left( {\mathbf{x}} \right) }^T}{{\mathbf{J}}_{fb}}\left( {\mathbf{x}} \right) } \right] _{m,n}} \end{aligned}$$

(30)

As with the standard Gauss–Newton method (which aims to solve non-linear least square problems), we obtain an approximation of the Hessian matrix by ignoring the second-order derivative term $h_2$:

$$\begin{aligned}&{\left[ {{{\mathbf{H}}_{fb}}\left( {\mathbf{x}} \right) } \right] _{m,n}} = {h_1} = {\left[ {{{\mathbf{J}}_{fb}}{{\left( {\mathbf{x}} \right) }^T}{{\mathbf{J}}_{fb}}\left( {\mathbf{x}} \right) } \right] _{m,n}} \end{aligned}$$

(31)

$$\begin{aligned}&{{\mathbf{H}}_{fb}}\left( {\mathbf{x}} \right) = {{\mathbf{J}}_{fb}}{\left( {\mathbf{x}} \right) ^T}{{\mathbf{J}}_{fb}}\left( {\mathbf{x}} \right) \end{aligned}$$

(32)

In the end, we sum over all of the pixels and obtain the Hessian approximation used in Sect. 3.4:

$$\begin{aligned} {{\mathbf{H}}_{fb}} = \sum \limits _{i = 1:n} {\sum \limits _{{\mathbf{x}} \in {\varOmega _i}} {{\mathbf{J}}_{fb}{{\left( {\mathbf{x}} \right) }^T}} } {\mathbf{J}}_{fb}\left( {\mathbf{x}} \right) \end{aligned}$$

(33)