Accurate and fast 3D head pose estimation with noisy RGBD images

Abstract

Head pose estimation plays an essential role in many high-level face analysis tasks. However, accurate and robust pose estimation with existing approaches remains challenging. In this paper, we propose a novel method for accurate three-dimensional (3D) head pose estimation with noisy depth maps and high-resolution color images that are typically produced by popular RGBD cameras such as the Microsoft Kinect. Our method combines the advantages of the high-resolution RGB image with the 3D information of the depth image. For better accuracy and robustness, features are first detected using only the color image, and then the 3D feature points used for matching are obtained by combining depth information. The outliers are then filtered with depth information using rules proposed for depth consistency, normal consistency, and re-projection consistency, which effectively eliminate the influence of depth noise. The pose parameters are then iteratively optimized using the Extended LM (Levenberg-Marquardt) method. Finally, a Kalman filter is used to smooth the parameters. To evaluate our method, we built a database of more than 10K RGBD images with ground-truth poses recorded using motion capture. Both qualitative and quantitative evaluations show that our method produces notably smaller errors than previous methods.

Appendix

For the given parameter Θ and energy function Φ(Θ), we need to find ΔΘ so that Φ(Θ + ΔΘ) decreases gradually until $\Phi (\hat \Theta )$ reaches a minimum. Then, $\hat \Theta$ is the optimal parameter. In the LM algorithm ΔΘ is calculated as follows:

$$ {\Delta}\Theta=-({J}^{T}{J}+\lambda{J})^{-1}{J}^{T}(\Phi(\Theta+{\Delta}\Theta)-\Phi(\Theta)) $$

(17)

where J is the Jacobian matrix, which is $\frac {\partial {\Phi (\Theta +{\Delta }\Theta )}}{\partial \Theta }$, and λ is the step size. In our algorithm, Jacobian matrix J degenerates into a vector, that consists of the directional derivative in each parameter direction.

$$ {J}=\left[ \frac{\partial \Phi}{\partial \theta},\frac{\partial \Phi}{\partial \psi},\frac{\partial \Phi}{\partial \phi},\cdots\right]^{T} $$

(18)

Jacobian matrix J can not be calculated using partial derivatives. In the process of solving the parameters, we use a sampling method to estimate it. We choose sufficiently small increments

$$\begin{array}{@{}rcl@{}} \left\{\begin{array}{llllll} \nabla_{\theta}=[d_{\theta},0,0,0,0,0]^{T} , \\ \nabla_{\psi}=[0,d_{\psi},0,0,0,0]^{T} , \\ \nabla_{\phi}=[0,0,d_{\phi},0,0,0]^{T} , \\ \nabla_{t_{x}}=[0,0,0,d_{t_{x}},0,0]^{T} , \\ \nabla_{t_{y}}=[0,0,0,0,d_{t_{y}},0]^{T} , \\ \nabla_{t_{z}}=[0,0,0,0,0,d_{t_{z}}]^{T} , \end{array}\right. \end{array} $$

(19)

We then use sampling to calculate Φ(Θ) and Φ(Θ +Δ _i Θ),i = 1, 2,⋯ , 6, and its directional derivative can be approached by the following formula:

$$\begin{array}{@{}rcl@{}} \left\{\begin{array}{llllll} \frac{\partial{\Phi(\Theta)}}{\partial{\theta}} \approx \frac{\Phi(\Theta)-\Phi(\Theta+\nabla_{\theta})}{d_{\theta}}, \\ \frac{\partial{\Phi(\Theta)}}{\partial{\psi}} \approx \frac{\Phi(\Theta)-\Phi(\Theta+\nabla_{\psi})}{d_{\psi}}, \\ \frac{\partial{\Phi(\Theta)}}{\partial{\phi}} \approx \frac{\Phi(\Theta)-\Phi(\Theta+\nabla_{\phi})}{d_{\phi}}, \\ \frac{\partial{\Phi(\Theta)}}{\partial{t_{x}}} \approx \frac{\Phi(\Theta)-\Phi(\Theta+\nabla_{t_{x}})}{d_{t_{x}}}, \\ \frac{\partial{\Phi(\Theta)}}{\partial{t_{y}}} \approx \frac{\Phi(\Theta)-\Phi(\Theta+\nabla_{t_{y}})}{d_{t_{y}}}, \\ \frac{\partial{\Phi(\Theta)}}{\partial{t_{z}}} \approx \frac{\Phi(\Theta)-\Phi(\Theta+\nabla_{t_{z}})}{d_{t_{z}}}, \end{array}\right. \end{array} $$

(20)

For parameter λ, if the ΔΘ of an iteration makes Φ(Θ) decrease, this ΔΘ is accepted, and then we need to reduce λ. If Φ(Θ) increases, then we need to increase λ to recalculate.