6DFLRNet: 6D rotation representation for head pose estimation based on facial landmarks and regression

Abstract

Head pose estimation methods can be generally classified into two categories: model-based and appearance-based methods. The model-based approach relies on facial landmarks for three-dimensional reconstruction, aiming to achieve high-precision results. However, this method is heavily dependent on the accuracy of these landmarks. The appearance-based approach utilizes images as input and employs feature extraction and calculations to generate outcomes. While the appearance-based method boasts greater robustness, its accuracy falls short of the former. In this paper, a new and effective hybrid method is proposed. This hybrid approach combines the strengths of both methods. Unlike the conventional model-based methods, the proposed method regards the facial landmarks in 2D images as a sequence of neural network inputs and then obtains the head pose estimation results for users by neural network regression. The proposed method solves the fuzzy rotation labeling problem by using a rotation matrix representation, introducing a 6D rotation matrix representation as an intermediate state of the rotation matrix to achieve effective direct regression. Introducing face processing enhances the robustness of the model in cross-dataset scenarios. The proposed method achieves remarkable results based on imprecise face recognition and a simplistic model. The proposed method can be divided into three parts. First, the proposed method applies face processing on the input image; second, the method detects facial landmarks; and third, it converts these facial landmarks into sequences and obtains the 6D rotation representation of the head pose by regression. Extensive experiments on the publicly available BIWI, PRIMA, and DrivFace datasets show that this method is functional and performs better than other state-of-the-art methods. Compared to other methods, this approach demonstrates an average performance improvement of at least 10% across the dataset.

Appendix

To facilitate readers’ better understanding of the formulas employed in the paper, this paper has provided a comprehensive description of all symbols utilized in the formulas within Table 7.

Table 7 Table of symbols

The derivation formula for converting 6D rotation matrix to rotation matrix is as follows:

$$\begin{aligned} f_{\text{ map } }\left( \left[ \begin{array}{cc} \mid &{} \mid \\ \alpha _1 &{} \alpha _2 \\ \mid &{} \mid \end{array}\right] \right) =\left[ \begin{array}{ccc} \mid &{} \mid &{} \mid \\ \beta _1 &{} \beta _2 &{} \beta _3 \\ \mid &{} \mid &{} \mid \end{array}\right] \end{aligned}$$

(12)

The last column vector can be obtained from the first two column vectors by calculation. This work refers to the method in [40] and uses Gram–Schmidt orthogonalization for the two-column vectors in the 6D rotation representation matrix to obtain an orthogonal matrix that satisfies the constraints.

$$\begin{aligned} \begin{gathered} \beta _1=\frac{\alpha _1}{\left\| \alpha _1\right\| } \\ \beta _2=\frac{\gamma _2}{\left\| \gamma _2\right\| }, \gamma _2=\alpha _2-\left( \beta _1 \cdot \alpha _2\right) \beta _1 \\ \beta _3=\beta _1 \times \beta _2 \end{gathered} \end{aligned}$$

(13)

The derivation process of the rotation matrix R is as follows:

$$\begin{aligned} \begin{aligned}&R_x(\alpha )=\left[ \begin{array}{ccc} 1 &{} 0 &{} 0 \\ 0 &{} \cos \alpha &{} -\sin \alpha \\ 0 &{} \sin \alpha &{} \cos \alpha \end{array}\right] \\ \end{aligned} \end{aligned}$$

(14)

$$\begin{aligned} \begin{aligned}&R_y(\beta )=\left[ \begin{array}{ccc} \cos \beta &{} 0 &{} \sin \beta \\ 0 &{} 1 &{} 0 \\ -\sin \beta &{} 0 &{} \cos \beta \end{array}\right] \\ \end{aligned} \end{aligned}$$

(15)

$$\begin{aligned} \begin{aligned}&R_z(\gamma )=\left[ \begin{array}{ccc} \cos \gamma &{} -\sin \gamma &{} 0 \\ \sin \gamma &{} \cos \gamma &{} 0 \\ 0 &{} 0 &{} 1 \end{array}\right] \end{aligned} \end{aligned}$$

(16)

The rotation matrix is obtained by rotating according to the X-Y-Z axis:

$$\begin{aligned} \begin{gathered} R=R_z(\gamma ) * R_y(\beta ) * R_x(\alpha ) \\ =\left[ \begin{array}{ccc} \cos \gamma \cos \beta &{} \cos \gamma \sin \beta \sin \alpha -\sin \gamma \cos \alpha &{} \cos \gamma \sin \beta \cos \alpha +\sin \gamma \sin \alpha \\ \sin \gamma \cos \beta &{} \sin \gamma \sin \beta \sin \alpha +\cos \gamma \cos \alpha &{} \sin \gamma \sin \beta \cos \alpha -\cos \gamma \sin \alpha \\ -\sin \beta &{} \cos \beta \sin \alpha &{} \cos \beta \cos \alpha \end{array}\right] \end{gathered} \end{aligned}$$

(17)