Head pose estimation using deep neural networks and 3D point clouds

Highlights

• We propose head pose estimation using deep neural networks and 3D point cloud.

• We adopt 3D point cloud data generated from depth to estimate 3D head pose.

• This is the first work of 3D point cloud based head pose estimation in a deep learning framework.

Abstract

In this paper, we propose head pose estimation using deep neural networks and 3D point cloud. Unlike existing methods that either take 2D RGB image or 2D depth image as input, we adopt 3D point cloud data generated from depth to estimate 3D head poses. To further improve robustness and accuracy of head pose estimation, we classify 3D angles of head poses into 36 classes with 5 ${}^{\circ }$ interval and predict the probability of each angle in a class based on multi-layer perceptron (MLP). While traditional iterative methods for head model construction require high computation and memory costs, the proposed method is lightweight and computationally efficient by utilizing a sampled 3D point cloud as input combined with a graph convolutional neural network (GCNN). Experimental results on Biwi Kinect Head Pose dataset show that the proposed method achieves outstanding performance in head pose estimation and outperforms state-of-the-art ones in terms of accuracy.

Introduction

Head pose estimation are used for various vision applications such as human computer interaction and virtual reality. Moreover, it is a necessary preprocessing step for identification, gaze estimation, facial expression recognition and face 3D reconstruction [1], [2]. Head pose estimation aims to estimate the 3D Euler angles (roll, yaw and pitch) of head poses. Most existing methods [1], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14] take 2D RGB image as input to learn a mapping between 2D and 3D spaces. It is feasible to use traditional methods for regression and classification of the facial landmarks. However, it is difficult to accurately estimate the 3D head pose in a complex environment. In recent years, researchers [1], [7], [8], [9], [10], [11], [12] have used the powerful learning ability of convolution neural networks (CNNs) to extract features for head pose estimation. However, a single RGB image does not contain 3D information, thus causing estimation errors by the mapping from 2D space to 3D space. Mid-range and short-range depth cameras enable people to utilize depth information from depth images in a very convenient way. Model-based methods use depth information for head pose estimation by registering a 3D model via rigid iterative closest points (ICP) [15], [16] and particle swarm optimization (PSO) [17]. Although directly feeding a depth image into CNN seems to work well in head pose estimation [18], it is difficult for 2D CNN to extract features from 2D image that fully coincide with 3D spatial information of depth image.

To tackle the lack of 3D spatial information, we introduce a new data format, 3D point cloud, for head pose estimation. Point cloud is a set of 3D points on the visible surface generated from a single depth image obtained by a depth camera. The point cloud converted from the depth image is regarded as ordered, but after removing irrelevant parts and sampling it to the fixed number of points, the input point cloud is out of order. Meanwhile, typical convolution architectures like CNNs do not work well on this kind of unordered data. When we feed the point cloud into a traditional CNN, we shuffle the points and the CNN regards the shuffled point cloud as a different one even though they are the same as the original one. PointNet++ [19] performs a grouping operation that gathers the nearest points to a group as the convolution input. Point cloud is a data format that is suitable for building a graph. If we use graph convolution, we do not need to construct the data shape for the traditional convolution by stacking points in a group. The depth image also contains 2.5D information with x and y coordinates in an implicit way. However, the point cloud transforms the implicit x and y coordinates into the explicit data format, which removes the background outside face or zero value pixels inside face that may mislead the head pose estimation, thus resulting in performance improvement. Fig. 1 shows some head point clouds with their RGB and depth images.

In this paper, we propose head pose estimation from a single depth image using deep neural networks and 3D point cloud. Specifically, we first segment the input image using a head mask provided by the dataset, then convert the depth image of head into a set of 3D point cloud. Prior to feeding the 3D points into the network, we downsample and normalize them to have a normal distribution for robust estimation. Next, we perform the abstraction layer of PointNet++ to extract features from the input 3D points. Based on the extracted feature, we classify the head pose angle into 36 classes with 5 ${}^{\circ }$ interval and predict the probability of each angle in a class based on multi-layer perceptron (MLP). Finally, we use the classification loss in the following regression layer to estimate the final head pose angle.

Compared with existing methods, main contributions of this paper are summarize as follows:

• We propose a network architecture for head pose estimation that takes 3D point cloud from depth as input instead of RGB image or depth image. Thus, the proposed network captures 3D spatial information better than those based on RGB image or depth image.

• We adopt graph convolution on the 3D point cloud to prevent stacking of points and generate sufficient dimensions for input, thus resulting in low memory cost and lightweight model.

• We perform regression networks after classification of poses to make head pose estimation robust. This is different from the previous work that uses the expectation instead of learning a function.