Spatial-aware stacked regression network for real-time 3D hand pose estimation

Highlights

• A stacked regression network for fast, robust and accurate 3D hand pose estimation is proposed.

• A pose re-parameterization is adopted to utilize the 3D spatial structure of hand.

• A spatial attention module is adopted to reduce the influence of irrelevant features.

• A cross-stage self-distillation module is adopted to achieve a lightweight network.

Abstract

Making full use of the spatial information of the depth data is crucial for 3D hand pose estimation from a single depth image. In this paper, we propose a Spatial-aware Stacked Regression Network (SSRN) for fast, robust and accurate 3D hand pose estimation from a single depth image. By adopting a differentiable pose re-parameterization process, our method efficiently encodes the pose-dependent 3D spatial structure of the depth data as spatial-aware representations. Taking such spatial-aware representations as inputs, the stacked regression network utilizes multi-joint spatial context and the 3D spatial relationship between the estimated pose and the depth data to predict a refined hand pose. To further improve the estimation accuracy, we adopt a spatial attention mechanism to reduce the influence of irrelevant features for pose regression. In order to improve the speed of the network, we propose a cross-stage self-distillation mechanism to distill knowledge within the network itself. Experiments on four datasets show that our proposed method achieves state-of-the-art accuracy with high running speed around 330 FPS on a single GPU and 35 FPS on a single CPU.

Introduction

Hand pose estimation plays an important role in applications of human–computer interaction in virtual reality and augmented reality. With the development of deep learning and low-cost depth camera, it has made significant progress in recent years [1], [2], [3]. Nevertheless, it is still challenging to achieve accurate and robust hand pose estimation performance due to large variations in hand orientations, high similarity among fingers, severe self-occlusion and poor quality of depth images. In addition, improving the speed of the algorithm is also an important issue in order to satisfy the requirements of interactive application scenarios.

Recently, deep neural network-based approaches have achieved drastic performance improvement in 3D hand pose estimation. One line of work for 3D hand poses estimation is holistic regression [4], [5], [6], [7], [8], [9], that is aiming to directly predict 3D pose parameters such as joint angles or 3D joint locations from the depth image. Regression-based methods are able to capture global constraints among different joints, thus they are robust to self-occlusion and poor quality images [2]. However, since these methods treat the depth image as a 2D image, they under-utilize 3D spatial information of the depth image, thus having relatively low precision.

Explicit consideration of 3D spatial properties of the depth image can significantly improve the accuracy of estimation [2]. One straightforward solution is converting the depth image to 3D data, such as voxels [10], [11] and points [12], [13], [14], [15], [16], and then applying a 3D deep learning method. An alternative way is to incorporate spatial-aware representations into 2D CNNs. These works utilize the fully convolutional network to estimate pixel-wise representations such as 3D heat-maps, 3D unit vector fields and approximated geodesic distance maps, from which the 3D hand joint locations can be inferred by post-processing [17], [18], [19] or a regression network [20]. Adopting spatial-aware representations allows 2D CNNs to consider both the 2D and 3D properties of the depth image and makes it easy to leverage stacked architecture to reevaluate the initial estimations. However, those methods still suffer from some limitations.

First, performing pixel-wise estimation is inefficient, because it needs a computationally expensive upsampling step to obtain high-resolution pixel-wise estimations [17], [19], [21]. Thus, compared with regression-based methods, these methods require more complex network structures and more computation. Second, when the depth data near the target joint points are missing or occluded, directly inferring the joint coordinates from the spatial-aware representations is unreliable [13].

In essence, the spatial-aware representations are embeddings of low-dimensional joint coordinates on high-dimensional image space. They reflect the spatial location of joints or the spatial relationship such as distance, direction and offset between the joint coordinates and each pixel of the depth image. They are able to provide rich 3D spatial information and powerful disambiguation clues for further optimization. We argue that the form of representation is more critical than the process of representation generation for iterative refinement. Based on this inspiration, we propose a stacked regression architecture, which directly encodes the previously estimated pose into spatial-aware representations by a pose re-parameterization, thus subsequent regression stages are able to utilize multi-joint spatial context and the 3D spatial relationship between the estimated pose and the depth data to perform iterative refinement.

In this paper, we propose a Spatial-aware Stacked Regression Network (SSRN) for fast, robust and accurate 3D hand pose estimation from a single depth image. SSRN has multiple pose regression modules, which are connected by the differentiable pose re-parameterization module. Specifically, the pose re-parameterization module generates spatial-aware representations from previously estimated pose directly. Then, the subsequent pose regression module predicts a more accurate pose based on multi-joint spatial context and the 3D spatial relationship between the estimated pose and the depth data on the spatial-aware representations. We regard the first pose regression module as the initial stage and the subsequent regression module as the refinement stage. The pose re-parameterization process is simple, fast and non-parametric. It can generate high-quality spatial-aware representations with little overhead in computation and storage. Furthermore, we integrate and explore multiple good practices including data augmentation, smooth L1 loss, localization refinement and coordinate decoupling to improve the performance of 2D CNN for 3D hand pose estimation.

Our main contributions can be summarized as follows:

• (1)  We adopt a differentiable pose re-parameterization process to generate spatial-aware representations. Compared with performing pixel-wise estimation, it can generate high-quality spatial-aware representations with little overhead in computation and storage.

• (2)  We incorporate the spatial-aware representations into a stacked regression network. Spatial-aware representations allow us to efficiently utilize the 3D spatial information in the depth image and multi-joint spatial context for accurate 3D hand pose estimation.

This paper is an extension of our conference paper [22]. The new contributions of this paper are summarized as follows:

• (1)  We propose a spatial attention mechanism to replace the global average pooling (GAP) and fully connected (FC) layer in the regression-based method to reduce the influence of irrelevant regions in the feature maps for pose estimation. Experimental results show that, with the spatial attention mechanism, the estimation accuracy of the regression-based method can be further improved.

• (2)  We propose a cross-stage self-distillation mechanism to align the features of the initial stage with the refinement stage, which allows us to adopt a more lightweight network in the initial stage while maintaining the estimation accuracy so that the whole network has faster reasoning speed.

• (3)  We conduct more extensive self-comparison experiments and a cross-dataset experiment. Experimental results show that our method has good generalization ability.

We evaluate our method on four publicly available 3D hand pose datasets (NYU [1], ICVL [23], MSRA [24], HANDS 2017 [2]). Our method achieves state-of-the-art accuracy on four datasets with fewer parameters and a faster frame rate. The reasoning speed of our method is around 330 FPS on a single GPU and 35 FPS on a single CPU.