A CNN-based 3D human pose estimation based on projection of depth and ridge data

Highlights

• We propose a CNN-based human pose estimation using depth and ridge data.

• We project the depth and ridge data on three orthogonal planes (XY, XZ, ZY).

• The projected depth and ridge data can eliminate the 3D information loss.

• Ridge data is introduced to avoid joint drift, which improves the accuracy of estimated poses.

• The proposed method achieved the state-of-the-art pose estimation accuracies.

Abstract

We propose a method that use a convolutional neural network (CNN) to estimate human pose by analyzing the projection of the depth and ridge data, which represent local maxima in a distance transform map. To fully utilize the 3D information of depth points, we propose a method to project the depth and ridge data on various directions. The proposed projection method can reduce the 3D information loss, the ridge data can avoid joint drift, and the CNN increases localization accuracy. The proposed method proceeds as follows. (1) We use depth data to segment the human from the background and extract ridge data from human silhouettes. (2) We project the depth and ridge data onto XY, XZ, and ZY planes. (3) ResNet-101 accepts six projected images and use 1  ×  1 convolution layers to generate 2D heatmaps and offsets. (4) We generate 2D keypoints per plane by using the soft-argmax operation. (5) We obtain 3D joint positions by using the fully-connected layers. In experiments on the SMMC-10, EVAL, and ITOP datasets, the proposed method achieved the state-of-the-art pose estimation accuracies. The proposed method can eliminate the 3D information loss and drift of joint positions that can occur during estimation of human pose.

Introduction

Human pose estimation is the task of finding human model parameters such as the length and orientation of body parts (e.g., head, torso, limbs) that fit an input image [1]. High-speed depth imaging devices permit extraction of rich information from depth silhouettes, so human pose estimation is simplified.

There are two approaches to identifying human poses from depth silhouettes: discriminative and generative. The discriminative approach is to find human joints from an observed input image using the pre-trained body part detectors. Kong et al. [2] proposed a geodesic feature based joint detector to localize body joints in a depth data. Jain et al. [3] proposed a head-torso detector based on Haar candidates and a template matching algorithm for each limb, but it required that upper-body and face be visible without any occlusion. Shotton et al. [4] proposed a human pose recognition approach that predicted an intermediate body-part representation to estimate the human pose but the prediction usually required expensive training steps and a large number of training samples to cover the wide human pose-space. Girshick et al. [5] used a regression forest to localize the joint positions directly from votes of each pixel, but it required increasingly complex training because their voting was modeled from the result of [4]. Wang et al. [6] used semi-local features extracted from randomly-sampled 4D sub-volumes of depth sequences and it required highly-complicated training for large processing time. Buys et al. [7] described a method to transform the depth data into the intermediate representation without background subtraction. Most of these discriminative approaches do not miss the body parts totally but are suffered from the occlusion problem because they can detect only visible parts, which degrades the human pose estimation accuracy seriously.

The generative approach is to find human joints by fitting the pre-defined human body model to an observed input image. Grest et al. [8] and Knoop et al. [9] proposed an iterative closest points (ICP) method to estimate human poses and to track human body parts, but they were computationally complicated, so they were not applicable in real-time systems. Rosenhahn et al. [10] presented a marker-less motion capture system that took the lower-dimensional human pose manifold into account by using soft constraints to model the motion restrictions during human pose optimization, but it could not be used for challenging outdoor scenes that included shadows and strong illumination changes. Straka et al. [11] addressed the occlusion problems by using graph-based inference, and Ye et al. [12] did so by using joint energy minimization, but they have limited applicabilities because they required multiple cameras. Zhang et al. [13] used a parameterized human shape model that estimated the human pose from all available data retrieved from several cameras. Although it improved human pose estimation, it still could not differentiate occluded body parts. Ganapathi et al. [14] proposed an efficient filtering algorithm for tracking human pose that combined an accurate generative model with a discriminative model that provided data-driven evidence about body part locations. Most of these generative approaches can solve the occlusion problem in some degree by exploiting prior knowledge of human body model and requires no training steps but fails to track the body parts entirely and takes long computation time because they fit a complicated human model to an observed input image iteratively.

Recently, convolutional neural networks (CNNs) have been successfully applied to the task of 3D human pose estimation. Toshev and Szegedy [15] proposed to directly regress the 2D Cartesian coordinate of joints in a holistic manner. Cao et al. [16] used the heatmaps that are intermediate representations for each joint to refine the joint position in the 2D pose estimation.

This paper proposes a method to identify human joints reliably. We propose use of ridge data, which constitute a novel representation of the human body. The ridge data are more plentiful and scale-invariant than the existing skeletonization techniques [17], [18]. Using these data, we achieve higher pose estimation accuracy (0.9868, 0.9835, and 0.9689 mAP) than the current state-of-the-art methods on SMMC-10, EVAL, and ITOP dataset.

The rest of this paper is organized as follows. Section 2 describes the proposed human depth silhouette segmentation. Section 3 describes the motivation, definition and extraction of ridge data, which constitute a novel feature for human pose estimation. Section 4 describes the projection method of the depth and ridge data. Section 5 describes the proposed CNN-based human pose estimation methods. Section 6 validates the proposed feature and methods using the experiments in the SMMC-10, EVAL, and ITOP datasets. Section 7 presents conclusions.