Adaptive data augmentation network for human pose estimation

Highlights

• The diversity of training samples will enhance the performance of human pose estimation.

• The appropriate number of pasted samples on the original image can affect network performance.

• Pasting the complete person is beneficial in enhancing the original image information.

• The generative adversarial network brings the synthetic image closer to real challenging cases.

Abstract

With the rapid development of convolutional neural networks (CNNs), the performance of human pose estimation has been significantly improved. However, state-of-the-art methods still face specific challenges, such as occluded keypoints and nearby persons. Unlike the classical network improvement strategy, our approach obtains adaptive data to obtain more accurate keypoints through data augmentation. In this paper, we propose Adaptive Data Augmentation Network (ADA-Net), which brings more adaptive training data by adding occlusion and interleaving information to the original image. First, we introduce Active Transmission Network (ATNet), which actively learns the transformation matrix from the original image and uses the matrix to synthesize new images for training. Second, we adopt an adversarial training strategy combined with the ATNet that allows us to capture more challenging cases. Extensive experiments show that our approach achieves comparable or even better results than most state-of-the-art methods. In particular, our ADA-Net outperforms High-Resolution Network (HRNet) by 1.1 and 0.7 points on the COCO test-dev set and MPII validation set, respectively.

Introduction

Human pose estimation identifies and locates the keypoints of all persons in a still image. This is a fundamental research technique for numerous visual applications including human motion analysis [43], human-computer interaction [17], animation [23], etc. With the rapid development of deep convolutional neural networks (DCNNs) [40], [21], the task of human pose estimation has made significant progress, and an obvious breakthrough has been achieved on standard benchmark datasets such as MS COCO [28] and MPII Human Pose [1]. However, there still exist many challenging cases, including occluded keypoints [14] and nearby persons [3], which cannot be well localized.

Insufficient training data can affect the performance of DCNN-based methods for keypoint locations, especially in the absence of challenging cases. For example, in [40], samples with occluded keypoints make it hard to train High-Resolution Network (HRNet) for accurate keypoint localization. In addition, it is costly to annotate the keypoints localization ourselves.

A common solution is to use data augmentation, which is a method of enhancing machine learning capabilities by generating additional samples. However, it requires technicians to design policies that capture prior knowledge in the corresponding domain [12]. Traditionally, data augmentation employs global image transformation (e.g., scaling, shifting, rotating, cropping, flipping, or color dithering), shown in Fig. 1. Although this method enhances the information within the training images, they are of little help in solving challenging cases [3]. The production of occluded data [21], [10] can be regarded as an effective tool. Cheng et al. [10] introduce a Cylinder Man Model to generate occlusion labels, which is hardly inspiring to us for 2D data augmentation. For 2D human pose estimation, Ke et al. [21] propose a Keypoint Masking Training strategy to enhance the information via copying a background patch and putting it onto a keypoint. While this method can simulate occluded data, it cannot significantly improve the final effect. The main reasons are twofold: 1) the training image only considers occluded keypoints without learning extra information; 2) the paste part used to occlude the keypoints is difficult to learn from samples of nearby persons, and such occlusion is not in line with the real scenes.

To solve realistic challenging cases, we propose the Adaptive Data Augmentation Network (ADA-Net), as shown in Fig. 2. First, we build a pool of complete persons. During training, a complete person is randomly selected and pasted into the original image to learn additional information. On the one hand, the complete person can easily overlap the different positions of the characters in the original image, thus forming more unique training data. On the other hand, all we see in real scenes are complete persons, so this is in line with the real scenes to add a complete person to the original image. Our research is practical from this perspective.

Then, ADA-Net applies the Active Transmission Network (ATNet) to add the complete person into the original image while changing its shape and position. Specifically, ATNet is an active approach that learns parameters from each original image to form different transformation matrices. The transformation matrix is used to obtain the shape and position of the complete person so that it can be better blended into the original image to produce a synthetic image. Furthermore, ATNet as a generator that acquires the corresponding transformation matrix to make the synthetic image as close as possible to real challenging cases. Finally, when using a pose estimation network as a discriminator for adversarial learning, ATNet can generate the most confusing training samples after receiving the original image and will synthesize more challenging cases. Thus, our method can obtain more adaptive training data by adding occlusion and interleaving information to the original image. In summary, our contributions can be summarized as follows:

• We build a pool of multiple complete persons. Exploring the appropriate number of complete persons within the pool is beneficial in enhancing the original image information and improving the overall performance of our method.

• We propose a novel network called ADA-Net. It exploits ATNet as a generator to actively learn the adaptive transformation matrix of the synthetic image, while using human pose estimation as a discriminator to bring the synthetic image closer to the real challenging cases.

• We evaluate our approach on both the COCO and MPII benchmarks. Experimental results show that our method is more effective than most state-of-the-art approaches.