Unsupervised cycle-consistent person pose transfer

Abstract

Person pose transfer, i.e., transferring the pose of a given person to a target pose, is a challenging task due to the complex interplay of appearance, pose, and background. Most of the previous works adopted the supervised framework and required paired person images with the same identity and different poses, which largely limits their applications. Besides, the background of the generated image may be altered from the original one due to some over-fitting issues, which is unfavorable for the pose transfer task. To tackle the above problems, we propose an unsupervised cycle-consistent person pose transfer approach. It is trained with unpaired cross-identity person images and can well preserve the background information. Compared with previous methods, our proposed approach achieves better results in the cross-identity person pose transfer task and similar results in self-identity one. Moreover, our method can serve as an effective data augmentation scheme for person recognition tasks, which is validated by extensive experiments on pedestrian re-identification and detection.

Introduction

This paper focuses on the person pose transfer problem, which is to transfer a person from one pose to another, as depicted in Fig. 1. It has several applications, such as image editing, video generation, and data augmentation. Meanwhile, it is exceptionally challenging due to the drastic appearance difference of the same person in different poses.

Recent years have witnessed several works dedicating to this problem, including some supervised methods [1], [2], [3], [4], [5] and unsupervised ones [6], [7], [8], most of which target at transferring person pose in self-identity scenarios during both train and test stage. However, it’s quite difficult to acquire image pairs with different poses of the same identity. Moreover, under the current generative adversarial network based pose transfer framework, pose transfer is conducted by inputting a given person image concatenated with key-point coordinates, which is obtained via a pre-trained pose estimation model, to the generative model. Considering the input pose key-point coordinates are from the same person during training, but in real scenarios, they are mostly from different persons, we presume those methods trained only in self-identity scenarios are sub-optimal for real-world person pose transfer applications. Because the key-point coordinates contain not only the pose information but also the body shape of one person, the existing self-identity person pose transfer methods can not well handle the cross-identity pose transfer task, which is the most common setting for pose transfer applications.

Compared with the self-identity pairs, we can easily acquire plenty of cross-identity training data without identity labels, i.e., person images with various poses but from different identities. Additionally, as shown in the top of Fig. 1, existing supervised methods can introduce unnecessary background change to the generated images because they force the output of the generator to be similar to the given ground truth image, whose background sometimes differs from the original person image. Considering the aim of the person pose transfer task, this background change is an over-fitting and is unfavorable. But we find there is currently no metric to evaluate the background change after the pose transfer.

Accordingly, we propose an unsupervised cycle-consistent person pose transfer framework which makes full use of the cross-identity training data and keeps the background unchanged, as illustrated in the bottom of Fig. 1. The target pose is extracted from another person image by the pretrained pose estimation model, which is OpenPose Human Pose Estimator (HPE) [9] in this paper. In existing supervised methods, the another person image must share the same identity with the input image, while it can belong to different identities in our proposed one. This difference dramatically enlarges the amount of target pose images and the diversity of person poses. What’s more, our method can even work on the raw images without the identity labels. During the training phase, we firstly transfer the source person to the target pose, then conduct the reverse pose transfer and obtain the reconstruction of the input image. By forcing the reconstruction to be consistent with the original one, we can avoid the background over-fitting for the generated image and preserve the background information in the input image.

Fig. 2 details the training pipeline of the proposed method. To fully exploit the training data, we take two images and their poses as the input and exchange their poses twice in each training iteration (firstly transferring to the other pose and then transferring back) [10], [11]. Meanwhile, we jointly optimize our model under the supervision of four kinds of losses during training to obtain the optimal generation effect. The first is the adversarial loss, which guarantees the visual quality of the generated person images. The second is the cycle-consistent loss, which minimizes the reconstruction error between the input person image and its reconstruction. The third is the identity-consistent loss (Fig. 3), which forces the generator to be an identity mapping function when the target pose is the same as the original one. What’s more, we propose a novel pose-consistent loss, which forces the pose of the generated person images to be consistent with the target pose and can effectively improve the pose transfer precision.

In summary, this paper makes the following contributions:

• We propose the unsupervised cycle-consistent person pose transfer framework. Different from the previous methods, it can get rid of the paired self-identity training data limitation and take any person images as training data, thus has much broader applications.

• Our cycle-consistent framework can well preserve the background information of the input images. Besides, we present a novel evaluation criterion “back-SSIM” to assess the background change in the generated images quantitatively.

• As demonstrated by extensive quantitative and qualitative experimental results, the proposed method achieves state-of-the-art performance in the cross-identity pose transfer task. Even if in the self-identity scenarios, it obtains similar results to those supervised ones.

• Our method can serve as an effective data augmentation scheme for person recognition tasks, such as pedestrian re-identification and detection.