Person image generation with attention-based injection network

Abstract

Person image generation becomes a challenging problem due to the content ambiguity and style inconsistency. In this paper, we propose a novel Attention-based Injection Network (AIN) to address this issue. Instead of directly learning the relationship between the source and target image, we decompose the process into two accessible modules, namely Semantic-guided Attention Network (SAN) and Pose-guided Attention Network (PAN). SAN is proposed to capture the semantic information which can embed the human attributes into the latent space via the semantic layout. PAN enables a natural re-coupling of the pose and appearance, which can selectively integrate features to complete the human pose transformation. Additionally, a semantic layout loss is proposed to focus on the semantic content similarity between the source and generated images. Compared with other methods, our networks can enforce the local textures and styles consistency between the source and generated image. Experiments show that superior both qualitative and quantitative results are obtained on Market-1501 and DeepFashion datasets. On the basis of AIN, our network can further achieve the data augmentation for person re-identification (Re-ID) with dramatically improving the person Re-ID accuracy.

Introduction

Person image generation aims to transfer a person image with arbitrary pose while retaining the appearance details of the source image. This task, first introduced in [1], has become an emerging popular topic in the computer vision community. It has shown great potential applications in many tasks, such as video generation [2], [3], [4], virtual clothes try-on [5], [6], [7], data augmentation for person-related vision task [8], [9], [10], [56]], etc. The key challenges of this task include the following two aspects: (i) Limited by the non-rigid human body deformation structure, it is difficult to directly transform the spatial misaligned body, particularly when only given local observations of the human body. (ii) Preserving clothing attributes including the texture and style imposes an unexpected difficulty in the generation process.

Notable results of deep learning have provided significant tools to achieve pose transfer task [11], [12], [13], [57]. Early studies in [10], [14], [15], [16] directly adopt a global predictive strategy to transfer pose by utilizing the U-net [17] structure to propagate low-level features. However, considering the non-rigid human body deformation structure, U-net based global method often fail to address the spatial misaligned problem between the source and the target pose. This leads to the generated image suffering from detail deficiency, which always results in over-smooth clothes. In order to improve the performance, more and more researches are devoted to better modeling the body deformation and the local feature transfer. Some methods in [8], [18], [19], [20] adopt the pose attention network to perform a local transfer to fit for the target body topology. Others in [21], [22], [23], [24] fuse the feature representations of appearance and pose by controlling the attention-based decoder. Although the aforementioned approaches can achie-ve better performance, the generated images still encounter some quality problems such as missing texture details and blurry boundary. That results from the model relying on the corresponding extracted feature for the detail reconstruction. Each intermediate image representation is difficult to reveal the region to be transferred. Besides, these approaches in [8], [18], [19], [20], [21], [22], [23], [24] only consider the source image and target pose as inputs, where the clothing textures and the human outline information may be ignored. It should be noted that to preserve the semantic information and replenish fine-grained appearance details is crucial from the viewpoint of the human visual perception.

To preserve the appearance and shape simultaneously, the semantic parsing map in [25], [26], [27], [28], [29], [30], [31] has received considerable attention to serve as an intermediate representation between the source and the synthesized images. Different from the key-point based pose representation, the semantic parsing map automatically provides a foreground mask and captures valuable prior information for the person image generation. However, these generators in [25], [26], [27], [28], [29], [30], [31] directly concatenate the source image, semantic parsing map and target pose as inputs into the basic U-Net, then dedicate in learning a mapping from the concatenated conditions to the target image in a forced manner. It inevitably leads to difficulty in directly transforming the spatial misaligned human body due to the inherent nature of the non-rigid human body.

Motivated by the aforementioned discussion, we propose a novel Attention-based Injection Network (AIN) to address the above challenging problems. Our generator is decomposed into two interconnected aware network, called Seman-tic-guided Attention Network (SAN) and Pose-guided Attention Network (PAN). SAN is designed for semantic encoding, and PAN is constructed for enabling a natural re-coupling of the pose and appearance. Specifically, our model first utilize SAN to capture clothing attributes automatically from the source image via semantic layouts. Then, the component clothing information represented by the semantic code is injected into the designed each block of PAN. Inside each block of PAN, we create an attention mechanism to infer the image regions of interest based on the target pose, and integrate the appearance and pose information with the help of the Adaptive Instance Normalization (AdaIN) layer in the affine transformation form. Each intermediate pose representation characterize the ability to better guide the image generation and retain texture details substantially. The desired image can be reconstructed following target features. Our approach can render textures from the source image into the synthesized map by learning the feature-level mapping, and deal well with the information missing and self-occlusion problems. Moreover, the proposed network benefits in alleviating the ambiguities in inferring unobserved pixels. Experiments on Market-1501 and DeepFashion datasets show that our network can achieve superiority performance on both qualitative and quantitative results.

Existing methods in [13], [16], [22], [30], [32] adopt style loss to enforce the textures of the target and the generated image to be similar around the corresponding pose joints. However, person images with the different pose can be drastically different in appearance under different views. This inevitably demands textures around joints changing with the different pose. Therefore, a new semantic layout loss is proposed to focus on the semantic content similarity between the source and the generated image. The representation of the semantic layout loss is captured by the correlation among the different channels of the image. We first separate component attributes from the person image via semantic layouts. Then it can be implemented by computing the Gram matrix [33] for each patch. Our loss can not only alleviate the influence of the background in pixel-level, but also enforce the local textures and styles consistency between the source and generated image. In summary, the main contributions of this paper can be summarized as follows:

• A novel Attention-based Injection Network (AIN) is proposed to tackle the challenge of pose transfer task with content and style inconsistency. Our method has the ability to preserve and replenish the details of the fine-grained semantic and appearance. It benefits in alleviating ambiguities in inferring unobserved pixels and self-occlusion problems. Experiments demonstrate the rationality and effectiveness of the proposed method.

• We create two attention network which called Semant-ic-guided Attention Network (SAN) and Pose-guided Attention Network (PAN). SAN is designed to automatically capture semantic information from the sou-rce image via semantic layouts. By effectively selecting and deforming important regions of the image code, PAN is proposed to enable a natural re-couping of the image and pose.

• A new semantic layout loss is designed to deal with the spatial misaligned of the source and target image. Our loss is able to enforce the component content and style consistency with well retaining the texture details and attributes.

• Our method exhibits superior performance both in the body shape and keeping clothing textures on benchmarks. Experiments show that it can enrich the pose variations by our method and improve the person re-identification accuracy.

In particular, the structure of the paper is as follows. In Section 2, we present related works on person image generation. In Section 3, we introduce the proposed method in detail. Subsequently, experiments results and extensive analysis are presented on two datasets (Market-1501 [34] and DeepFashion [35]) in Section 4. The application and conclusion are given in Section 5 and Section 6, respectively.