A novel data augmentation scheme for pedestrian detection with attribute preserving GAN

Abstract

Recently pedestrian detection has progressed significantly. However, detecting pedestrians of small scale or in heavy occlusions is still notoriously difficult. Besides, the generalization ability of pre-trained detectors across different datasets remains to be improved. Both of these issues can be attributed to insufficient training data coverage. To cope with this, we present an efficient data augmentation scheme by transferring pedestrians from other datasets into the target scene with a novel Attribute Preserving Generative Adversarial Networks (APGAN). The proposed methodology consists of two steps: pedestrian embedding and style transfer. The former step can simulate pedestrian images of various scale and occlusion, in any pose or background, thus greatly promoting the data variation. The latter step aims to make the generated samples more realistic while guarantee the data coverage. To achieve this goal, we propose APGAN, which pursues both good visual quality and attribute preserving after style transfer. With the proposed method, we can make effective sample augmentations to improve the generalization ability of the trained detectors and enhance its robustness to scale change and occlusions. Extensive experiment results validate the effectiveness and advantages of our method.

Introduction

Pedestrian detection has made great progress in recent years. The performance on public datasets seems pleasing, however, there are still several challenges remaining unresolved. First, pedestrian detection in real applications needs to handle complex lighting conditions, background change, pose variations, occlusions, and scale changes. While the public datasets only cover limited data variation, thus existing methods might be difficult to handle these complex situations, especially for small-scale and occluded pedestrians. What’s more, the domain gap between public datasets also makes the pre-trained detectors generalize poorly across different datasets. As illustrated in Fig. 1, detectors trained from typical training data can hardly deal with the above challenges.

Apart from the deficiency of algorithm, the insufficient training sample coverage is also an important reason for the unpleasing detection performance, particularly for the methods based on deep learning. However, collecting sufficient pedestrian samples is impossible in fact. Therefore, many researchers resort to data augmentation strategies to increase data coverage by making full use of available training data. Common data augmentation methods include random cropping, color jittering, random deformation, etc. However, these methods can only introduce limited data variation, so they improve the performance slightly. Recently, several works propose crop-and-paste data augmentation schemes for object detection [1], [2] and instance segmentation [3], which is, cropping some object foregrounds and pasting them into the target scene by following some rules. However, these methods do not consider whether the pasted patches fit the target scene. As a result, the augmentation samples may look unrealistic and hinder the model learning. Meanwhile, many works [4], [5], [6], [7] leverage the Generative Adversarial Networks (GAN) to conduct domain adaptation in person re-identification problem. They can simulate the target resolution and illumination conditions to some extent, but can hardly generate pedestrian samples of other scale or in other occlusions.

To tackle the above issue, this paper proposes a novel person transferable data augmentation approach for pedestrian detection. As shown in Fig. 2, it involves two stages: (1) embed persons from other datasets into the target scene randomly with the guide of scene semantics; (2) crop pedestrian patches from the pedestrian embedding images, transfer their style into the target domain and embed them back to obtain the generated training samples. The newly generated images and labels can be combined with the original training data for pedestrian detector training. Pedestrian embedding mainly has two benefits. First, it can significantly promote the diversity of pedestrian samples to improve the generalization ability of the detectors. Second, we can simulate certain target data augmentation requirements (e.g. occlusion, small scale) during the embedding stage to promote the detection performance in such special situations.

To make the generated samples look more realistic while guaranteeing the data variation, we propose APGAN in Stage 2, which is a novel variant of CycleGAN [8]. The proposed APGAN can transfer the person style from the source domain into the target domain with the person attributes preserved, such as clothing colors and dress patterns. The preserved attribute can guarantee enough variations of the embedding pedestrians.

Original CycleGAN only considers whether the generated sample looks realistic or not. Compared with it, our proposed APGAN pursuits both good visual quality and attribute preserving by introducing two extra losses. One is Masked Reconstruction Loss (MR-Loss), which restrains the background as well as the attributes of the source persons unchanged during style transfer. Another is Total Variation Loss (TV-Loss), which enforces the smooth spatial color transformation of person images.

In summary, this paper makes the following contributions:

• A novel data augmentation scheme for pedestrian detection is proposed. It can effectively promote the variation of training data by transferring persons from other datasets to the target scene.

• We propose an efficient APGAN by introducing the novel Masked Reconstruction Loss to CycleGAN, to achieve good visual quality as well as attribute preserving after the style transfer.

• Our approach consistently improves the performance of two representative pedestrian detectors, i.e., Adapted FasterRCNN [9] and Asymptotic Localization Fitting Networks (ALFNet) [10], especially in detecting small-scale as well as occluded pedestrians on CityScapes dataset [11]. It also enhances their generalization ability across different datasets including Caltech [12], KITTI [13], INRIA [14], ETH [15], and TUD-Brussels [16].

The paper is organized as follows: In Section 2, we introduce some related work about pedestrian detection, data augmentation and image-to-image translation; In Section 3, we introduce the two steps of our data augmentation scheme: pedestrian embedding and style transfer; In Section 4, we demonstrate the experiment results of our augmentation method on pedestrian detection; In Section 5, we make the conclusion.