Learning refined attribute-aligned network with attribute selection for person re-identification

Abstract

Effective person re-identification (Re-ID) is often required in real applications. While most exiting approaches either assume the detected pedestrian bounding box well-aligned or utilize limited human structural information (pose, attention, segmentation) to calibrate the misalignment. However, the value of utilizing attributes for pedestrian alignment is still under explored. Furthermore, the hierarchy of attributes in previous works has been largely ignored, appearance feature and attribute feature are often fused in a rigid way. This directly limits the discriminatory and robustness of feature representation. In this paper, we propose a Refined Attribute-aligned Network (RAN), which consists of a coarse-alignment and a fine-alignment module. First, the pre-trained part and attribute predictor are used to generate body parts and candidate attributes. Then the body parts are used for coarse alignment and the attributes are selected by an agent. The agent is optimized with policy gradient algorithm, which can maximize the accumulative reward to increase the probability of proper attribute selection. Finally, for the fine-alignment, the attribute maps and body part features are aggregated by a bilinear-pooling layer to support accurate Re-ID. Extensive experimental results based on multiple datasets including CUHK03, DukeMTMC and Market-1501 demonstrate the superiority of our method over state-of-the-art methods.

Introduction

Smart camera constitutes one of the most important information technologies impacting various aspects of our everyday life, including how we track and monitor in different types of public spaces. Generally, the main objective of intelligent Re-ID system is to retrieve pedestrian images across non-overlapping cameras over different time. Re-ID becomes an increasingly vital visual analytics task and enjoys a wide range of real applications.

The misalignment of body part (i.e. that the body parts of query images are misaligned across the gallery image, as shown in Fig. 1(a).) can greatly degrade the performance of existing Re-ID systems. To overcome this limitation, recent approaches have tried to leverage the localization information and combine the representation over them [15], [25]. Proper alignment plays an important role in supporting effective Re-ID. As shown in Fig. 1(b), the mainstream alignment strategies used in existing Re-ID methods can be generally divided into three independent groups: attention-based, part-based and pose-based.

A few researchers try to employ attention method to refine the features via deep learning [24], [26], [48], [50], [51], [55]. Basic idea is to take advantage of the cues from high-level semantic to align pedestrian indirectly, which can eliminate the useless noise features and enhance the importance of the meaningful local parts. Nevertheless, the learning of attention is not explicitly supervised, making it much less effective to enhance feature’s quality.

On the other hand, a group of works compute the local representations by partitioning the pedestrian image into cells [12], [39], [43], [52], [56]. For instance, person bounding boxes are segmented into horizontal stripes or grids to extract features. Then, metric measurement is applied based on the relevances among the parts. Whereas these methods subject to the lack of fine-grained part localization within the bounding box, which makes it unreliable for pedestrian alignment.

Another group of strategies utilize pose estimator to achieve proper alignment via detecting the key points of the pedestrian [14], [35], [46], [47], [57]. Pose-guided methods do provide exact key points for calibration. However, these strategies are excessive relied on highly-accurate pose estimation, even state-of-the-art pose estimation frameworks are often error-prone in person re-identification datasets.

Human attributes learning [21], [22], [27], [31], [32], [36] has been proven to be an effective way to improve person retrieval systems. Human attributes describe the visual properties of a distinguishable part of body, clothes or accessories. In fact, attribute features contain detailed localization information, which is complementary to the global and local features. Thus, taking full use of attributes are especially effective for fine-grained Re-ID task. However, conventional methods utilize fixed categories of attributes to aid Re-ID task, which are not capable enough to deal with the great variance of different pedestrian images. The variance can come from either inaccurate pedestrian detection results, or the pedestrians themselves due to different types of clothing, occlusions, poses, or even the camera viewpoints. Thus fusing fixed categories of attributes with pedestrian feature will bring noisy information when the improper attributes are selected. In view of this, we introduce an attribute selection model to determine which attribute to use adaptively. However, this procedure is non-differentiable. To handle this problem, we introduce Reinforcement Learning (RL) techniques to optimize such an attribute selection model. As shown in Fig. 1(c), the proposed RAN contains two alignment modules, i.e. Coarse Alignment module and Fine Alignment module.

Firstly, we pre-train a part predictor and an attribute extractor on RAP [16], which includes fine-labeled attributes within body parts in Section 3.1. Thus the pre-trained part and attribute predictors can generate coarse body parts and candidate attributes. The part predictor outputs body localization grids of a pedestrian to extract part features, which can be used for a coarse alignment in body part level.

In order to mine the high-level semantic cues of attributes, we design an agent network for attribute selection to refine the part features. In particular, we utilize an off-line searching method for objective (attribute selection) strategy to generate reward. The agent is trained in a straightforward and explicitly supervised fashion, which ensures that RAN enjoys high robustness and generalization to handle the misalignment issue. Finally, we compute the bilinear mapping of the part features and selected attribute maps for fine alignment, which can produce the final feature representation for Re-ID. Extensive experiments performed on different large scale datasets including CUHK03 [17], DukeMTMC [30], [59] and Market-1501 [58] demonstrate the effectiveness of the proposed RAN. To summarize, our main contributions are highlighted as follows:

• First, we propose the Refined Attribute-aligned Network (RAN) to handle the misalignment in Re-ID task in a coarse-to-fine fashion. The Coarse Alignment (CA) module preliminarily aligns the human body parts with part features. The Fine Alignment (FA) module utilizes the high-level semantic cues and localization information of attribute feature to further enhance the part features from CA.

• Second, an agent is proposed for attribute selection with reinforcement learning, which can select proper attribute features to fuse with the part-level features via bilinear pooling. The agent provides significant flexibility for attribute selection and further enhance the attribute features for fine alignment.