Cross-modality person re-identification via multi-task learning

Highlights

• We take the initiative to investigate the importance and strategy of exploiting person body information for cross-modality VI-PReID.

• We formulate a multi-task learning model by building each individual branch for a different recognition task relevant to person Re-ID.

• We design a task translation module to transfer the person body information from the person mask prediction branch into the VI-PReID branch.

Abstract

Despite its promising preliminary results, existing cross-modality Visible-Infrared Person Re-IDentification (VI-PReID) models incorporating semantic (person) masks simply use these person masks as selection maps to separate person features from background regions. Such models do not dedicate to extracting more modality-invariant person body features in the VI-PReID network itself, thus leading to suboptimal results in VI-PReID. Differently, we aim to better capture person body information in the VI-PReID network itself for VI-PReID by exploiting the inner relations between person mask prediction and VI-PReID. To this end, a novel multi-task learning model is presented in this paper, where person body features obtained by person mask prediction potentially facilitate the extraction of discriminative modality-shared person body information for VI-PReID. On top of that, considering the task difference between person mask prediction and VI-PReID, we propose a novel task translation sub-network to transfer discriminative person body information, extracted by person mask prediction, into VI-PReID. Doing so enables our model to better exploit discriminative and modality-invariant person body information. Thanks to more discriminative modality-shared features, our method outperforms previous state-of-the-arts by a significant margin on several benchmark datasets. Our intriguing findings validate the effectiveness of extracting discriminative person body features for the VI-PReID task.

Introduction

Person Re-IDentification (PReID) aims at matching a probe pedestrian image with those in a large-scale gallery collected by using some non-overlapping cameras at diverse locations. It is one of the core steps in many computer vision applications, including video surveillance [1] and people tracking [2].

Most existing PReID models [3], [4] concentrate on solving the visible image-based PReID task (VV-PReID), which assumes that visible (RGB) images are captured under good illumination conditions. However, this assumption makes VV-PReID models impractical to deal with real-world applications because visible cameras in such scenarios are incapable of capturing discriminative information under poor lighting conditions. Therefore, most existing surveillance systems opt for dual-mode cameras, where visible cameras provide detailed visual characteristics in daytimes and infrared (IR) cameras capture discriminative information in dark environments. Consequently, recent research focus has shifted to visible-infrared PReID (VI-PReID) problem, overcoming the limitations of existing PReID techniques and further facilitating their real-life applications.

It turns out that extracting discriminative modality-shared person information from the input images is essential to solve two grand challenges in VI-PReID, i.e., cross-modality variations caused by the large modality discrepancy between RGB and thermal images, and intra-modality variations caused by different viewpoints, human poses changing, etc. [6], [7], [8], [9]. Given the person retrieval dataset RAP [5], Fig. 1 summaries most of the pedestrian attributes that may appear in RGB images and IR images, respectively. It can be seen that most modality-shared person features are closely related to person body information. This indicates that person body information is vital to discriminative modality-shared features for VI-PReID. However, most existing VI-PReID models advocate the use of image labels to supervise training, which tends to learn features from the entire image but fails to dig precise modality-shared person body information. Therefore, the overall performance of such systems, especially the feature extraction part, is unsatisfactory. This motivates us to investigate better ways to exploit more modality-shared person body information for VI-PReID.

While many single-modality VV-PReID models have paid attention to person body information, the study of extracting such information from cross-modality VI-PReID is still in its infancy. The major reason is that the transition from VV-PReID to VI-PReID does not seem straightforward. Concretely, as shown in Fig. 2(a), most existing VV-PReID models carry out a two-step approach, where the first step is to use pre-trained models (e.g., pose estimation, key point detection and human parsing) to extract person body information. In the subsequent step, this person body information serves as a feature selection mask to help align body parts for the PReID task. This simple operation of feature selection (multiplication) may work well for single-modality person ReID because it can regulate feature maps to focus on the person body region, such as texture or appearance information of person body.

However, such primitive feature selection strategies suffer when applied to cross-modality VI-PReID due to the following reasons. First, the amount of original modality-invariant person information extracted in VI-PReID models is far less than that of single-modality person information extracted in VV-PReID models. After feature selection, some discriminative person information for VI-PReID may be mistakenly discarded. As a result, the simple feature selection strategy will not boost but may even degrade the performance of a cross-modality VI-PReID model. Secondly, taking only the extracted person body information as selection masks cannot help capture more modality-invariant features that are related to person bodies in the feature extractor of the VI-PReID model. Accordingly, as shown in Fig. 2(c), these VI-PReID models still cannot well exploit the features that contain precise person body information for VI-PReID. The above analysis urges us to come up with a better strategy to extract modality-invariant person information for VI-PReID.

Actually, there are strong inner relations between person mask prediction and VI-PReID. On one hand, VI-PReID can provide person-related semantic information, e.g., view-invariant person information, to aid person mask prediction segmenting persons with different poses, views, etc.. On the other hand, person mask prediction can impel VI-PReID to extract more discriminative modality-shared features, e.g., abundant person body information, which is the key to reducing cross-modality variations as well as intra-modality variations. To this end, as shown in Fig. 2(b), we propose a novel multi-task learning framework, which simultaneously learns two vision tasks from different supervisory signals, plus a uniform loss function, to build an improved feature extraction for the purpose of cross-modality person re-identification. In this framework, some modality-invariant person body information (e.g., precise shapes, contours and appearances) will be captured by exploiting the interactions between the two tasks. Accordingly, as shown in Fig. 2(c) and (d), our proposed VI-PReID model can effectively exploit more person body information in the feature learning stage via multi-task learning than those obtained in a feature selection way. This will greatly boost the performance of VI-PReID.

Meanwhile, although the two tasks, i.e., person mask prediction and VI-PReID, are highly related, they are still quite different. Specifically, person mask prediction aims to segment all the persons in the given images from backgrounds without caring one certain person identity, while VI-PReID aims to identify all the person identities in the given images. Therefore, directly introducing the features extracted by the person mask prediction branch into the VI-PReID branch may obtain suboptimal results. Considering that, a task translation sub-network is specially designed to bootstrap the discriminative information, extracted by person mask prediction task, into the VI-PReID task.

In summary, the main contributions of this paper are as follows:

• We take the initiative to investigate the importance and strategy of exploiting person body information for cross-modality VI-PReID. Eventually, our strategy can enhance the discriminability of extracted modality-shared features and further reduce the cross-modality variations as well as intra-modality variations.

• We formulate a novel multi-task learning model by building each individual branch for a different recognition task relevant to person Re-ID. In doing so, our model can concurrently capture discriminative modality-invariant person body information via person mask prediction and VI-PReID. Unlike existing models, which only employ person body information for feature selection, our model exploits the inner relations between person mask prediction and VI-PReID, thus enabling the interactions of the person body information across the two tasks.

• A task translation sub-network is put in place to transfer discriminative person body information, extracted by the person mask prediction sub-branch into the VI-PReID sub-branch. This reduces the feature discrepancy between the two different tasks, thus leading to better person body information exploitation in the VI-PReID sub-branch.

The rest of this paper is organized as the follows. We briefly describe some previous works related to the visible image based person Re-ID and cross-modality person Re-ID first in Section 2, followed by the details of our newly proposed method in Section 3. Several experiments are conducted to validate the proposed model in Section 4, Finally, in Section 5, a brief conclusion is made.