Soft pseudo-Label shrinkage for unsupervised domain adaptive person re-identification

Highlights

• We changed the title to “Soft Pseudo-Label Shrinkage for Unsupervised Domain Adaptive Person Re-identification”.

Abstract

One effective way to tackle unsupervised domain adaptation (UDA) on person re-identification (Re-ID) is to use clustering-based self-training approach, where a model is trained with hard pseudo-labels obtained from a clustering method. Using a hard pseudo-label, a sample is assigned to the cluster with the highest probability, which is sensitive to the incorrect clustering result due to imperfect clustering algorithms. Soft pseudo-labels can mitigate this issue by representing the sample with the full range of class probabilities from all clusters. Specifically, soft pseudo-labels comprise probabilities of full range classes, because they consider both the hard samples and easy samples. This will distract the model from learning more discriminative features in the hard examples. To solve this issue, we propose a coarse-to-fine refinement mechanism to produce robust refined soft pseudo-labels by progressively focusing more on the hard samples while less on the easy samples. The proposed refined soft pseudo-labels can be readily integrated into cross-entropy loss as a strong supervision to guide the model to learn more discriminative features. Extensive experiments demonstrate that our proposed method outperforms the state-of-the-art unsupervised domain adaptation approaches on person Re-ID with a considerable margin. Code will be available at: http://github.com/Dingyuan-Zheng/ctf-UDA.

Introduction

Person re-identification (Re-ID) has been attracting great attentions in the past decade due to its wide applications in surveillance systems, such as tracing criminal suspects or missing populations. Given a target person, person Re-ID aims to retrieve all the images of this person captured by various cameras. Although many existing approaches have achieved impressive performance on supervised person Re-ID [1], [2], [3], they depend on a large amount of labeled data which is very costly and expensive to collect in the real-world scenarios. Moreover, these methods perform well on a single dataset (domain) and their performance deteriorates significantly when the test images are from a different dataset (domain) due to the significant distribution mismatch between the source and target domains. To solve these issues, unsupervised domain adaptive person re-identification methods have been proposed to effectively utilize a large amount of unlabelled data in the target domain, and transfer the model discrimination learned from a labeled source dataset (domain) to unlabelled target dataset (domain).

Although unsupervised domain adaptation (UDA) for person Re-ID have been investigated extensively in recent works [4], [5], it is still an open research topic due to two main challenges. One challenge is the limited model transferability due to the large domain misalignment between source and target domains, and the other one is the unknown number of identities and complex intra-domain variations in the unlabelled target domain. The first issue is attempted to be solved by aligning feature distributions between source and target domains [6], [7] or converting images from source domains to follow the style for the images in target domains using the generative adversarial networks (GAN) [8], [9], [10]. To tackle the second issue, images in the target domain are grouped using clustering methods, i.e., $k$-means or DBSCAN [11], and in turn, the clustering generated pseudo-labels are adopted as supervised information to train the network in a supervised manner [12]. Although these approaches achieve competitive results with heuristic training techniques [13], [14], [15], label noise is inevitably introduced due to imperfect clustering algorithms, compromising the final identification performance.

To mitigate the label noise issue, recent works [16], [17], [18] introduce soft label into clustering-based UDA person Re-ID task. A soft label attempts to represent the person identity with multiple class probabilities instead of one hard class index from clustering method. Therefore, the selection of classes and the probability assigned to each class are crucial for a soft label. Early work [16] proposes to assign uniform weights ($1/k$) for each selected top-$k$ nearest neighbors based on feature distance, where each neighbor is with a unique class identity. Recently, Zhang et al. [17] proposes more sophisticated probability allocation mechanism for the top-$k$ nearest neighbors. To further improve the quality of soft labels, Ge et al. [18] proposes to adopt the full range of pseudo-classes generating from clustering methods with auto-learned weights as the soft pseudo-labels, rather than only relying on the nearest $k$ classes. Such labels are produced from the classifiers of the two mutually learned networks.

As pointed out in previous work [19], [20], [21], more discriminative features can be learned if we enforce the model to focus on misclassified (hard) samples. Similarly as in the UDA person Re-ID task, if a person identity is represented by the full range of pseudo-classes with both hard and easy samples, it will hinder the model from learning discriminative features in the target domain. One example to illustrate the easy/hard examples generated from a clustering method is shown in Fig. 1. A person image (red bounding box) is assigned to a cluster, and this cluster is denoted as anchor cluster. The images (red/blue bounding boxes in the anchor cluster) within the anchor cluster are with slight intra-ID appearance differences. However, some other person images (blue bounding boxes in the hard cluster) with the same ground-truth identity are assigned to a different (hard) cluster due to different camera viewpoints and illumination conditions. These images with large intra-ID variance are the hard positive samples of this person. Besides, some person images (green bounding boxes) with different ground-truth identities and with large inter-ID appearance differences are assigned to a distinct (easy) cluster, and these images are regarded as easy samples. We observe that the cluster centroid of the hard cluster is normally closer to the anchor cluster centroid compared with that of the easy cluster. In this paper, we propose to use class centroid distance to classify easy classes and hard classes.

Inspired by the above analysis, we propose a coarse-to-fine refinement network (CTFRN) to generate robust refined soft pseudo-labels, represented by refined pseudo-classes rather than the full range of pseudo-classes. In turn, such refined soft pseudo-labels are applied to the cross-entropy loss as a strong supervision to guide the model to learn more discriminative features focusing on hard samples. Specifically, we first produce initial soft pseudo-labels which are represented by the full range of pseudo-classes as in [18]. Then, we retain the pseudo-classes corresponding to hard samples while discarding the well-classified classes mainly consists of easy samples. This process follows a coarse-to-fine manner as illustrated in Fig. 2. For example, in Fig. 2, person image $P_1$ which is close to both cluster A and B is assigned with hard pseudo-label B, and its initial full-class soft pseudo-label contains its probabilities to be classified to all classes. After performing coarse-to-fine refinement on such a full-class label, the final refined soft pseudo-label of $P_1$ only keeps the probabilities for pseudo-class A and B. For $P_1$, pseudo-class A and B contain hard samples, and learning on hard samples will facilitate the network to learn discriminative features. Benefiting from the proposed coarse-to-fine refinement network (CTFRN), the obtained model maximizes the transferability from the labeled source domain to the unlabelled target domain and it promotes the final identification performance.

The contributions of our work can be summarized into three folds:

• Observing that using hard pseudo-label or full-class soft pseudo-label for UDA on person Re-ID can distract model from learning discriminative features, we propose to refine the soft pseudo-labels by retaining those pseudo-classes containing hard samples while gradually discarding the well-classified classes that consist of easy samples, and thus learn more discriminative features from hard examples.

• We propose a coarse-to-fine progressive soft pseudo-label refine mechanism, where more pseudo-classes are selected in the beginning of the training, while less pseudo-classes are used at the end of the training. We also propose to use temporally averaged pseudo-class centroid to classify easy classes and hard classes, which is more robust than simply using sample features.

• Extensive experiments on four unsupervised domain adaptation Re-ID tasks demonstrate the superiority of our proposed method. Specifically, our method outperforms the state-of-the-art method [18] by 4.9%, 2.1%, 4.0%, and 3.1% in terms of mAP on Duke-to-Market, Market-to-Duke, Duke-to-MSMT, and Market-to-MSMT tasks, respectively.