Cross-Domain Transformer with Adaptive Thresholding for Domain Adaptive Semantic Segmentation

Abstract

The goal of unsupervised domain adaptive semantic segmentation (UDA-SS) is to learn a model using annotated data from the source domain and generate accurate dense predictions for the unlabeled target domain. UDA methods based on Transformer utilize self-attention mechanism to learn features within source and target domains. However, in the presence of significant distribution shift between the two domains, the noisy pseudo-labels could hinder the model’s adaptation to the target domain. In this work, we proposed to incorporate self-attention and cross-domain attention to learn domain-invariant features. Specifically, we design a weight-sharing multi-branch cross-domain Transformer, where the cross-domain branch is used to align domains at the feature level with the aid of cross-domain attention. Moreover, we introduce an adaptive thresholding strategy for pseudo-label selection, which dynamically adjusts the proportion of pseudo-labels that are used in training based on the model’s adaptation status. Our approach guarantees the reliability of the pseudo labels while allowing more target domain samples to contribute to model training. Extensive experiments show that our proposed method consistently outperforms the baseline and achieves competitive results on GTA5$\rightarrow$Cityscapes, Synthia$\rightarrow$Cityscapes, and Cityscapes$\rightarrow$ACDC benchmark.