Boosting unsupervised domain adaptation: A Fourier approach

doi:10.1016/j.knosys.2023.110325

Knowledge-Based Systems

Volume 264, 15 March 2023, 110325

https://doi.org/10.1016/j.knosys.2023.110325 Get rights and content

Abstract

By using unsupervised domain adaptation (UDA), knowledge is transferred from a label-rich source domain to a target domain that contains relevant information but has no labels. Most existing UDA algorithms primarily align domain-invariant features are primarily aligned during training, whereas target-specific information is ignored when learning the domain-invariant features. To address this issue, we attempted to boost the performance of unsupervised domain adaptation using a Fourier approach (FUDA). Specifically. FUDA is inspired by the fact that the amplitude of the Fourier spectrum mainly primarily preserves low-level statistics. Thus, the source domain can be augmented in FUDA to effectively equip with some low-level information in a target domain by fusing the amplitude of the two domains in the Fourier domain. Meanwhile, we propose Fourier transform channel attention, which represents the weight of Fourier transform to capture feature diversity. On the basis of Fourier analysis, we further show that the conventional attention that is built upon global average pooling is a special case of our proposed attention. Our method is evaluated by using four domain adaptation benchmarks, such as Office-31, Office-Home, VisDA-2017 and DomainNet, demonstrating the effectiveness of our FUDA.

Introduction

Deep learning has made significant progress in various vision tasks, such as object detection [1], [2] and semantic segmentation [3], [4]. High-quality training data are required to achieve impressive performance gains. However, in practical scenarios, manually labeling sufficient training data frequently requires considerable manpower and resources costs. Another disadvantage of deep neural networks is the lack of sufficient generalization ability for new datasets of the problem of domain shift [5], [6].

To solve the problem of domain shift, unsupervised domain adaptation (UDA) [7], [8], [9] is typically used as an effective method. The two main types of UDA include discrepancy-based and consensus-based UDA [10], [11], [12], which primarily aim to align the domains distribution by minimizing a well-designed statistical metric. The second one is an adversarial-based method [13], [14] that distinguishes between the two domains using a domain discriminator, and confuses the domain discriminator using a feature extractor. However, these discrepancy-based and adversarial-based methods all directly input the original image into the model, ignoring the processing of the original image.

To address the aforementioned problems, in this paper, we adopted the Fourier approach to boost the performance of Unsupervised Domain Adaptation, dubbed FUDA. Our motivation comes from a well-known property of the Fourier transformation [15], [16], [17], [18]: the phase component of Fourier spectrum preserves high-level semantics of the original signal, while the amplitude component contains low-level statistics. For better understanding, we present example of the images reconstructed from only amplitude information and only phase information, as well as the original image in Fig. 1, Fig. 2. According to Fig. 2, we find that different images have different amplitude components. Meanwhile, from Fig. 1, we find that the amplitude is mainly related to the semantic information of the image. Based on this observations, FDA [19] have recently developed a Fourier-based method for domain adaptation. They propose a simple image translation strategy by replacing the amplitude spectrum of a source image with that of a random target image. By simply training on the amplitude-transferred source images, their method achieves a remarkable performance. Inspired by above work, we further explore Fourier-based methods for domain adaptation, which consists Fourier transform and Fourier channel attention. (1) Fourier transform: we extract the amplitude of the target domain and fuse the amplitude of the two domains, we find that the augmented new image can capture the color and style information of the target domain as shown in Fig. 2. Thus, we fuse the amplitude of the two domains and generate augmented source domain image towards target domain image by inverse Fourier transform. (2) to effectively focus on the core information of the feature, we propose to leverage Fourier transform channel attention instead of the typical attention that is based on global average pooling (GAP) to better capture rich input pattern information. Notably, our proposed FUDA is a versatile approach that can be incorporated into large amount of exiting UDA methods. In experiment section, we incorporate FUDA with the current state-of-the-art UDA methods called SCDA [20] on multiple cross-domain benchmarks to verify the effectiveness of our proposed FUDA approach. On four widely used benchmarks include Office-31, Office-Home, VisDA-2017 and DomainNet, comprehensive experiments validate that our proposed FUDA approach can largely boost the performance of existing algorithms for UDA.

Thus far, the contributions of this paper are summarized as follows:

•
We leverage the Fourier approach to boost the performance of Unsupervised Domain Adaptation (UDA), which solves the domain shift problem in UDA.
•
We reveal that fusing the amplitude of the target domain into the source domain can capture the style information of the target domain, and thus develop a new Fourier transform to augment the source domain and improve the performance of the UDA.
•
We propose a Fourier transform channel attention mechanism that can capture rich input pattern information, which is more suitable for UDA.
•
We conduct extensive experiments to verify our proposed FUDA, which achieve a new SOTA performance on four standard domain adaptation benchmarks.

Section snippets

Related work

Fourier-based Method. The Fourier transform has wide applications in the field of machine learning [21]. Several works have revealed the low-level information of an image where the amplitude is the main concern, such as the color and style of the image. The phase is primarily concerned with the high-level information of the image, such as the object of the image. [19] introduced the Fourier transform perspective into domain adaptation for the first time and trained the model by simply replacing

Methodology

In unsupervised domain adaptation, we have two domains, one is the labeled source domain, denoted as $D_{s}$ , where $y_{i}^{s} \in {1, 2, \dots, C}$ is the labels corresponding to the source domain, and $D_{t}$ denote the target domain. The source domain and the target domain share the same label space, however, their data probability distributions are not the same. When the model trained on the source domain is directly used on the target domain, the performance is often degraded owing to the difference in the

Benchmarks and experimental settings

Office-31 [33] contains 31 types of data, all of which are office data, and the data sources are Amazon (A), Webcam (W) and DSLR (D). It contains 31 categories from 4,110 images shared by three domains. To test our FUDA, we construct all six domain adaptation tasks, i.e., A $\to$ W, …, A $\to$ D

Office-Home [34] is a new dataset released in 2017, containing 65 objects, mainly for research in the field of domain adaptation, including Artistic images (A), Clipart Art (C), Product images (P) and

Conclusion

We have proposed a simple method for domain alignment that can be easily integrated into a learning system that transforms unsupervised domain adaptation into supervised domain adaptation. It is important to pay attention to proper attention, which is why we propose a Fourier channel attention paradigm.

We found our method, despite being simple, outperformed both the baseline and the current state of the art, which is considerably more complex. This suggests that a fast Fourier transform can

CRediT authorship contribution statement

Mengzhu Wang: Conceptualization, Methodology, Software. Shanshan Wang: Visualization, Investigation. Ye Wang: Data curation. Wei Wang: Data curation, Writing – original draft. Tianyi Liang: Software, Validation. Junyang Chen: Supervision. Zhigang Luo: Supervision.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under Grants No. 62106003 and the University Synergy Innovation Program of Anhui Province (GXXT-2021-005).

References (55)

WangM. et al.
Reducing bi-level feature redundancy for unsupervised domain adaptation
Pattern Recognit.
(2023)
WangS. et al.
BP-triplet net for unsupervised domain adaptation: A Bayesian perspective
Pattern Recognit.
(2023)
EllisD.I. et al.
Rapid and quantitative detection of the microbial spoilage of beef by Fourier transform infrared spectroscopy and machine learning
Anal. Chim. Acta
(2004)
AinamJ.-P. et al.
Unsupervised domain adaptation for person re-identification with iterative soft clustering
Knowl.-Based Syst.
(2021)
E. Xie, J. Ding, W. Wang, X. Zhan, H. Xu, P. Sun, Z. Li, P. Luo, Detco: Unsupervised contrastive learning for object...
PapageorgiouC. et al.
A trainable system for object detection
Int. J. Comput. Vis.
(2000)
GuoY. et al.
A review of semantic segmentation using deep neural networks
Int. J. Multimed. Inf. Retr.
(2018)
J. Long, E. Shelhamer, T. Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE...
Ben-DavidS. et al.
Analysis of representations for domain adaptation
Adv. Neural Inf. Process. Syst.
(2007)
Ben-DavidS. et al.
A theory of learning from different domains
Mach. Learn.
(2010)

WangM. et al.

Interbn: Channel fusion for adversarial unsupervised domain adaptation

WangM. et al.

TFC: Transformer fused convolution for adversarial domain adaptation

IEEE Trans. Comput. Soc. Syst.

(2022)

GaninY. et al.

Unsupervised domain adaptation by backpropagation

WangM. et al.

Semantic data augmentation based distance metric learning for domain generalization

LongM. et al.

Conditional adversarial domain adaptation

(2017)

GaninY. et al.

Domain-adversarial training of neural networks

J. Mach. Learn. Res.

(2016)

OppenheimA.V. et al.

The importance of phase in signals

Proc. IEEE

(1981)

OppenheimA. et al.

Phase in speech and pictures

HansenB.C. et al.

Structural sparseness and spatial phase alignment in natural scenes

J. Opt. Soc. Amer. A

(2007)

PiotrowskiL.N. et al.

A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase

Perception

(1982)

Y. Yang, S. Soatto, Fda: Fourier domain adaptation for semantic segmentation, in: Proceedings of the IEEE/CVF...

S. Li, M. Xie, F. Lv, C.H. Liu, J. Liang, C. Qin, W. Li, Semantic concentration for domain adaptation, in: Proceedings...

Q. Xu, R. Zhang, Y. Zhang, Y. Wang, Q. Tian, A Fourier-based Framework for Domain Generalization, in: Proceedings of...

Q. Liu, C. Chen, J. Qin, Q. Dou, P.-A. Heng, Feddg: Federated domain generalization on medical image segmentation via...

HuangJ. et al.

RDA: Robust domain adaptation via Fourier adversarial attacking

(2021)

WeissK. et al.

A survey of transfer learning

J. Big Data

(2016)

TzengE. et al.

Deep domain confusion: Maximizing for domain invariance

(2014)

Cited by (5)

Video Generalized Semantic Segmentation via Non-Salient Feature Reasoning and Consistency
2024, Knowledge-Based Systems
Video semantic segmentation is beneficial for dynamic scene processing in real-world environments, and achieves superior performance on independent and identically distributed data. However, it suffers from performance degradation in environments with various domain styles, which is known as the distribution shift problem. Although some previous studies on image generalized semantic segmentation considered the distribution shift problem, temporal-frame information could not be used to obtain more accurate prediction. Thus, in this study, we explore a new task, known as the video generalized semantic segmentation (VGSS) task, which establishes a connection between continuous frames and domain generalization. We propose a novel method named Non-Salient Feature Reasoning and Consistency (NSFRC) for this task. Specifically, we first define the class-wise non-salient feature, which describes the features of the class-wise non-salient region that carry more generalized information. We then propose a class-wise non-salient feature reasoning strategy to select and enhance generalized channels adaptively. This strategy adopts a new form to use domain-invariant features by treating the domain-invariant features as prior information to assist domain-invariant model learning. Finally, we propose a non-salient centroid alignment loss to alleviate the temporally inconsistent and negative transfer problems in the VGSS task. We also extend our video-based framework to the image generalized semantic segmentation (IGSS) task. Experiments demonstrate that our NSFRC framework yields significant improvements in both the VGSS and IGSS tasks. To explain the idea of this research in a clear and attractive way, we provide the visual abstract shown in Fig. 1.
WCAL: Weighted and center-aware adaptation learning for partial domain adaptation
2024, Engineering Applications of Artificial Intelligence
Partial domain adaptation, which aims to transfer knowledge from a source domain with rich labels to a unlabeled target domain where target class space is a subspace of source class space, is a challenging task in pattern recognition. Previous partial domain adaptation approaches tend to immerse in filtering out anomaly categories by weighting and the importance of ignoring the transferability of generated features. In the light of this, this article proposes a novel partial domain adaptation method, dubbed Weighted and Center-aware Adaptation Learning (WCAL). Specifically, WCAL presents a weighted adversarial learning module learns a category classifier to filter out the outlier categories from source domain. Also, it seeks a domain discriminator for cross-domain to further address the negative transfer. Then, the Center-aware adaptation learning module minimizes the distribution discrepancy across domains, which makes the features more transferability for the adaptation model. Extensive experiments on popular domain adaptation datasets testify that the proposed WCAL approach exceeds state-of-the-art baselines significantly with a large margin, in terms of average classification result, for example, 3.36% and 1.71% on Office-Home and Office-Caltech, respectively.
Casting a BAIT for offline and online source-free domain adaptation
2023, Computer Vision and Image Understanding
We address the source-free domain adaptation (SFDA) problem, where only the source model is available during adaptation to the target domain. We consider two settings: the offline setting where all target data can be visited multiple times (epochs) to arrive at a prediction for each target sample, and the online setting where the target data needs to be directly classified upon arrival. Inspired by diverse classifier based domain adaptation methods, in this paper we introduce a second classifier, but with another classifier head fixed. When adapting to the target domain, the additional classifier initialized from source classifier is expected to find misclassified features. Next, when updating the feature extractor, those features will be pushed towards the right side of the source decision boundary, thus achieving source-free domain adaptation. Experimental results show that the proposed method achieves competitive results for offline SFDA on several benchmark datasets compared with existing DA and SFDA methods, and our method surpasses by a large margin other SFDA methods under online source-free domain adaptation setting.
A Fourier Transform Framework for Domain Adaptation
2024, arXiv
Casting a BAIT for Offline and Online Source-free Domain Adaptation
2020, arXiv

¹: These authors contributed to the work equally and should be regarded as co-first authors.

View full text

Boosting unsupervised domain adaptation: A Fourier approach

Abstract

Introduction

Section snippets

Related work

Methodology

Benchmarks and experimental settings

Conclusion

CRediT authorship contribution statement

Declaration of Competing Interest

Acknowledgments

Pattern Recognit.

Pattern Recognit.

Anal. Chim. Acta

Knowl.-Based Syst.

A trainable system for object detection

Int. J. Comput. Vis.

A review of semantic segmentation using deep neural networks

Int. J. Multimed. Inf. Retr.

Analysis of representations for domain adaptation

Adv. Neural Inf. Process. Syst.

A theory of learning from different domains

Mach. Learn.

Interbn: Channel fusion for adversarial unsupervised domain adaptation

TFC: Transformer fused convolution for adversarial domain adaptation

IEEE Trans. Comput. Soc. Syst.

Unsupervised domain adaptation by backpropagation

Semantic data augmentation based distance metric learning for domain generalization

Conditional adversarial domain adaptation

Domain-adversarial training of neural networks

J. Mach. Learn. Res.

The importance of phase in signals

Proc. IEEE

Phase in speech and pictures

Structural sparseness and spatial phase alignment in natural scenes

J. Opt. Soc. Amer. A

A demonstration of the visual importance and flexibility of spatial-frequency amplitude and phase

Perception

RDA: Robust domain adaptation via Fourier adversarial attacking

A survey of transfer learning

J. Big Data

Deep domain confusion: Maximizing for domain invariance