skip to main content
10.1145/3580305.3599313acmconferencesArticle/Chapter ViewAbstractPublication PageskddConference Proceedingsconference-collections
research-article

Domain-Specific Risk Minimization for Domain Generalization

Published: 04 August 2023 Publication History

Abstract

Domain generalization (DG) approaches typically use the hypothesis learned on source domains for inference on the unseen target domain. However, such a hypothesis can be arbitrarily far from the optimal one for the target domain, induced by a gap termed ''adaptivity gap.'' Without exploiting the domain information from the unseen test samples, adaptivity gap estimation and minimization are intractable, which hinders us to robustify a model to any unknown distribution. In this paper, we first establish a generalization bound that explicitly considers the adaptivity gap. Our bound motivates two strategies to reduce the gap: the first one is ensembling multiple classifiers to enrich the hypothesis space, then we propose effective gap estimation methods for guiding the selection of a better hypothesis for the target. The other method is minimizing the gap directly by adapting model parameters using online target samples. We thus propose Domain-specific Risk Minimization (DRM). During training, DRM models the distributions of different source domains separately; for inference, DRM performs online model steering using the source hypothesis for each arriving target sample. Extensive experiments demonstrate the effectiveness of the proposed DRM for domain generalization. Code is available at: https://github.com/yfzhang114/AdaNPC.

Supplementary Material

MP4 File (video1492042883.mp4)
Domain-Specific Risk Minimization for Out-of-Distribution Generalization. Comment: This is a 20-min video.
MP4 File (rtfp0168-2min-promo.mp4)
Short video

References

[1]
Isabela Albuquerque, Jo ao Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. 2019. Generalizing to unseen domains via distribution matching. arXiv preprint arXiv:1911.00804 (2019).
[2]
Isabela Albuquerque, Jo ao Monteiro, Mohammad Darvishi, Tiago H Falk, and Ioannis Mitliagkas. 2020. Adversarial target-invariant representation learning for domain generalization. In Arxiv.
[3]
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019).
[4]
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. 2010. A theory of learning from different domains. Machine learning (2010).
[5]
Shai Ben-David, John Blitzer, Koby Crammer, and Fernando Pereira. 2006. Analysis of representations for domain adaptation. In NIPS.
[6]
Aharon Ben-Tal, Laurent El Ghaoui, and Arkadi Nemirovski. 2009. Robust Optimization. Princeton university press.
[7]
Gilles Blanchard, Aniket Anand Deshmukh, Ürün Dogan, Gyemin Lee, and Clayton Scott. 2021. Domain Generalization by Marginal Transfer Learning. J. Mach. Learn. Res. (2021).
[8]
Xu Chu, Yujie Jin, Wenwu Zhu, Yasha Wang, Xin Wang, Shanghang Zhang, and Hong Mei. 2022. DNA: Domain Generalization with Diversified Neural Averaging. In International Conference on Machine Learning. PMLR, 4010--4034.
[9]
Erick Delage and Yinyu Ye. 2010. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations research (2010).
[10]
Zhengming Ding and Yun Fu. 2017. Deep domain generalization with structured low-rank constraint. IEEE Transactions on Image Processing, Vol. 27, 1 (2017), 304--313.
[11]
Pedro M Domingos. 1997. Why Does Bagging Work? A Bayesian Account and its Implications. In KDD. Citeseer, 155--158.
[12]
Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. 2021. Adaptive methods for real-world domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14340--14349.
[13]
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, Francc ois Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research, Vol. 17, 1 (2016), 2096--2030.
[14]
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. 2015. Domain generalization for object recognition with multi-task autoencoders. In ICCV.
[15]
Ishaan Gulrajani and David Lopez-Paz. 2021. In Search of Lost Domain Generalization. In ICLR.
[16]
Trevor Hastie and Robert Tibshirani. 1993. Varying-coefficient models. Journal of the Royal Statistical Society: Series B (Methodological), Vol. 55, 4 (1993), 757--779.
[17]
Zhaolin Hu and L Jeff Hong. 2013. Kullback-Leibler divergence constrained distributionally robust optimization. Available at Optimization Online (2013).
[18]
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. 2020. Self-challenging improves cross-domain generalization. In ECCV.
[19]
Yusuke Iwasawa and Yutaka Matsuo. 2021. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, Vol. 34 (2021), 2427--2440.
[20]
Samory Kpotufe and Guillaume Martinet. 2018. Marginal singularity, and the benefits of labels in covariate-shift. In Conference On Learning Theory. PMLR, 1882--1886.
[21]
David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. 2021. Out-of-distribution generalization via risk extrapolation (rex). In ICML.
[22]
Dong-Hyun Lee et al. 2013. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML.
[23]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. 2018c. Learning to generalize: Meta-learning for domain generalization. In AAAI.
[24]
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. 2017. Deeper, broader and artier domain generalization. In ICCV.
[25]
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. 2018a. Domain generalization with adversarial feature learning. In CVPR.
[26]
Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. 2018b. Deep domain generalization via conditional invariant adversarial networks. In ECCV.
[27]
Jian Liang, Ran He, and Tieniu Tan. 2023. A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts. arXiv preprint arXiv:2303.15361 (2023).
[28]
Jian Liang, Dapeng Hu, and Jiashi Feng. 2020. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning. PMLR, 6028--6039.
[29]
Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. 2021c. Learning causal semantic representation for out-of-distribution prediction. Advances in Neural Information Processing Systems, Vol. 34 (2021), 6155--6170.
[30]
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. 2021a. Just Train Twice: Improving Group Robustness without Training Group Information. In International Conference on Machine Learning (ICML).
[31]
Xiaofeng Liu, Bo Hu, Linghao Jin, Xu Han, Fangxu Xing, Jinsong Ouyang, Jun Lu, Georges EL Fakhri, and Jonghye Woo. 2021b. Domain generalization under conditional and label shifts via variational bayesian inference. arXiv preprint arXiv:2107.10931 (2021).
[32]
Wang Lu, Jindong Wang, Haoliang Li, Yiqiang Chen, and Xing Xie. 2022. Domain-invariant Feature Exploration for Domain Generalization. Transactions on Machine Learning Research (TMLR) (2022).
[33]
Wang Lu, Jindong Wang, Xinwei Sun, Yiqiang Chen, and Xing Xie. 2023. Out-of-distribution Representation Learning for Time Series Classification. In International Conference on Learning Representations (ICLR).
[34]
Paul Michel, Tatsunori Hashimoto, and Graham Neubig. 2021. Modeling the Second Player in Distributionally Robust Optimization. In International Conference on Learning Representations (ICLR).
[35]
K. Muandet, D. Balduzzi, and B. Schölkopf. 2013. Domain Generalization via Invariant Feature Representation. In ICML.
[36]
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. 2021. Reducing Domain Gap by Reducing Style Bias. In CVPR.
[37]
Lizhen Nie, Mao Ye, Qiang Liu, and Dan Nicolae. 2020. Vcnet and functional targeted regularization for learning causal effects of continuous treatments. ICLR (2020).
[38]
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. 2016. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, Vol. 29 (2016).
[39]
Changdae Oh, Heeji Won, Junhyuk So, Taero Kim, Yewon Kim, Hosik Choi, and Kyungwoo Song. 2022. Learning Fair Representation via Distributional Contrastive Disentanglement. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1295--1305.
[40]
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. 2019. Moment matching for multi-source domain adaptation. In ICCV.
[41]
Alexandre Rame, Corentin Dancette, and Matthieu Cord. 2022. Fishr: Invariant gradient variances for out-of-distribution generalization. In International Conference on Machine Learning. PMLR, 18347--18377.
[42]
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2020. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International conference on learning representations (ICLR).
[43]
Mattia Segu, Alessio Tonioni, and Federico Tombari. 2020. Batch normalization embeddings for deep domain generalization. arXiv preprint arXiv:2011.12672 (2020).
[44]
Weili Shi, Ronghang Zhu, and Sheng Li. 2022b. Pairwise Adversarial Training for Unsupervised Class-imbalanced Domain Adaptation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 1598--1606.
[45]
Yuge Shi, Jeffrey Seely, Philip Torr, Siddharth N, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. 2022a. Gradient Matching for Domain Generalization. In International Conference on Learning Representations. https://openreview.net/forum?id=vDwBW49HmO
[46]
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. 2017. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv:1710.10571 (2017).
[47]
Matthew Staib and Stefanie Jegelka. 2019. Distributionally robust optimization and generalization in kernel methods. Advances in Neural Information Processing Systems (NeurIPS) (2019).
[48]
Petar Stojanov, Zijian Li, Mingming Gong, Ruichu Cai, Jaime G. Carbonell, and Kun Zhang. 2021. Domain Adaptation with Invariant Representation Learning: What Transformations to Learn?. In NeurIPS.
[49]
Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. In ECCV.
[50]
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. 2020. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning. PMLR, 9229--9248.
[51]
Damien Teney, Seong Joon Oh, and Ehsan Abbasnejad. 2022. ID and OOD Performance Are Sometimes Inversely Correlated on Real-world Datasets. arXiv preprint arXiv:2209.00613 (2022).
[52]
Antonio Torralba and Alexei A Efros. 2011. Unbiased look at dataset bias. In CVPR.
[53]
Vladimir Vapnik. 1999. The nature of statistical learning theory. Springer science & business media.
[54]
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2021. Tent: Fully Test-Time Adaptation by Entropy Minimization. In ICLR.
[55]
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. 2022. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering (2022).
[56]
Shujun Wang, Lequan Yu, Kang Li, Xin Yang, Chi-Wing Fu, and Pheng-Ann Heng. 2020. Dofe: Domain-oriented feature embedding for generalizable fundus image segmentation on unseen datasets. IEEE Transactions on Medical Imaging, Vol. 39, 12 (2020), 4237--4248.
[57]
Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. 2020. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 (2020).
[58]
Shiqi Yang, Yaxing Wang, Joost van de Weijer, Luis Herranz, and Shangling Jui. 2021. Generalized source-free domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8978--8987.
[59]
Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. 2022. OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7947--7958.
[60]
Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. 2022d. Towards principled disentanglement for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
[61]
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. 2013. Domain adaptation under target and conditional shift. In International Conference on Machine Learning. PMLR, 819--827.
[62]
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. 2021b. Adaptive risk minimization: Learning to adapt to domain shift. NeurIPS (2021).
[63]
Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng. 2021a. Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision. arXiv preprint arXiv:2107.09249 (2021).
[64]
YiFan Zhang, Feng Li, Zhang Zhang, Liang Wang, Dacheng Tao, and Tieniu Tan. 2022a. Generalizable Person Re-identification Without Demographics. https://openreview.net/forum?id=VNdFPD5wqjh
[65]
YiFan Zhang, Xue Wang, Jian Liang, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. 2023 b. Free Lunch for Domain Adversarial Training: Environment Label Smoothing. International Conference on Learning Representations (ICLR) (2023).
[66]
YiFan Zhang, Hanlin Zhang, Zachary Chase Lipton, Li Erran Li, and Eric Xing. 2022c. Exploring transformer backbones for heterogeneous treatment effect estimation. In NeurIPS ML Safety Workshop.
[67]
Yi-Fan Zhang, Xue Wang, Kexin Jin, Kun Yuan, Zhang Zhang, Liang Wang, Rong Jin, and Tieniu Tan. 2023 a. AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation. ICML (2023).
[68]
Yi-Fan Zhang, Zhang Zhang, Da Li, Zhen Jia, Liang Wang, and Tieniu Tan. 2022b. Learning domain invariant representations for generalizable person re-identification. IEEE Transactions on Image Processing (2022).
[69]
Han Zhao, Remi Tachet Des Combes, Kun Zhang, and Geoffrey Gordon. 2019. On learning invariant representations for domain adaptation. In ICML. PMLR.io

Cited By

View all
  • (2025)Multi-Source Domain Generalization for Machine Remaining Useful Life Prediction via Risk Minimization-Based Test-Time AdaptationIEEE Transactions on Industrial Informatics10.1109/TII.2024.346370521:2(1140-1149)Online publication date: Feb-2025
  • (2025)EDDA:An Efficient Divide-and-Conquer Domain Adapter for Automatics Modulation RecognitionIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2024.345355919:1(140-153)Online publication date: Jan-2025
  • (2025)An adaptive source-free unsupervised domain adaptation method for mechanical fault detectionMechanical Systems and Signal Processing10.1016/j.ymssp.2025.112475228(112475)Online publication date: Apr-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
KDD '23: Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining
August 2023
5996 pages
ISBN:9798400701030
DOI:10.1145/3580305
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 August 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. adaptivity gap
  2. domain generalization
  3. test-time adaptation

Qualifiers

  • Research-article

Funding Sources

Conference

KDD '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,133 of 8,635 submissions, 13%

Upcoming Conference

KDD '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)241
  • Downloads (Last 6 weeks)25
Reflects downloads up to 02 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Multi-Source Domain Generalization for Machine Remaining Useful Life Prediction via Risk Minimization-Based Test-Time AdaptationIEEE Transactions on Industrial Informatics10.1109/TII.2024.346370521:2(1140-1149)Online publication date: Feb-2025
  • (2025)EDDA:An Efficient Divide-and-Conquer Domain Adapter for Automatics Modulation RecognitionIEEE Journal of Selected Topics in Signal Processing10.1109/JSTSP.2024.345355919:1(140-153)Online publication date: Jan-2025
  • (2025)An adaptive source-free unsupervised domain adaptation method for mechanical fault detectionMechanical Systems and Signal Processing10.1016/j.ymssp.2025.112475228(112475)Online publication date: Apr-2025
  • (2025)AdaGPAR: Generalizable Pedestrian Attribute Recognition via Test-time AdaptationMachine Intelligence Research10.1007/s11633-024-1504-6Online publication date: 7-Jan-2025
  • (2024)DGMamba: Domain Generalization via Generalized State Space ModelProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681247(3607-3616)Online publication date: 28-Oct-2024
  • (2024)CrossHAR: Generalizing Cross-dataset Human Activity Recognition via Hierarchical Self-Supervised PretrainingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36595978:2(1-26)Online publication date: 15-May-2024
  • (2024)Diverse Intra- and Inter-Domain Activity Style Fusion for Cross-Person Generalization in Activity RecognitionProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671828(4213-4222)Online publication date: 25-Aug-2024
  • (2024)Federated Learning for Generalization, Robustness, Fairness: A Survey and BenchmarkIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.341886246:12(9387-9406)Online publication date: Dec-2024
  • (2024)Transfer Adaptation Learning for Target Recognition in SAR Images: A SurveyIEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing10.1109/JSTARS.2024.343444817(13577-13601)Online publication date: 2024
  • (2024)Stable Heterogeneous Treatment Effect Estimation across Out-of-Distribution Populations2024 IEEE 40th International Conference on Data Engineering (ICDE)10.1109/ICDE60146.2024.00091(1117-1130)Online publication date: 13-May-2024
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media