Abstract
Vision Transformer (ViT) has achieved amazing results in many visual applications where training and testing instances are drawn from the independent and identical distribution (I.I.D.). The performance will drop drastically when the distribution of testing instances is different from that of training ones in real open environments. To tackle this challenge, we propose a Stable Vision Transformer (SViT) for out-of-distribution (OOD) generalization. In particular, the SViT weights the samples to eliminate spurious correlations of token features in Vision Transformer and finally boosts the performance for OOD generalization. According to the structure and feature extraction characteristics of the ViT models, we design two forms of learning sample weights: SViT(C) and SViT(T). To demonstrate the effectiveness of two forms of SViT for OOD generalization, we conduct extensive experiments on the popular PACS and OfficeHome datasets and compare them with SOTA methods. The experimental results demonstrate the effectiveness of SViT(C) and SViT(T) for various OOD generalization tasks.
This work was supported by the National Natural Science Foundation of China (Grant No.62372468), in part by the National Natural Science Foundation of China (Grant No.61671480), Shandong Natural Science Foundation (Grant No. ZR2023MF008) and Qingdao Natural Science Foundation (Grant No. 23-2-1-161-zyyd-jch).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cui, P., Athey, S.: Stable learning establishes some common ground between causal inference and machine learning. Nat. Mach. Intell. 4(2), 110–115 (2022)
Dosovitskiy, A., et al.: An image is worth 16x16 words: transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
Gretton, A., Fukumizu, K., Teo, C., Song, L., Schölkopf, B., Smola, A.: A kernel statistical test of independence. In: Advances in Neural Information Processing Systems 20 (2007)
Huang, Z., Wang, H., Xing, E.P., Huang, D.: Self-challenging improves cross-domain generalization. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 124–140. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_8
Iwasawa, Y., Matsuo, Y.: Test-time classifier adjustment module for model-agnostic domain generalization. Adv. Neural. Inf. Process. Syst. 34, 2427–2440 (2021)
Krueger, D., et al.: Out-of-distribution generalization via risk extrapolation (rex). In: International Conference on Machine Learning, pp. 5815–5826. PMLR (2021)
Kuang, K., Cui, P., Athey, S., Xiong, R., Li, B.: Stable prediction across unknown environments. In: proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1617–1626 (2018)
Kuang, K., Xiong, R., Cui, P., Athey, S., Li, B.: Stable prediction with model misspecification and agnostic distribution shift. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 4485–4492 (2020)
Li, D., Yang, Y., Song, Y.Z., Hospedales, T.M.: Deeper, broader and artier domain generalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5542–5550 (2017)
Li, H., Pan, S.J., Wang, S., Kot, A.C.: Domain generalization with adversarial feature learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5400–5409 (2018)
Li, Y., Mao, H., Girshick, R., He, K.: Exploring plain vision transformer backbones for object detection. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022. ECCV 2022. LNCS, vol. 13669. Springer, Cham. https://doi.org/10.1007/978-3-031-20077-9_17
Liu, Z., et al.: Swin transformer: Hierarchical vision transformer using shifted windows. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012–10022 (2021)
Matsuura, T., Harada, T.: Domain generalization using a mixture of multiple latent domains. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 11749–11756 (2020)
Niu, L., Li, W., Xu, D.: Visual recognition by learning from web data: a weakly supervised domain generalization approach. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774–2783 (2015)
Peters, J., Bühlmann, P., Meinshausen, N.: Causal inference by using invariant prediction: identification and confidence intervals. J. Royal Statist. Soc. Ser. B (Statist. Methodol.) 78(5), 947–1012 (2016)
Shen, Z., Cui, P., Liu, J., Zhang, T., Li, B., Chen, Z.: Stable learning via differentiated variable decorrelation. In: Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2185–2193 (2020)
Shen, Z., Cui, P., Zhang, T., Kunag, K.: Stable learning via sample reweighting. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 5692–5699 (2020)
Strobl, E.V., Zhang, K., Visweswaran, S.: Approximate Kernel-based conditional independence tests for fast non-parametric causal discovery. J. Causal Infer. 7(1), 17 (2019)
Sultana, M., Naseer, M., Khan, M.H., Khan, S., Khan, F.S.: Self-distilled vision transformer for domain generalization. In: Proceedings of the Asian Conference on Computer Vision, pp. 3068–3085 (2022)
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International Conference on Machine Learning, pp. 10347–10357. PMLR (2021)
Träuble, F., et al.: On disentangled representations learned from correlated data. In: International Conference on Machine Learning, pp. 10401–10412. PMLR (2021)
Venkateswara, H., Eusebio, J., Chakraborty, S., Panchanathan, S.: Deep hashing network for unsupervised domain adaptation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5018–5027 (2017)
Wang, Z., Loog, M., Van Gemert, J.: Respecting domain relations: hypothesis invariance for domain generalization. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 9756–9763. IEEE (2021)
Wu, H., et al.: CvT: introducing convolutions to vision transformers. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22–31 (2021)
Xu, R., Zhang, X., Shen, Z., Zhang, T., Cui, P.: A theoretical analysis on independence-driven importance weighting for covariate-shift generalization. In: International Conference on Machine Learning, pp. 24803–24829. PMLR (2022)
Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., Wang, J.: CausalVAE: disentangled representation learning via neural structural causal models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9593–9602 (2021)
Zhang, C., et al.: Delving deep into the generalization of vision transformers under distribution shifts. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7277–7286 (2022)
Zhang, X., Cui, P., Xu, R., Zhou, L., He, Y., Shen, Z.: Deep stable learning for out-of-distribution generalization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5372–5382 (2021)
Zheng, S., et al.: Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6881–6890 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Yu, H., Liu, B., Wang, Y., Zhang, K., Tao, D., Liu, W. (2024). A Stable Vision Transformer for Out-of-Distribution Generalization. In: Liu, Q., et al. Pattern Recognition and Computer Vision. PRCV 2023. Lecture Notes in Computer Science, vol 14432. Springer, Singapore. https://doi.org/10.1007/978-981-99-8543-2_27
Download citation
DOI: https://doi.org/10.1007/978-981-99-8543-2_27
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8542-5
Online ISBN: 978-981-99-8543-2
eBook Packages: Computer ScienceComputer Science (R0)