Abstract
Recent studies show that advanced natural language understanding (NLU) models may exploit dataset biases to achieve superior performance on in-distribution datasets but fail to generalize to out-of-distribution datasets that do not contain such biases. Previous works have made promising progress in mitigating dataset biases with an extra model to estimate them. However, these methods rely on prior bias knowledge or tedious model-tuning tricks which may be hard to apply widely. To tackle the above problem, we propose to model biases by shuffling the words of the input sample, as word shuffling can break the semantics that relies on correct word order while keeping the biases that are unaffected. Thanks to word shuffling, we further propose IDOS, a unified debiasing method that enables bias estimation and debiased prediction by one single NLU model. Experimental results on three NLU benchmarks show that despite its simplicity, our method improves the generalization ability of NLU models and achieves a comparable performance to previous debiasing methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bowman, S.R., Angeli, G., Potts, C., Manning, C.D.: A large annotated corpus for learning natural language inference. In: Proceedings of EMNLP (2015)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL (2019)
Dou, S., et al.: Decorrelate irrelevant, purify relevant: overcome textual spurious correlations from a feature perspective. In: Proceedings of COLING (2022)
Feder, A., et al.: Causal inference in natural language processing: estimation, prediction, interpretation and beyond. In: Transactions of the Association for Computational Linguistics (2022)
Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S., Smith, N.A.: Annotation artifacts in natural language inference data. In: Proceedings of NAACL (2018)
Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. (2002)
Joshi, N., Pan, X., He, H.: Are all spurious features in natural language alike? An analysis through a causal lens. In: Proceedings of EMNLP (2022)
Karimi Mahabadi, R., Belinkov, Y., Henderson, J.: End-to-end bias mitigation by modelling biases in corpora. In: Proceedings of ACL (2020)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollar, P.: Focal loss for dense object detection. In: Proceedings of ICCV (2017)
Liu, Y., et al.: Roberta: a robustly optimized bert pretraining approach (2019)
McCoy, T., Pavlick, E., Linzen, T.: Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. In: Proceedings of ACL (2019)
Meissner, J.M., Sugawara, S., Aizawa, A.: Debiasing masks: a new framework for shortcut mitigation in NLU. In: Proceedings of EMNLP (2022)
Min, J., McCoy, R.T., Das, D., Pitler, E., Linzen, T.: Syntactic data augmentation increases robustness to inference heuristics. In: Proceedings of ACL (2020)
Nie, Y., Williams, A., Dinan, E., Bansal, M., Weston, J., Kiela, D.: Adversarial NLI: a new benchmark for natural language understanding. In: Proceedings of ACL (2020)
Sanh, V., Wolf, T., Belinkov, Y., Rush, A.M.: Learning from others’ mistakes: avoiding dataset biases without modeling them. In: Proceedings of ICLR (2021)
Schuster, T., et al.: Towards debiasing fact verification models. In: Proceedings of EMNLP (2019)
Shah, D.S., Schwartz, H.A., Hovy, D.: Predictive biases in natural language processing models: a conceptual framework and overview. In: Proceedings of ACL (2020)
Thorne, J., Vlachos, A., Christodoulopoulos, C., Mittal, A.: Fever: a large-scale dataset for fact extraction and verification. In: Proceedings of NAACL (2018)
Utama, P.A., Moosavi, N.S., Gurevych, I.: Mind the trade-off: debiasing NLU models without degrading the in-distribution performance. In: Proceedings of ACL (2020)
Utama, P.A., Moosavi, N.S., Gurevych, I.: Towards debiasing NLU models from unknown biases. In: Proceedings of EMNLP (2020)
Williams, A., Nangia, N., Bowman, S.: A broad-coverage challenge corpus for sentence understanding through inference. In: Proceedings of NAACL (2018)
Wu, T., Gui, T.: Less is better: recovering intended-feature subspace to robustify NLU models. In: Proceedings of COLING (2022)
Wu, Y., Gardner, M., Stenetorp, P., Dasigi, P.: Generating data to mitigate spurious correlations in natural language inference datasets. In: Proceedings of ACL (2022)
Yaghoobzadeh, Y., Mehri, S., Tachet des Combes, R., Hazen, T.J., Sordoni, A.: Increasing robustness to spurious correlations using forgettable examples. In: Proceedings of EACL (2021)
Ye, J., Ouyang, Y., Wu, Z., Dai, X.: Out-of-distribution generalization challenge in dialog state tracking. In: NeurIPS 2022 Workshop on Distribution Shifts: Connecting Methods and Applications (2022)
Zhang, Y., Baldridge, J., He, L.: Paws: paraphrase adversaries from word scrambling. In: Proceedings of NAACL (2019)
Acknowledgements
The authors would like to thank the anonymous reviewers for their helpful comments. Zhen Wu is the corresponding author. This research is supported by the National Natural Science Foundation of China (No. 61936012, 62206126 and 61976114).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tang, Y., Ouyang, Y., Wu, Z., Zhang, B., Zhang, J., Dai, X. (2023). IDOS: A Unified Debiasing Method via Word Shuffling. In: Liu, F., Duan, N., Xu, Q., Hong, Y. (eds) Natural Language Processing and Chinese Computing. NLPCC 2023. Lecture Notes in Computer Science(), vol 14303. Springer, Cham. https://doi.org/10.1007/978-3-031-44696-2_25
Download citation
DOI: https://doi.org/10.1007/978-3-031-44696-2_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-44695-5
Online ISBN: 978-3-031-44696-2
eBook Packages: Computer ScienceComputer Science (R0)
