Abstract
Multilingual pre-trained representations are not well-aligned by nature, which harms their performance on cross-lingual tasks. Previous methods propose to post-align the multilingual pre-trained representations by multi-view alignment or contrastive learning. However, we argue that both methods are not suitable for the cross-lingual classification objective, and in this paper we propose a simple yet effective method to better align the pre-trained representations. On the basis of cross-lingual data augmentations, we make a minor modification to the canonical contrastive loss, to remove false-negative examples which should not be contrasted. Augmentations with the same class are brought close to the anchor sample, and augmentations with different class are pushed apart. Experiment results on three cross-lingual tasks from XTREME benchmark show our method could improve the transfer performance by a large margin with no additional resource needed. We also provide in-detail analysis and comparison between different post-alignment strategies.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguist. 7, 597–610 (2019)
Cao, S., Kitaev, N., Klein, D.: Multilingual alignment of contextual word representations. In: Proceedings of ICLR (2020)
Chi, Z., et al.: InfoXLM: an information-theoretic framework for cross-lingual language model pre-training. In: Proceedings of ACL (2021)
Chi, Z., et al.: XLM-E: cross-lingual language model pre-training via ELECTRA. CoRR (2021)
Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale. In: Proceedings of ACL (2020)
Conneau, A., et al.: Xnli: Evaluating cross-lingual sentence representations. In: Proceedings of EMNLP (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of ACL (2019)
Dou, Z., Neubig, G.: Word alignment by fine-tuning embeddings on parallel corpora. In: Proceedings of ACL (2021)
Gao, T., Yao, X., Chen, D.: SimCSE: simple contrastive learning of sentence embeddings. In: Proceedings of EMNLP (2021)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of CVPR (2020)
Hu, J., Ruder, S., Siddhant, A., Neubig, G., Firat, O., Johnson, M.: Xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization. CoRR abs/2003.11080 (2020)
Khosla, P., et al.: Supervised contrastive learning. In: Proceedings of NeurIPS (2020)
Kulshreshtha, S., Redondo Garcia, J.L., Chang, C.Y.: Cross-lingual alignment methods for multilingual BERT: a comparative study. In: Proceedings of ACL (2020)
Lai, S., Huang, H., Jing, D., Chen, Y., Xu, J., Liu, J.: Saliency-based multi-view mixed language training for zero-shot cross-lingual classification. In: Proceedings of ACL (2021)
Lample, G., Conneau, A.: Cross-lingual language model pretraining. Adv. Neural Inf. Process. Syst. (NeurIPS) (2019)
Lample, G., Conneau, A., Denoyer, L., Ranzato, M.: Unsupervised machine translation using monolingual corpora only. In: Proceedings of ICLR (2018)
Liang, Y., et al.: Xglue: a new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv (2020)
van den Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. CoRR (2018)
Pan, L., Hang, C.W., Qi, H., Shah, A., Potdar, S., Yu, M.: Multilingual BERT post-pretraining alignment. In: Proceedings of ACL (2021)
Qin, L., Ni, M., Zhang, Y., Che, W.: CoSDA-ML: multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP. In: Proceedings of IJCAI (2020)
Singh, J., McCann, B., Keskar, N.S., Xiong, C., Socher, R.: XLDA: cross-lingual data augmentation for natural language inference and question answering. CoRR (2019)
Tao, C., et al.: Exploring the equivalence of siamese self-supervised learning via a unified gradient framework. CoRR (2021)
Wang, L., Zhao, W., Liu, J.: Aligning cross-lingual sentence representations with dual momentum contrast. In: Proceedings of EMNLP (2021)
Wang, T., Isola, P.: Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In: Proceedings of ICML (2020)
Wei, X., Weng, R., Hu, Y., Xing, L., Yu, H., Luo, W.: On learning universal representations across languages. In: Proceedings of ICLR (2021)
Yang, Y., Zhang, Y., Tar, C., Baldridge, J.: PAWS-X: a cross-lingual adversarial dataset for paraphrase identification. In: Proceedings of EMNLP (2019)
Yang, Z., Cheng, Y., Liu, Y., Sun, M.: Reducing word omission errors in neural machine translation: a contrastive learning approach. In: Proceedings of ACL (2019)
Zheng, B., et al.: Consistency regularization for cross-lingual fine-tuning. In: Proceedings of ACL (2021)
Acknowledgement
This research work is supported by the National Key R &D Program of China (2020AAA0108001), the National Nature Science Foundation of China (No. 61976016, 61976015 and 61876198) and Toshiba (China) Co., Ltd. The authors would like to thank the anonymous reviewers for their valuable comments and suggestions to improve this paper.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, S. et al. (2022). Supervised Contrastive Learning for Cross-Lingual Transfer Learning. In: Sun, M., et al. Chinese Computational Linguistics. CCL 2022. Lecture Notes in Computer Science(), vol 13603. Springer, Cham. https://doi.org/10.1007/978-3-031-18315-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-031-18315-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-18314-0
Online ISBN: 978-3-031-18315-7
eBook Packages: Computer ScienceComputer Science (R0)