Abstract
The objective of cross-lingual sentence embedding learning is to map sentences into a shared representation space, where semantically similar sentence representations are closer together, while distinct sentence representations exhibit clear differentiation. This paper proposes a novel sentence embedding model called MTLAN, which incorporates multi-task learning and auxiliary networks. The model utilizes the LaBSE model for extracting sentence features and undergoes joint training on tasks related to sentence semantic representation and distance measurement. Furthermore, an auxiliary network is employed to enhance the contextual expression of words within sentences. To address the issue of limited resources for low-resource languages, we construct a pseudo-corpus dataset using a multilingual dictionary for unsupervised learning. We conduct experiments on multiple publicly available datasets, including STS and SICK, to evaluate both monolingual sentence similarity and cross-lingual semantic similarity. The empirical results demonstrate the significant superiority of our proposed model over state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Pham, H., Luong, M.T., Manning, C.D.: Learning distributed representations for multilingual text sequences. In: Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 88–94 (2015)
Qin, L., Ni, M., Zhang, Y., Che, W.: CoSDA-ML: multi-lingual code-switching data augmentation for zero-shot cross-lingual NLP, pp. 3853–3860 (2020). https://doi.org/10.24963/ijcai.2020/533
Conneau, A., Lample, G.: Cross-lingual language model pretraining 32 (2019)
Conneau, A., et al.: Unsupervised cross-lingual representation learning at scale, pp. 8440–8451 (2020)
Liu, J., et al.: RankCSE: Unsupervised sentence representations learning via learning to rank (2023)
Nie, Z., Zhang, R., Mao, Y.: On the inadequacy of optimizing alignment and uniformity in contrastive learning of sentence representations. In: The Eleventh International Conference on Learning Representations (2023)
Agirre, E., Cer, D., Diab, M., Gonzalez-Agirre, A.: Semeval-2012 task 6: a pilot on semantic textual similarity. In: *SEM 2012: The First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385–393 (2012)
Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., Specia, L.: Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055 (2017)
Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., Zamparelli, R., et al.: A sick cure for the evaluation of compositional distributional semantic models. In: LREC, pp. 216–223. Reykjavik (2014)
Artetxe, M., Schwenk, H.: Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Trans. Assoc. Comput. Linguist. 7, 597–610 (2019)
Logeswaran, L., Lee, H.: An efficient framework for learning sentence representations (2018)
Conneau, A., Kiela, D., Schwenk, H., Barrault, L., Bordes, A.: Supervised learning of universal sentence representations from natural language inference data, pp. 670–680 (2017)
Feng, F., Yang, Y., Cer, D., Arivazhagan, N., Wang, W.: Language-agnostic BERT sentence embedding, pp. 878–891, May 2022
Zhang, Y., Yang, Q.: A survey on multi-task learning. IEEE Trans. Knowl. Data Eng. 34(12), 5586–5609 (2021)
Lample, G., Conneau, A., Denoyer, L., Ranzato, M.: Unsupervised machine translation using monolingual corpora only (2018)
Conneau, A., et al.: XNLI: evaluating cross-lingual sentence representations. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475–2485 (2018)
Agirre, E., et al.: Semeval-2014 task 10: multilingual semantic textual similarity. In: Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pp. 81–91 (2014)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding, pp. 4171–4186, June 2019
Yan, Y., Li, R., Wang, S., Zhang, F., Wu, W., Xu, W.: ConSERT: a contrastive framework for self-supervised sentence representation transfer, pp. 5065–5075, August 2021
Gao, T., Yao, X., Chen, D.: SimCSE: simple contrastive learning of sentence embeddings, pp. 6894–6910, November 2021
Wu, X., Gao, C., Lin, Z., Han, J., Wang, Z., Hu, S.: InfoCSE: information-aggregated contrastive learning of sentence embeddings, pp. 3060–3070, December 2022
Jiang, T., et al.: PromptBERT: improving BERT sentence embeddings with prompts, pp. 8826–8837, December 2022
Seonwoo, Y., et al.: Ranking-enhanced unsupervised sentence representation learning (2023)
Goswami, K., Dutta, S., Assem, H., Fransen, T., McCrae, J.P.: Cross-lingual sentence embedding using multi-task learning. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 9099–9113. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, November 2021
Acknowledgments
This work is supported by Natural Science Foundation of Heilongjiang Province under grant number LH2021F015.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, G., Wang, T., Yang, W., Yan, Z., Zhan, K. (2024). MTLAN: Multi-Task Learning and Auxiliary Network for Enhanced Sentence Embedding. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Lecture Notes in Computer Science, vol 14449. Springer, Singapore. https://doi.org/10.1007/978-981-99-8067-3_2
Download citation
DOI: https://doi.org/10.1007/978-981-99-8067-3_2
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8066-6
Online ISBN: 978-981-99-8067-3
eBook Packages: Computer ScienceComputer Science (R0)