Abstract
Code-switching (CS) is the phenomenon that occurs when a speaker alternates between two or more languages within an utterance or discourse. In this work, we investigate the existence of code-switching in formal text, namely proceedings of multilingual institutions. Our study is carried out on the Arabic-English code-mixing in a parallel corpus extracted from official documents of United Nations. We build a parallel code-switched corpus with two reference translations one in pure Arabic and the other in pure English. We also carry out a human evaluation of this resource in the aim to use it to evaluate the translation of code-switched documents. To the best of our knowledge, this kind of corpora does not exist. The one we propose is unique. This paper examines several methods to translate code-switched corpus: conventional statistical machine translation, the end-to-end neural machine translation and multitask-learning.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Abidi, K., Menacer, M.A., Smaïli, K.: Calyou: a comparable spoken Algerian corpus harvested from Youtube. In: 18th Annual Conference of the International Communication Association (Interspeech) (2017)
Abidi, K., Smaïli, K.: An empirical study of the Algerian dialect of social network. In: ICNLSSP 2017 - International Conference on Natural Language, Signal and Speech Processing. Casablanca, Morocco, December 2017. https://hal.inria.fr/hal-01659997
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)
Bullock, B.E., Hinrichs, L., Toribio, A.J.: World englishes, code-switching, and convergence. Oxford Handbook of World Englishes (2014)
Carpuat, M.: Mixed language and code-switching in the canadian hansard. In: Proceedings of the First Workshop on Computational Approaches to Code Switching, pp. 107–115 (2014)
Eisele, A., Chen, Y.: MultiUN: a multilingual corpus from united nation documents. In: Tapias, D., et al., (eds.) Proceedings of the Seventh Conference on International Language Resources and Evaluation, pp. 2868–2872. European Language Resources Association (ELRA), May 2010
Gambäck, B., Das, A.: Comparing the level of code-switching in corpora. In: Chair, N.C.C., et al. (eds.) Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA), Paris, France, May 2016
Garg, S., Parekh, T., Jyothi, P.: Dual language models for code mixed speech recognition. CoRR abs/1711.01048 (2017), http://arxiv.org/abs/1711.01048
Landis, J.R., Koch, G.G.: The measurement of observer agreement for categorical data. Biometrics pp. 159–174 (1977)
Molina, G., et al.: Overview for the second shared task on language identification in code-switched data. In: Proceedings of the Second Workshop on Computational Approaches to Code Switching, pp. 40–49 (2016)
Poplack, S.: Sometimes i’ll start a sentence in spanish y termino en espanol: toward a typology of code-switching1. Linguistics 18(7–8), 581–618 (1980)
Toshniwal, S., et al.: Multilingual speech recognition with a single end-to-end model. arXiv preprint arXiv:1711.01694 (2017)
Watanabe, S., Hori, T., Hershey, J.: Language Independent End-to-End Architecture for Joint Language Identification and Speech Recognition, pp. 265–271, December 2017
Yoder, M., Rijhwani, S., Rosé, C., Levin, L.: Code-switching as a social act: the case of Arabic Wikipedia talk pages. In: Proceedings of the Second Workshop on NLP and Computational Social Science, pp. 73–82 (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Menacer, M.A., Langlois, D., Jouvet, D., Fohr, D., Mella, O., Smaïli, K. (2019). Machine Translation on a Parallel Code-Switched Corpus. In: Meurs, MJ., Rudzicz, F. (eds) Advances in Artificial Intelligence. Canadian AI 2019. Lecture Notes in Computer Science(), vol 11489. Springer, Cham. https://doi.org/10.1007/978-3-030-18305-9_40
Download citation
DOI: https://doi.org/10.1007/978-3-030-18305-9_40
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-18304-2
Online ISBN: 978-3-030-18305-9
eBook Packages: Computer ScienceComputer Science (R0)