Abstract
In this paper, we aim to systematically compare the impact of different pre-processing techniques and different pre-trained word embeddings on the translation quality of a neural machine translation model that translates from the Arabic language to English. For the pre-processing, we compare Arabic segmentation, Arabic normalization, and English lower-casing. For the pre-trained embeddings, we use pre-trained models trained based on three context-independent models; Word2Vec, GloVe, and FastText. Our experimental results show that pre-processing techniques help to improve the translation quality with a gain of BLEU score up to \(+1.91\) point. Furthermore, we find that the impact of pre-trained word embeddings strictly depends on the training data size.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Abdelali, A., Darwish, K., Durrani, N., Mubarak, H.: Farasa: a fast and furious segmenter for Arabic. In: Proceedings of the Demonstrations Session, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, 12-17 June 2016, pp. 11–16. The Association for Computational Linguistics (2016)
Almahairi, A., Cho, K., Habash, N., Courville, A.: First result on Arabic neural machine translation. ArXiv abs/1606.02680 (2016)
Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7-9 May 2015, Conference Track Proceedings (2015)
Bensalah, N., Ayad, H., Adib, A., Ibn El Farouk, A.: LSTM vs. GRU for Arabic machine translation. In: Abraham, A., et al. (eds.) SoCPaR 2020. AISC, vol. 1383, pp. 156–165. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-73689-7_16
Cettolo, M., Girardi, C., Federico, M.: WIT3: web inventory of transcribed and translated talks. In: Proceedings of the 16th Annual conference of the European Association for Machine Translation, EAMT 2012, Trento, Italy, 28-30 May 2012, pp. 261–268. European Association for Machine Translation (2012)
Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, 25-29 October 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724–1734. ACL (2014)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7-9 May 2015, Conference Track Proceedings (2015)
Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003, Edmonton, Canada, 27 May - 1 June 2003. The Association for Computational Linguistics (2003)
Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., Joulin, A.: Advances in pre-training distributed word representations. In: Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018) (2018)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Oudah, M., Almahairi, A., Habash, N.: The impact of preprocessing on Arabic-English statistical and neural machine translation. In: Proceedings of Machine Translation Summit XVII Volume 1: Research Track, MTSummit 2019, Dublin, Ireland, 19-23 August 2019, pp. 214–221. European Association for Machine Translation (2019)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pp. 311-318. ACL 2002, Association for Computational Linguistics, USA (2002)
Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543 (2014)
Qi, Y., Sachan, D., Felix, M., Padmanabhan, S., Neubig, G.: When and why are pre-trained word embeddings useful for neural machine translation? In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 529–535. Association for Computational Linguistics (2018)
Schuster, M., Paliwal, K.K.: Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45(11), 2673–2681 (1997)
Soliman, A.B., Eissa, K., El-Beltagy, S.R.: Aravec: A set of Arabic word embedding models for use in Arabic NLP. In: Third International Conference On Arabic Computational Linguistics, ACLING 2017, 5-6 November 2017, Dubai, United Arab Emirates. Procedia Computer Science, vol. 117, pp. 256–265. Elsevier (2017)
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15(1), 1929–1958 (2014)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems. vol. 27. Curran Associates, Inc. (2014)
Wang, C., Nulty, P., Lillis, D.: A comparative study on word embeddings in deep learning for text classification. In: NLPIR 2020: 4th International Conference on Natural Language Processing and Information Retrieval, Seoul, Republic of Korea, 18-20 December 2020, pp. 37–46. ACM (2020)
Zakraoui, J., Saleh, M., Al-Maadeed, S., AlJa’am, J.M.: Evaluation of Arabic to English machine translation systems. In: 2020 11th International Conference on Information and Communication Systems (ICICS), pp. 185–190. IEEE (2020)
Zouidine, M., Khalil, M.: A comparative study of pre-trained word embeddings for Arabic sentiment analysis. In: 46th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2022, Los Alamitos, CA, USA, 27 June - 1 July 2022, pp. 1243–1248. IEEE (2022)
Zouidine, M., Khalil, M., Farouk, A.I.E.: Policy gradient for Arabic to English neural machine translation. In: Lazaar, M., Duvallet, C., Touhafi, A., Al Achhab, M. (eds.) Proceedings of the 5th International Conference on Big Data and Internet of Things. BDIoT 2021. Lecture Notes in Networks and Systems, vol. 489, pp. 469–480. Springer, Cham (2021). https://doi.org/10.1007/978-3-031-07969-6_35
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zouidine, M., Khalil, M., El Farouk, A.I. (2023). Pre-processing and Pre-trained Word Embedding Techniques for Arabic Machine Translation. In: Abraham, A., Pllana, S., Casalino, G., Ma, K., Bajaj, A. (eds) Intelligent Systems Design and Applications. ISDA 2022. Lecture Notes in Networks and Systems, vol 715. Springer, Cham. https://doi.org/10.1007/978-3-031-35507-3_12
Download citation
DOI: https://doi.org/10.1007/978-3-031-35507-3_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-35506-6
Online ISBN: 978-3-031-35507-3
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)