Abstract
The main purpose of this work is to explore the use of Attention-based Recurrent Neural Networks for text language identification. The most common, statistical language identification approaches are effective but need a long text to perform well. To address this problem, we propose the neural model based on the Long Short-Term Memory Neural Network augmented with the Attention Mechanism. The evaluation of the proposed method incorporates tests on texts written in disparate styles and tests on the Twitter posts corpus which comprises short and noisy texts. As a baseline, we apply a widely used statistical method based on a frequency of occurrences of n-grams. Additionally, we investigate the impact of an Attention Mechanism in the proposed method by comparing the results with the outcome of the model without an Attention Mechanism. As a result, the proposed model outperforms the baseline and achieves 97,98% accuracy on the test corpus covering 36 languages and keeps the accuracy also for the Twitter corpus achieving 91,6% accuracy.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
- 4.
- 5.
- 6.
The input vector we use contains 100 characters.
- 7.
Different cell sizes were used during experimentation, including 50, 150, 200, 500 dimensional hidden layers, one and two BiLSTM layers. The best results were achieved for 2 layers, each for 200 neurons.
- 8.
References
Aslam, J.A., Frost, M.: An information-theoretic measure for document similarity. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval, SIGIR 2003, pp. 449–450. ACM, New York (2003). https://doi.org/10.1145/860435.860545
Cavnar, W.B., Trenkle, J.M.: N-gram-based text categorization. In: Proceedings of SDAIR-94, 3rd Annual Symposium on Document Analysis and Information Retrieval, pp. 161–175 (1994)
Chang, J.C., Lin, C.: Recurrent-neural-network for language detection on twitter code-switching corpus. CoRR abs/1412.4314 (2014). http://arxiv.org/abs/1412.4314
Damashek, M.: Gauging similarity with n-grams: Language-independent categorization of text. Science 267(5199), 843–849 (1995). http://gnowledge.sourceforge.net/damashek-ngrams.pdf
Du, C., H.L.: Text classification research with attention-based recurrent neural networks. Int. J. Comput. 13(1) (2018). https://doi.org/10.15837/ijccc.2018.1.3142
Grefenstette, G.: Comparing two language identification schemes. In: 3rd International Conference on Statistical Analysis of Textual Data (1995)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735
Kim, Y., Jernite, Y., Sontag, D., Rush, A.M.: Character-aware neural language models. In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, AAAI 2016, pp. 2741–2749. AAAI Press (2016)
Kocmi, T., Bojar, O.: LanideNN: multilingual language identification on character window. CoRR abs/1701.03338 (2017). http://arxiv.org/abs/1701.03338
Kruengkrai, C., Srichaivattana, P., Sornlertlamvanich, V., Isahara, H.: Language Identification Based On String Kernels, pp. 926–929, November 2005. https://doi.org/10.1109/ISCIT.2005.1567018
Lodhi, H., Saunders, C., Shawe-Taylor, J., Cristianini, N., Watkins, C.: Text classification using string kernels. J. Mach. Learn. Res. 2, 419–444 (2002). https://doi.org/10.1162/153244302760200687
Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421. Association for Computational Linguistics (2015). https://doi.org/10.18653/v1/D15-1166, http://aclweb.org/anthology/D15-1166
Raffel, C., Ellis, D.P.W.: Feed-forward networks with attention can solve some long-term memory problems. CoRR abs/1512.08756 (2015). http://arxiv.org/abs/1512.08756
Selamat, A.: Improved N-grams approach for web page language identification. In: Nguyen, N.T. (ed.) Transactions on Computational Collective Intelligence V. LNCS, vol. 6910, pp. 1–26. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24016-4_1
Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., Xu, B.: Attention-based bidirectional long short-term memory networks for relation classification. In: ACL (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Perełkiewicz, M., Poświata, R. (2019). Text Language Identification Using Attention-Based Recurrent Neural Networks. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2019. Lecture Notes in Computer Science(), vol 11508. Springer, Cham. https://doi.org/10.1007/978-3-030-20912-4_18
Download citation
DOI: https://doi.org/10.1007/978-3-030-20912-4_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-20911-7
Online ISBN: 978-3-030-20912-4
eBook Packages: Computer ScienceComputer Science (R0)