Abstract
Rapid classification of documents is of high-importance in many multilingual settings (such as international institutions or Internet search engines). This has been, for years, a well-known problem, addressed by different techniques, with excellent results. We address this problem by a simple n-grams based technique, a variation of techniques of this family. Our n-grams-based classification is very robust and successful, even for 20-fold classification, and even for short text strings. We give a detailed study for different lengths of strings and size of n-grams and we explore what classification parameters give the best performance. There is no requirement for vocabularies, but only for a few training documents. As a main corpus, we used a EU set of documents in 20 languages. Experimental comparison shows that our approach gives better results than four other popular approaches.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Beesley, K.R.: Language Identifier: A Computer Program for Automatic Natural-Language Identification of On-Line Text. In: Languages at Crossroads: Proceeding of the 29 th Annual Conference of the American Translator Association, pp. 47–54 (1988)
Bennett, W.R.: Scientific and engineering problem-solving with the computer. Prentice-Hall, Inc., Englewood Cliffs, New Jersey (1976)
Cavnar, W.B., Trenkle, J.M.: N-gram-based text categorization. In: Proceedings of the 1994 Symposium On Document Analysis and Information Retrieval, University of Nevada, Las Vegas (April 1994)
da Silva, J.F., Lopes, G.P.: Identification of document language is not yet a completely solved problem. In: CIMCA. International Conference on Computational Intelligence for Modeling, Control & Automation, IEEE Computer Society Press, Los Alamitos (2006)
Damashek, M.: Gauging similarity with n-grams: Language independent categorization of text. Science 267, 843–848 (1995)
Dunham, M.H.: Data Mining Introduction and Advanced Topics. Southern Methodist University, Pearson Education Inc., New Jersey (2003)
Dunning, T.: Statistical Identification of Language. Technical Report Technical report. CRL MCCS-94-273, Computing Research Lab, New Mexico State University (1994)
Kešelj, V., Peng, F., Cercone, N., Thomas, C.: N-gram-based author profiles for authorship attribution. In: PACLING’03. Proceedings of the Conference Pacific Association for Computational Linguistics, August 2003, Dalhousie University, Halifax, Nova Scotia, Canada (2003)
Martins, B., Silva, M.J.: Language identification in web pages. In: Proceedings of the 2005 ACM symposium on Applied computing, ACM Press, New York (2006)
Pegn, F., Schuurmans, D., Wang, S.: Language and Task Independent Text Categorization with Simple Language Models. In: Proceedings of Human Language Technology Conference, pp. 110–117 (2003)
Schmitt, J.: Trigram-based method of language identification. U.S. Patent number: 5062143 (October 1991)
Sibun, P., Reynar, J.C.: Language identification: Examining the issues. In: Proceedings of 5th Symposium on Document Analysis and Information Retrieval (1996)
Tauritz, D.: Application of n-grams. Department of Computer Science, University of Missouri-Rolla
Tomović, A., Janičić, P., Kešelj, V.: n-Gram-based classification and unsupervised hierarchical clustering of genome sequences. Computer Methods and Programs in Biomedicine 81(2), 137–153 (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Tomović, A., Janičić, P. (2007). A Variant of N-Gram Based Language Classification. In: Basili, R., Pazienza, M.T. (eds) AI*IA 2007: Artificial Intelligence and Human-Oriented Computing. AI*IA 2007. Lecture Notes in Computer Science(), vol 4733. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74782-6_36
Download citation
DOI: https://doi.org/10.1007/978-3-540-74782-6_36
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74781-9
Online ISBN: 978-3-540-74782-6
eBook Packages: Computer ScienceComputer Science (R0)