Abstract
This paper focuses on part-of-speech (POS, category) tagging based on word probability estimated using morpheme unigrams and category patterns within a word. The word-N-gram-based POS-tagging model is difficult to adapt to agglutinative languages such as Korean, Turkish and Hungarian, among others, due to the high productivity of words. Thus, many of the stochastic studies on Korean POS-tagging have been conducted based on morpheme N-grams. However, the morpheme-N-gram model also has difficulty coping with data sparseness when augmenting contextual information in order to assure sufficient performance. In addition, the model has difficulty conceiving the relationship of morphemes within a word. The present POS-tagging algorithm (a) resolves the data-sparseness problem thanks to a morpheme-unigram-based approach and (b) involves the relationship of morphemes within a word by estimating the weight of the category of a morpheme in a category pattern constituting a word. With the proposed model, a performance similar to that with other models that use more than just the morpheme-unigram model was observed.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Brill, E.: A simple rule-based part of speech tagger. In: Proceedings of the third conference on Applied natural language processing, pp. 152–155 (1992)
Charniak, E., Hendrickson, C., Jacobson, N., Perkowitz, M.: Equations for part-of-speech tagging. In: Proceedings of the Eleventh National Conference on Artificial Intelligence, pp. 784–789 (1993)
Cutting, D., Kupiec, J., Pedersen, J., Sibun, B.: A practical part-of-speech tagger. In: Proceedings of the third conference on Applied natural language processing, pp. 133–140 (1992)
Hakkani-Tür, D.Z., Oflazer, K., Tür, G.: Statistical Morphological Disambiguation for Agglutinative Languages. In: Proceedings of Computational Linguistics, pp. 285–291 (2000)
Lee, G.B., Cha, J.W., Lee, J.H.: Hybrid POS tagging with generalized unknown-word handling. In: Proceedings of the 2nd international workshop on information retrieval with Asian languages, pp. 43–50 (1997)
Lee, S.H.: A Korean part-of-speech tagging system with handling unknown words. Master’s thesis from Korea Advanced Institute of Science and Technology (1995)
Lee, S.Z., Tsujii, J.I., Rim, H.C.: Hidden Markov Model-Based Korean Part-of-speech Tagging Considering High Agglutinativity, Word-Spacing, and Lexical Correlativity. In: Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, pp. 384–391 (2000)
Manning, C.D., Schütze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Cambridge (2000)
Oflazer, K., Kuruöz, Ì.: Tagging and Morphological Disambiguation of Turkish Text. In: Applied Natural Language Processing Conference, pp. 144–149 (1994)
Weischedel, R., Meteer, M., Schwartz, R., Ramshaw, L., Palmucci, J.: Coping with ambiguity and unknown words through probabilistic models. Computational Linguistics, 359-382 (1993)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kang, My., Jung, Sw., Park, Ks., Kwon, Hc. (2007). Part-of-Speech Tagging Using Word Probability Based on Category Patterns. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2007. Lecture Notes in Computer Science, vol 4394. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-70939-8_11
Download citation
DOI: https://doi.org/10.1007/978-3-540-70939-8_11
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-70938-1
Online ISBN: 978-3-540-70939-8
eBook Packages: Computer ScienceComputer Science (R0)