Abstract
One of the most important prior tasks for robust part-ofspeech tagging is the correct tokenization or segmentation of the texts. This task can involve processes which are much more complex than the simple identification of the different sentences in the text and each of their individual components, but it is often obviated in many current applications.
Nevertheless, this preprocessing step is an indispensable task in practice, and it is particularly difficult to tackle it with scientific precision without falling repeatedly in the analysis of the specific casuistry of every phenomenon detected.
In this work, we have developed a scheme of preprocessing oriented towards the disambiguation and robust tagging of Galician. Nevertheless, it is a proposal of a general architecture that can be applied to other languages, such as Spanish, with very slight modifications.
This work has been partially supported by the European Union (under FEDER project 1FD97-0047-C04-02), by the Spanish Government (under project TIC2000- 0370-C02-01), and by the Galician Government (under project PGIDT99XI10502B).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brants, T. (2000). TNT-A statistical part-of-speech tagger. In Proceedings of the Sixth Applied Natural Language Processing Conference (ANLP’2000), Seattle.
Chanod, J.-P.; Tapanainen, P. (1996). A Non-deterministic Tokeniser for Finite-State Parsing. In Proceedings of the Workshop on Extended finite state models of language (ECAI’96), Budapest.
Grefenstette, G.; Tapanainen, P. (1994). What is a word, What is a sentence? Problems of Tokenization. In Proceedings of 3rd Conference on Computational Lexicongraphy and Text Research (COMPLEX’94), July 7–10.
Mikev, A. (1999). A knowledge-free Method for Capitalized Word Disambiguation. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, June 20–26, Maryland.
Mikev, A. (2000). Document Centered Approach to Text Normalization. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’2000), July 24–28, Athens, pp. 136–143.
Mikev, A. (2000). Tagging Sentence Boundaries. In Proceedings of the First Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL’2000), Seatle, pp. 264–271
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Graña, J., Barcala, F.M., Vilares, J. (2002). Formal Methods of Tokenization for Part-of-Speech Tagging. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2002. Lecture Notes in Computer Science, vol 2276. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45715-1_22
Download citation
DOI: https://doi.org/10.1007/3-540-45715-1_22
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-43219-7
Online ISBN: 978-3-540-45715-2
eBook Packages: Springer Book Archive