Abstract
Current taggers assume that input texts are already tokenized, i.e. correctly segmented in tokens or high level information units that identify each individual component of the texts. This working hypothesis is unrealistic, due to the heterogeneous nature of the application texts and their sources. The greatest troubles arise when this segmentation is ambiguous. The choice of the correct segmentation alternative depends on the context, which is precisely what taggers study.
In this work, we develop a tagger able not only to decide the tag to be assigned to every token, but also to decide whether some of them form or not the same term, according to different segmentation alternatives. For this task, we design an extension of the Viterbi algorithm able to evaluate streams of tokens of different lengths over the same structure. We also compare its time and space complexities with those of the classic and iterative versions of the algorithm.
This work has been partially supported by the Spanish Government (under the projects TIC2000-0370-C02-01 and HP2001-0044), and by the Galician Government (under the project PGIDT01PXI10506PN).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brants, T. (1999). Cascaded Markov models. In Proc. of the Ninth Conference of the European Chapter of the Association for Computational Linguistics (EACL 99), Bergen, Norway, pp. 118–125.
Brants, T. (2000). TNT-A statistical part-of-speech tagger. In Proc. of the Sixth Applied Natural Language Processing Conference (ANLP 2000), Seattle, WA.
Graña, J.; Chappelier, J.-C.; Vilares, M. (2001). Integrating external dictionaries into part-of-speech taggers. In Proc. of the Euroconference on Recent Advances in Natural Language Processing (RANLP 2001), Tzigov Chark, Bulgaria, pp. 122–128.
Rozenknop, A.; Silaghi, M. (2001). Algorithme de décodage de treillis selon le critère du coût moyen pour la reconnaissance de la parole. In Actes de la 8ème conférence sur le Traitement Automatique des Langues Naturelles (TALN 2001), Tours, France, pp. 391–396.
Viterbi, A.J. (1967). Error bounds for convolutional codes and an asymptotically optimal decoding algorithm. IEEE Trans. Information Theory, vol. IT-13 (April), pp. 260–269.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Graña, J., Alonso, M.A., Vilares, M. (2002). A Common Solution for Tokenization and Part-of-Speech Tagging. In: Sojka, P., Kopeček, I., Pala, K. (eds) Text, Speech and Dialogue. TSD 2002. Lecture Notes in Computer Science(), vol 2448. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-46154-X_1
Download citation
DOI: https://doi.org/10.1007/3-540-46154-X_1
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-44129-8
Online ISBN: 978-3-540-46154-8
eBook Packages: Springer Book Archive