Abstract
In this paper, we evaluate using the SRX (Segmentation Rules eXchange) standard for specifying sentence segmentation rules. The rules were originally created for a proofreading tool called LanguageTool. As proofreading tools are quite sensitive to segmentation errors, the underlying segmentation mechanisms must be sufficiently reliable. Even though SRX allows only regular expressions as a means for specifying sentence breaks and exceptions to those breaks, our evaluation shows that it is sufficient for the task, both in terms of the performance of the algorithm used and correctness of results. Moreover, it offers interoperability with different tools, which in turn allows maintaining consistent segmentation in a whole language-processing pipeline. Our rules are available on an open-source license (LGPL), which also helped in receiving valuable community input from our users.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Gale, W.A., Church, K.W.: A Program for Aligning Sentences in Bilingual Corpora. Computational Linguistics 19(1), 75–102 (1993)
Mikheev, A.: Text segmentation. In: The Oxford Handbook of Computational Linguistics, ch. 10, pp. 201–218. Oxford University Press, Oxford (2003)
Palmer, D.D., Hearst, M.A.: Adaptive multilingual sentence boundary disambiguation. Computational Linguistics 23(2), 241–269 (1997)
Schmid, H.: Unsupervised learning of period disambiguation for tokenization. Internal Report, University of Stuttgart (2000)
Przepiórkowski, A.: The IPI PAN Corpus: Preliminary version. IPI PAN, Warszawa (2004)
Miłkowski, M.: Developing an open-source, rule-based proofreading tool. Software – Practice and Experience 40(7), 543–566 (2010)
SRX 2.0 Standard Specification, http://www.lisa.org/fileadmin/standards/srx20.html
TMX 1.4b Standard Specification, http://www.lisa.org/tmx/tmx.htm
RFC4646: Tags for Identifying Languages, http://www.ietf.org/rfc/rfc4646.txt
Rudolf, M.: Metody automatycznej analizy korpusu tekstów polskich. Uniwersytet Warszawski, Warszawa (2004)
Mazur, P.P.: Text segmentation in Polish. In: Proceedings of the 5th International Conference on Intelligent Systems Design and Applications, pp. 43–48. IEEE Computer Society, Washington DC (2005)
Kim, J., Ohta, T., Teteisi, Y., Tsujii, J.: GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics 19, suppl.1, i180– i182 (2003)
Nazarczuk, M.: Wstępne przygotowanie korpusu Słownika frekwencyjnego polszczyzny współczesnej do dystrybucji na CD-ROM. Praca magisterska napisana pod kierunkiem dra hab. Janusza S. Bienia. Warszawa (1997)
Przepiórkowski, A., Górski, R.G., Lewandowska-Tomaszczyk, B., Łaziński, M.: Towards the National Corpus of Polish. In: Proceedings of the 6th Language Resources and Evaluation Conference, Marrakesh (2008)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Miłkowski, M., Lipski, J. (2011). Using SRX Standard for Sentence Segmentation. In: Vetulani, Z. (eds) Human Language Technology. Challenges for Computer Science and Linguistics. LTC 2009. Lecture Notes in Computer Science(), vol 6562. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-20095-3_16
Download citation
DOI: https://doi.org/10.1007/978-3-642-20095-3_16
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-20094-6
Online ISBN: 978-3-642-20095-3
eBook Packages: Computer ScienceComputer Science (R0)