Abstract
We show a diagnostic evaluation of DIPETT, a broad-coverage parser of English sentences. We consider the TSNLP suite as a diagnostic tool, and propose an alternative broader-coverage test suite of test sentences extracted from Quirk et al. We compare the diagnostic effectiveness of the two suites, and draw a few general conclusions. The evaluation results were used to make significant improvements to DIPETT.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Balkan, L., Meijer, S., Arnold, D., Dauphin, E., Estival, D., Falkedal, K., Lehmann, S., Regnier-Prost, S. (1994). Test Suite Design Guidelines and Methodology. Report to LRE 62-089 (D-WP2.1). University of Essex, UK.
Balkan, L., Meijer, S., Arnold, D., Dauphin, E., Estival, D., Falkedal, K., Lehmann, S., Netter, K., Regnier-Prost, S. (1994). Issues in Test Suite Design. Report to LRE 62-089 (DWP2.1). University of Essex, UK.
Balkan, L., Netter, K., Arnold, D., Meijer, S. (1994). TSNLP: Test Suites for Natural Language Processing. In Proceedings of the Language Engineering Convention, ELSNET, Centre for Cognitive Science, University of Edinburgh (pp. 17–22). Edinburgh, Scotland.
Barker, K., Copeck, T., Delisle, S. and Szpakowicz, S. (1997) “Systematic Construction of a Versatile Case System”, J. Natural Language Engineering, vol. 3, no. 4, 279–315.
Barker, K., Delisle, S., Szpakowicz, S. (1998). Test-Driving Tanka: Evaluating a Semi-Automatic System of Text Analysis for Knowledge Acquisition. In Proceedings of the Twelfth Canadian Conference on Artificial Intelligence (LNAI 1418). Vancouver, Canada.
Carroll, J., Briscoe, E. & Sanfilippo, A. (1998). Parser Evaluation: a Survey and a New Proposal. In Proceedings of the International Conference on Language Resources and Evaluation. Granada, Spain. ftp://cogs.susx.ac.uk/pub/users/johnca/lre98-final.ps.
Cole, R., Mariani, J., Uszkoreit, H., Zaenen, A. & Zue, V. (1996). Survey of the state of the art in human language technology. http://www.cse.ogi.edu/CSLU/HLTsurvey/HLTsurvey.html
Delisle, S. (1994). “Text Processing without A-Priori Domain Knowledge: Semi-Automatic Linguistic Analysis for Incremental Knowledge Acquisition”, Ph.D. thesis, TR-94-02, Department of Computer Science, University of Ottawa.
Delisle, S. and Szpakowicz, S. (1995) “Realistic Parsing: Practical Solutions of Difficult Problems”, Proc PACLING’95, 59–68.
Delisle, S., Barker, K., Copeck, T., and Szpakowicz, S. (1996) “Interactive Semantic Analysis of Technical Texts”, Computational Intelligence vol. 12, no. 2, 273–306.
Delisle, S., Szpakowicz, S. (1997) “Extraction of Predicate-Argument Structures from Texts”, Proc Conf on Recent Advances in Natural Language Processing RANLP97, Bulgaria, September 1997, 318–323.
Estival, D., Falkedal, K., Lehmann, S., Balkan, L.,., Meijer, S., Arnold, D., Regnier-Prost, S., Dauphin, E., Netter, K., Oepen, S. (1994). Test Suite Design: Annotation Scheme. Report to LRE 62-089 (D-WP2.2). University of Essex, UK.
Lehmann, S., Oopen, S. Regnier-Prost, S., Netter, K., Lux, V., Klein, J., Falkedal, L., Fouvry, F., Estival, D., Dauphin, E., Compagnion, H., Baur, J., Balkan, L. & Arnold, D. (1996). TSNLP — test suites for natural language processing. In Proceedings of the International Conference on Computational Linguistics, COLING-96 (pp. 711–716). Copenhagen, Denmark.
Quirk, R. & Greenbaum, S. (1973). A University Grammar of English, Longman.
Quirk, R., Greenbaum, S., Leech, G., Svartvik, J. (1985). A Comprehensive Grammar of the English Language, Longman.
Soames, S. & Permutter, D. (1979). Syntactic Argumentation and the structure of English. University of California Press, Berkeley. Los Angeles, 1979.
Srinivas, B., Sarkar, A., Doran, C. & Hockey, B. (1998). Grammar & Parser Evaluation in the XTAG Project. In Proceedings of the International Conference on Language Resources and Evaluation. Granada, Spain.
Winograd, T. (1983). Language as a Cognitive Process (Volume 1: Syntax), Addison-Wesley.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2000 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Scarlett, E., Szpakowicz, S. (2000). The Power of the TSNLP: Lessons from a Diagnostic Evaluation of a Broad-Coverage Parser. In: Hamilton, H.J. (eds) Advances in Artificial Intelligence. Canadian AI 2000. Lecture Notes in Computer Science(), vol 1822. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-45486-1_12
Download citation
DOI: https://doi.org/10.1007/3-540-45486-1_12
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-67557-0
Online ISBN: 978-3-540-45486-1
eBook Packages: Springer Book Archive