Abstract
Automated evaluation is crucial in the context of automated text summaries, as is the case with evaluation of any of the language technologies. While the quality of a summary is determined by both content and form of a summary, throughout the literature there has been extensive study on the automatic and semi-automatic evaluation of content of summaries and most such applications have been largely successful. What lacks is a careful investigation of automated evaluation of readability aspects of a summary. In this work we dissect readability into five parameters and try to automate the evaluation of grammaticality of text summaries. We use surface level methods like Ngrams and LCS sequence on POS-tag sequences and chunk-tag sequences to capture acceptable grammatical constructions, and these approaches have produced impressive results. Our results show that it is possible to use relatively shallow features to quantify degree of acceptance of grammaticality.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Hirschman, L., Mani, I.: Evaluation (2001)
Flesch, R.: A new readability yardstick. Journal of Applied Psychology 32, 221–233 (1948)
Gunning, R.: The technique of clear writing. McGraw-Hill International Book Co., New York (1952)
Lapata, M.: Probabilistic text structuring: Experiments with sentence ordering. In: Proceedings of the annual meeting of the Association for Computational Linguistics, pp. 545–552. The Association of Computational Linguistics (2003)
Lapata, M., Barzilay, R.: Automatic evaluation of text coherence: Models and representations. In: Kaelbling, L.P., Saffiotti, A. (eds.) IJCAI, pp. 1085–1090. Professional Book Center (2005)
Barzilay, R., Lapata, M.: Modeling local coherence: An entity-based approach. Comput. Linguist. 34, 1–34 (2008)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL 2002: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, Morristown, NJ, USA, pp. 311–318. Association for Computational Linguistics (2002)
Wan, S., Dale, R., Dras, M.: Searching for grammaticality: Propagating dependencies in the viterbi algorithm. In: Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG 2005). Association for Computational Linguistics (2005)
Mutton, A., Dras, M., Wan, S., Dale, R.: Automatic evaluation of sentence-level fluency. In: ACL. The Association for Computer Linguistics (2007)
Chae, J., Nenkova, A.: Predicting the fluency of text with shallow structural features: Case studies of machine translation and human-written text. In: EACL, pp. 139–147. The Association for Computer Linguistics (2009)
Pitler, E., Nenkova, A.: Revisiting readability: A unified framework for predicting text quality. In: EMNLP, pp. 186–195. ACL (2008)
Feng, L., Elhadad, N., Huenerfauth, M.: Cognitively motivated features for readability assessment. In: Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009), Athens, Greece, pp. 229–237. Association for Computational Linguistics (2009)
Feng, L.: Automatic readability assessment for people with intellectual disabilities. In: SIGACCESS Accessibility and Computing, pp. 84–91 (2009)
Toutanova, K., Manning, C.D.: Enriching the knowledge sources used in a maximum entropy part-of-speech tagger. In: Proceedings of the 2000 Joint SIGDAT conference on Empirical methods in natural language processing and very large corpora, Morristown, NJ, USA, pp. 63–70. Association for Computational Linguistics (2000)
Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: NAACL 2003: Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, Morristown, NJ, USA, pp. 173–180. Association for Computational Linguistics (2003)
Brants, T.: Tnt: a statistical part-of-speech tagger. In: Proceedings of the sixth conference on Applied natural language processing, Morristown, NJ, USA, pp. 224–231. Association for Computational Linguistics (2000)
Cormen, T.H., Leiserson, C.E., Rivest, R.L.: Introduction to Algorithms. The MIT press and McGraw-Hill (1990)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2010 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Vadlapudi, R., Katragadda, R. (2010). Quantitative Evaluation of Grammaticality of Summaries. In: Gelbukh, A. (eds) Computational Linguistics and Intelligent Text Processing. CICLing 2010. Lecture Notes in Computer Science, vol 6008. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-12116-6_62
Download citation
DOI: https://doi.org/10.1007/978-3-642-12116-6_62
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-12115-9
Online ISBN: 978-3-642-12116-6
eBook Packages: Computer ScienceComputer Science (R0)