Abstract
This paper deals with a new strategy to evaluate a Natural Language Processing (NLP) complex task using the Turing test. Automatic summarization based on sentence compression requires to asses informativeness and modify inner sentence structures. This is much more intrinsically related with real rephrasing than plain sentence extraction and ranking paradigm so new evaluation methods are needed. We propose a novel imitation game to evaluate Automatic Summarization by Compression (ASC). Rationale of this Turing-like evaluation could be applied to many other NLP complex tasks like Machine translation or Text Generation. We show that a state of the art ASC system can pass such a test and simulate a human summary in 60% of the cases.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Tratz, S., Hovy, E.: Summarisation Evaluation Using Transformed Basic Elements. In: Workshop Text Analysis Conference (TAC 2008), Gaithersburg, MD, USA (2008)
Louis, A., Nenkova, A.: Automatically Evaluating Content Selection in Summarization without Human Models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Singapour, August 6-7, pp. 306–314. ACL (2009)
Saggion, H., Torres-Moreno, J.-M., da Cunha, I., SanJuan, E.: Multilingual summarization evaluation without human models. In: Proceedings of the 23rd International Conference on Computational Linguistics: Posters (COLING 2010), Beijing, Chine, pp. 1059–1067. ACL (2010)
Torres-Moreno, J.-M., Saggion, H., da Cunha, I., SanJuan, E.: Summary Evaluation With and Without References. Polibits: Research Journal on Computer Science and Computer Engineering with Applications 42, 13–19 (2010)
Molina, A., Torres-Moreno, J.-M., SanJuan, E., da Cunha, I., Sierra, G., Velázquez-Morales, P.: Discourse segmentation for sentence compression. In: Batyrshin, I., Sidorov, G. (eds.) MICAI 2011, Part I. LNCS, vol. 7094, pp. 316–327. Springer, Heidelberg (2011)
Molina, A., Torres-Moreno, J.-M., SanJuan, E., da Cunha, I., Martínez, G.E.S.: Discursive sentence compression. In: Gelbukh, A. (ed.) CICLing 2013, Part II. LNCS, vol. 7817, pp. 394–407. Springer, Heidelberg (2013)
Turing, A.M.: Computing machinery and intelligence. Mind 59(236), 433–460 (1950)
Harnad, S.: Minds, Machines and Turing. Journal of Logic, Language and Information 9(4), 425–445 (2000)
da Cunha, I., Torres-Moreno, J.-M., Sierra, G.: On the Development of the RST Spanish Treebank. In: Linguistic Annotation Workshop, pp. 1–10. The Association for Computer Linguistics (2011)
da Cunha, I., SanJuan, E., Torres-Moreno, J.-M., Lloberes, M., Castellón, I.: DiSeg 1.0: The first system for Spanish Discourse Segmentation. Expert Systems with Applications 39(2), 1671–1678 (2012)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Molina, A., SanJuan, E., Torres-Moreno, JM. (2013). A Turing Test to Evaluate a Complex Summarization Task. In: Forner, P., Müller, H., Paredes, R., Rosso, P., Stein, B. (eds) Information Access Evaluation. Multilinguality, Multimodality, and Visualization. CLEF 2013. Lecture Notes in Computer Science, vol 8138. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-40802-1_9
Download citation
DOI: https://doi.org/10.1007/978-3-642-40802-1_9
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-40801-4
Online ISBN: 978-3-642-40802-1
eBook Packages: Computer ScienceComputer Science (R0)