Abstract
CALLISTO is a text summarization system that depends on machine learning techniques and is therefore sensitive to pre-established biases that may not be wholly appropriate. We set out to test whether other biases, modifying the space that CALLISTO explores, lead to improvements in the overall quality of the summaries produced. We present an automatic evaluation framework that relies on a summary quality measure proposed by Lin and Hovy. It appears to be the first evaluation of a text summarization system conducted automatically on a large corpus of news stories. We show the practicality of our methodology on a few experiments with the Machine Learning module of CALLISTO. We conclude that this framework gives reliable hints on the adequacy of a bias and could be useful in developing automatic text summarization systems that work with Machine Learning techniques.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Brandow, R., Mitze, K., Rau, L.: Automatic condensation of electronic publications by sentence selection. Information Processing and Management 31(5), 675–685 (1995)
Copeck, T., Japkowicz, N., Szpakowicz, S.: Text Summarization as Controlled Search. In: Canadian AI 2002, pp. 268–280 (2002)
Document Understanding Conferences, National Institute of Standards and Technology, http://duc.nist.gov/
Hull, D.: Using statistical testing in the evaluation of retrieval experiments. In: SIGIR 1993, pp. 329–338 (1993)
Lin, C.-Y.: ROUGE, Recall-Oriented Understudy for Gisting Evaluation, http://www.isi.edu/~cyl/ROUGE/
Lin, C.-Y., Hovy, E.H.: Manual and Automatic Evaluations of Summaries. In: Workshop on Automatic Summarization, ACL 2002 (2002)
Lin, C.-Y., Hovy, E.H.: Automatic Evaluation of Summaries Using N-gram Co-occurrence Statistics. In: HLT-NAACL 2003, pp. 150–157 (2003)
Mani, I.: Automatic Summarization. John Benjamins, Amsterdam (2001)
Papieni, K., Rouckos, S., Ward, T.: Zhu. W.-J. BLEU: a Method for Automatic Evaluation of Machine Translation. IBM Research Report
Rigouste, L.: Evolution of a Text Summarizer in an Automatic Evaluation Framework. Master’s thesis, http://www.site.uottawa.ca/~rigouste/thesis.ps
Rigouste, L., Szpakowicz, S., Japkowicz, N., Copeck, T.: An Automatic Evaluation Framework for Improving a Configurable Text Summarizer. TR-2004-01, SITE, University of Ottawa, http://www.site.uottawa.ca/~szpak/recent_papers/TR-2004-01.pdf
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2004 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Rigouste, L., Szpakowicz, S., Japkowicz, N., Copeck, T. (2004). An Automatic Evaluation Framework for Improving a Configurable Text Summarizer. In: Tawfik, A.Y., Goodwin, S.D. (eds) Advances in Artificial Intelligence. Canadian AI 2004. Lecture Notes in Computer Science(), vol 3060. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-24840-8_49
Download citation
DOI: https://doi.org/10.1007/978-3-540-24840-8_49
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-22004-6
Online ISBN: 978-3-540-24840-8
eBook Packages: Springer Book Archive