skip to main content
10.1145/3685650.3685671acmconferencesArticle/Chapter ViewAbstractPublication PagesdocengConference Proceedingsconference-collections
short-paper

Assessing the Reliability and Validity of the Measures for Automatic Text Summarization

Published: 18 September 2024 Publication History

Abstract

Automatic Text Summarization (ATS) is a research area that originated in the late 1950s and has gained increasing importance with the surging amount of text data available today. One of the key challenges in this area is how to quantitatively assess the quality of the summaries produced. The three most widely quantitative measures used for this task are: ROUGE, BLEU and BERTScore. This paper attempts to comparatively evaluate the validity and reliability of such measures. The concept of Shannon' entropy from information theory served as background for this work. Experiments were conducted using the CNN corpus, focusing on news articles written in English.

References

[1]
Zakariae Alami Merrouni, Bouchra Frikh, and Brahim Ouhbi. 2023. EXABSUM: a new text summarization approach for generating extractive and abstractive summaries. Journal of Big Data 10, 1 (2023), 163.
[2]
H. P. Edmundson. 1969. New methods in automatic extracting. Journal of the ACM (JACM) 16, 2 (1969), 264--285.
[3]
Wafaa S El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021. Automatic text summarization: A comprehensive survey. Expert systems with applications 165 (2021), 113679.
[4]
Som Gupta and S. K Gupta. 2019. Abstractive summarization: An overview of the state of the art. Expert Systems with Applications 121 (2019), 49--65.
[5]
Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74--81.
[6]
Hui Lin and Vincent Ng. 2019. Abstractive summarization: A survey of the state of the art. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 9815--9822.
[7]
Rafael Dueire Lins, Rafael Ferreira de Mello, and Steve J. Simske. 2020. ACM DocEng'2020 Competition on Extractive Text Summarization. In Proc. of the ACM Symposium on Document Engineering 2020 (Virtual Event, CA, USA) (DocEng '20). Association for Computing Machinery, NY, USA, Article 3, 4 pages.
[8]
Rafael Dueire Lins, Hilario Oliveira, Luciano Cabral, Jamilson Batista, Bruno Tenorio, Rafael Ferreira, Rinaldo Lima, Gabriel de França Pereira e Silva, and Steven J Simske. 2019. The CNN-corpus: A large textual corpus for single-document extractive summarization. In Proceedings of the ACM Symposium on Document Engineering 2019. 1--10.
[9]
H. P. Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development 2, 2 (1958), 159--165.
[10]
F Middleton. 2023. Reliability vs. Validity in Research | Difference, Types and Examples. https://www.scribbr.com/methodology/reliability-vs-validity/ (2023).
[11]
Hilário Oliveira and Rafael Dueire Lins. 2024. Assessing Abstractive and Extractive Methods for Automatic News Summarization. In ACM Symposium on Document Engineering 2024 (DocEng '24), August 20-23, 2024, San Jose, CA, USA. ACM, New York, NY, USA, 10 pages. https://doi.org/10.1145/3685650
[12]
Hilário Oliveira, Rinaldo Lima, Rafael Dueire Lins, Fred Freitas, Marcelo Riss, and Steven J. Simske. 2016. Assessing Concept Weighting in Integer Linear Programming Based Single-document Summarization. In Proceedings of the 2016 ACM Symposium on Document Engineering (Vienna, Austria) (DocEng'16). ACM, New York, NY, USA, 205--208.
[13]
Hilário Oliveira, Rinaldo Lima, Rafael Dueire Lins, Fred Freitas, Marcelo Riss, and Steven J Simske. 2016. A concept-based integer linear programming approach for single-document summarization. In 2016 5th Brazilian Conference on Intelligent Systems (BRACIS). IEEE, 403--408.
[14]
Hilário Oliveira, Rafael Dueire Lins, Rinaldo Lima, Fred Freitas, and Steven J Simske. 2017. A regression-based approach using integer linear programming for single-document summarization. In 2017 IEEE 29th International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 270--277.
[15]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (Philadelphia, Pennsylvania) (ACL '02). Assoc. for Computational Linguistics, USA, 311--318.
[16]
C. Shannon. 1948. A Mathematical Theory of Communication. The Bell System Technical Journal 27 (1948), 379--423.
[17]
Steven Simske and Marie Vans. 2021. Functional Applications of Text Analytics Systems. River Publishers Series in Document Engineering.
[18]
Dima Suleiman and Arafat A. Awajan. 2020. Deep Learning Based Abstractive Text Summarization: Approaches, Datasets, Evaluation Measures, and Challenges. Mathematical Problems in Engineering 2020 (2020), 1--29.
[19]
Ayesha Ayub Syed, Ford Lumban Gaol, and Tokuro Matsuo. 2021. A Survey of the State-of-the-Art Models in Neural Abstractive Text Summarization. IEEE Access 9 (2021), 13248--13265.
[20]
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020. PEGASUS: Pre-Training with Extracted Gap-Sentences for Abstractive Summarization. In Proceedings of the 37th International Conference on Machine Learning (ICML'20). JMLR.org, Article 1051, 12 pages.
[21]
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
[22]
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. 2023. Benchmarking large language models for news summarization. arXiv preprint arXiv:2301.13848 (2023).

Cited By

View all
  • (2024)Assessing Abstractive and Extractive Methods for Automatic News SummarizationProceedings of the ACM Symposium on Document Engineering 202410.1145/3685650.3685664(1-10)Online publication date: 20-Aug-2024

Index Terms

  1. Assessing the Reliability and Validity of the Measures for Automatic Text Summarization

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    DocEng '24: Proceedings of the ACM Symposium on Document Engineering 2024
    August 2024
    131 pages
    ISBN:9798400711695
    DOI:10.1145/3685650
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 18 September 2024

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. BERTScore
    2. BLEU
    3. ROUGE
    4. Text summarization
    5. and CNN-corpus

    Qualifiers

    • Short-paper
    • Research
    • Refereed limited

    Conference

    DocEng '24
    Sponsor:
    DocEng '24: ACM Symposium on Document Engineering 2024
    August 20 - 23, 2024
    CA, San Jose, USA

    Acceptance Rates

    DocEng '24 Paper Acceptance Rate 16 of 27 submissions, 59%;
    Overall Acceptance Rate 194 of 564 submissions, 34%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)58
    • Downloads (Last 6 weeks)5
    Reflects downloads up to 19 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Assessing Abstractive and Extractive Methods for Automatic News SummarizationProceedings of the ACM Symposium on Document Engineering 202410.1145/3685650.3685664(1-10)Online publication date: 20-Aug-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media