research-article

Public Access

CATS: Customizable Abstractive Topic-based Summarization

Authors:

Seyed Ali Bahrainian,

George Zerveas,

Fabio Crestani,

Carsten EickhoffAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 40, Issue 1

Article No.: 5, Pages 1 - 24

https://doi.org/10.1145/3464299

Published: 25 October 2021 Publication History

All formats PDF

Abstract

Neural sequence-to-sequence models are the state-of-the-art approach used in abstractive summarization of textual documents, useful for producing condensed versions of source text narratives without being restricted to using only words from the original text. Despite the advances in abstractive summarization, custom generation of summaries (e.g., towards a user’s preference) remains unexplored. In this article, we present CATS, an abstractive neural summarization model that summarizes content in a sequence-to-sequence fashion while also introducing a new mechanism to control the underlying latent topic distribution of the produced summaries. We empirically illustrate the efficacy of our model in producing customized summaries and present findings that facilitate the design of such systems. We use the well-known CNN/DailyMail dataset to evaluate our model. Furthermore, we present a transfer-learning method and demonstrate the effectiveness of our approach in a low resource setting, i.e., abstractive summarization of meetings minutes, where combining the main available meetings’ transcripts datasets, AMI and International Computer Science Institute(ICSI), results in merely a few hundred training documents.

References

[1]

Mohammad Aliannejadi, Morgan Harvey, Luca Costa, Matthew Pointon, and Fabio Crestani. 2019. Understanding mobile search task relevance and user behaviour in context. In Proceedings of the 2019 Conference on Human Information Interaction and Retrieval. 143–151.

Digital Library

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations, ICLR Conference Track Proceedings.

[3]

Seyed Ali Bahrainian. 2019. Just-In-Time Information Retrieval and Summarization for Personal Assistance. Ph.D. Dissertation. Università della Svizzera italiana.

[4]

Seyed Ali Bahrainian and Fabio Crestani. 2018. Augmentation of human memory: Anticipating topics that continue in the next meeting. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. 150–159.

Digital Library

[5]

Seyed Ali Bahrainian and Andreas Dengel. 2015. Sentiment analysis of texts by capturing underlying sentiment patterns. Web Intelligence 13, 1 (2015), 53–68.

[6]

Seyed Ali Bahrainian, Ida Mele, and Fabio Crestani. 2018. Predicting topics in scholarly papers. In Proceedings of the European Conference on Information Retrieval. Springer, 16–28.

[7]

Michele Banko, Vibhu O. Mittal, and Michael J. Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, 318–325.

Digital Library

[8]

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research 3 (2003), 993–1022.

Digital Library

[9]

Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, Volume 1 (Long Papers), Marilyn A. Walker, Heng Ji, and Amanda Stent (Eds.). Association for Computational Linguistics, 1662–1675.

[10]

Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.675–686.

[11]

Trevor Cohn and Mirella Lapata. 2008. Sentence compression beyond word deletion. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1. Association for Computational Linguistics, 137–144.

Digital Library

[12]

Fabio Crestani and Heather Du. 2006. Written versus spoken queries: A qualitative and quantitative comparative analysis. Journal of the American Society for Information Science and Technology 57, 7 (2006), 881–890.

Digital Library

[13]

Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Proceedings of the Advances in Neural Information Processing Systems. 13042–13054.

Digital Library

[14]

Ferenc Galkó and Carsten Eickhoff. 2018. Biomedical question answering via weighted neural network passage retrieval. In Proceedings of the European Conference on Information Retrieval. Springer, 523–528.

[15]

Sebastian Gehrmann, Yuntian Deng, and Alexander M. Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.4098–4109.

[16]

Alex Graves and Jürgen Schmidhuber. 2005. Framewise phoneme classification with bidirectional LSTM and other neural network architectures. Neural Networks 18, 5–6 (2005), 602–610.

Digital Library

[17]

K. Greff, R. K. Srivastava, J. Koutnik, B. R. Steunebrink, and J. Schmidhuber. 2017. LSTM: A search space odyssey. IEEE Transactions on Neural Networks and Learning Systems 28, 10 (2017), 2222–2232.

[18]

Thomas L. Griffiths and Mark Steyvers. 2004. Finding scientific topics. In Proceedings of the National Academy of Sciences. 5228–5235.

[19]

Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv:1603.06393. Retrieved from https://arxiv.org/abs/1603.06393.

[20]

Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv:1606.08415. Retrieved from https://arxiv.org/abs/1606.08415.

[21]

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the Advances in Neural Information Processing Systems. 1693–1701.

Digital Library

[22]

Wan Ting Hsu, Chieh-Kai Lin, Ming-Ying Lee, Kerui Min, Jing Tang, and Min Sun. 2018. A unified model for extractive and abstractive summarization using inconsistency loss. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. 132–141.

[23]

Wojciech Kryscinski, Romain Paulus, Caiming Xiong, and Richard Socher. 2018. Improving abstraction in text summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1808–1817.

[24]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv:1910.13461. Retrieved from https://arxiv.org/abs/1910.13461.

[25]

Wei Li, Xinyan Xiao, Yajuan Lyu, and Yuanzhuo Wang. 2018. Improving neural abstractive document summarization with explicit information selection modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1787–1796.

[26]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of the Workshop on Text Summarization Branches Out.

[27]

Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.

Digital Library

[28]

Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and Beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning. 280–290.

[29]

Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv:1705.04304.

[30]

Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training.

[31]

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67.

[32]

Nuzhah Gooda Sahib, Anastasios Tombros, and Tony Stockman. 2012. A comparative analysis of the information-seeking behavior of visually impaired and sighted searchers. Journal of the American Society for Information Science and Technology 63, 2 (2012), 377–391.

Digital Library

[33]

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. 1073–1083.

[34]

R. J. Senter and Edgar A. Smith. 1967. Automated readability index. Technical Report. CINCINNATI UNIV OH.

[35]

Guokan Shang, Wensi Ding, Zekun Zhang, Antoine J.-P. Tixier, Polykarpos Meladianos, Michalis Vazirgiannis, and Jean-Pierre Lorré. 2018. Unsupervised abstractive meeting summarization with multi-sentence compression and budgeted submodular maximization. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers. 664–674.

[36]

Paul Thomas, Daniel McDuff, Mary Czerwinski, and Nick Craswell. 2017. MISC: A data set of information-seeking conversations. In Proceedings of the 1st International Workshop on Conversational Approaches to Information Retrieval.

[37]

Anastasios Tombros and Mark Sanderson. 1998. Advantages of query biased summaries in information retrieval. In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2–10.

Digital Library

[38]

Johanne R. Trippas, Damiano Spina, Lawrence Cavedon, Hideo Joho, and Mark Sanderson. 2018. Informing the design of spoken conversational search: Perspective paper. In Proceedings of the 2018 Conference on Human Information Interaction and Retrieval. ACM, 32–41.

Digital Library

[39]

Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.

[40]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Advances in Neural Information Processing Systems. 5998–6008.

Digital Library

[41]

Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Proceedings of the Advances in Neural Information Processing Systems. 2692–2700.

Digital Library

[42]

Chong Wang, David Blei, and David Heckerman. 2008. Continuous time dynamic topic models. In Proceedings of the 24th Conference on Uncertinity in Artificial Intellegence (2008).

Digital Library

[43]

Li Wang, Junlin Yao, Yunzhe Tao, Li Zhong, Wei Liu, and Qiang Du. 2018. A reinforced topic-aware convolutional sequence-to-sequence model for abstractive text summarization. In Proceedings of the 27th International Joint Conference on Artificial Intelligence. 4453–4460.

Digital Library

[44]

Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, and Mingyuan Zhou. 2020. Friendly topic assistant for transformer based abstractive summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. 485–497.

[45]

Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2017. Topic aware neural response generation. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.

Digital Library

[46]

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the International Conference on Machine Learning. 2048–2057.

Digital Library

[47]

Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. ProphetNet: Predicting future n-gram for sequence-to-sequence pre-training. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16–20 November 2020 (Findings of ACL), Vol. EMNLP 2020. 2401–2410.

[48]

David Zajic, Bonnie Dorr, and Richard Schwartz. 2004. Bbn/umd at duc-2004: Topiary. In Proceedings of the HLT-NAACL 2004 Document Understanding Workshop. 112–119.

Cited By

Xia ZSun XWu XPan QMo KPeng ZFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)SummarFlex: Exploring Personalized News Filtering and Reading with Query-Focused Hierarchical SummarizationCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681854(216-222)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681854
Shi KPeng XLu HZhu YNiu Z(2024)Multiple Knowledge-Enhanced Meteorological Social Briefing GenerationIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.329825211:2(2002-2013)Online publication date: Apr-2024
https://doi.org/10.1109/TCSS.2023.3298252
Liu WSun YYu BWang HPeng QHou MGuo HWang HLiu C(2024)Automatic Text Summarization Method Based on Improved TextRank Algorithm and K-Means ClusteringKnowledge-Based Systems10.1016/j.knosys.2024.111447287:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.knosys.2024.111447
Show More Cited By

Index Terms

CATS: Customizable Abstractive Topic-based Summarization
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Natural language generation
  2. Machine learning
    1. Machine learning approaches
      1. Factorization methods
        Latent Dirichlet allocation
      2. Neural networks

Recommendations

Graph-based abstractive biomedical text summarization
Graphical abstract

Display Omitted
Highlights
- A graph generation and frequent itemset mining approach have been used for the generation of extractive summaries.
- The T5 model has been adopted to generate abstractive summaries in the biomedical domain.
- The ROUGE metric has been ...
Abstract
Summarization is the process of compressing a text to obtain its important informative parts. In recent years, various methods have been presented to extract important parts of textual documents to present them in a summarized form. The first ...
Topic Attentional Neural Network for Abstractive Document Summarization
Advances in Knowledge Discovery and Data Mining
Abstract
Abstractive summarization is a renewed and challenging task of document summarization. Recently, neural networks, especially attentional encoder-docoder architecture, have achieved impressive progress in abstractive document summarization. However,...
Abstractive Summarization Improved by WordNet-Based Extractive Sentences
Natural Language Processing and Chinese Computing
Abstract
Recently, the seq2seq abstractive summarization models have achieved good results on the CNN/Daily Mail dataset. Still, how to improve abstractive methods with extractive methods is a good research direction, since extractive methods have their ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 40, Issue 1

January 2022

599 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/3483337

Editor:
Min Zhang
Tsinghua University, China

Issue’s Table of Contents

Copyright © 2021 Association for Computing Machinery.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 October 2021

Accepted: 01 April 2021

Revised: 01 April 2021

Received: 01 July 2020

Published in TOIS Volume 40, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

NSF
SNSF
ODNI
IARPA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
771
Total Downloads

Downloads (Last 12 months)247
Downloads (Last 6 weeks)36

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xia ZSun XWu XPan QMo KPeng ZFarzan RLópez CCardoso Llach DQuercia DMustafa MNiu SWong-Villacrés M(2024)SummarFlex: Exploring Personalized News Filtering and Reading with Query-Focused Hierarchical SummarizationCompanion Publication of the 2024 Conference on Computer-Supported Cooperative Work and Social Computing10.1145/3678884.3681854(216-222)Online publication date: 11-Nov-2024
https://dl.acm.org/doi/10.1145/3678884.3681854
Shi KPeng XLu HZhu YNiu Z(2024)Multiple Knowledge-Enhanced Meteorological Social Briefing GenerationIEEE Transactions on Computational Social Systems10.1109/TCSS.2023.329825211:2(2002-2013)Online publication date: Apr-2024
https://doi.org/10.1109/TCSS.2023.3298252
Liu WSun YYu BWang HPeng QHou MGuo HWang HLiu C(2024)Automatic Text Summarization Method Based on Improved TextRank Algorithm and K-Means ClusteringKnowledge-Based Systems10.1016/j.knosys.2024.111447287:COnline publication date: 16-May-2024
https://dl.acm.org/doi/10.1016/j.knosys.2024.111447
Chen XSong XJing LLi SHu LNie L(2023)Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language ModelACM Transactions on Information Systems10.1145/360636842:2(1-25)Online publication date: 6-Oct-2023
https://dl.acm.org/doi/10.1145/3606368

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents