short-paper

Lightweight Meta-Learning for Low-Resource Abstractive Summarization

Authors:

Youngjoong KoAuthors Info & Claims

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2629 - 2633

https://doi.org/10.1145/3477495.3531908

Published: 07 July 2022 Publication History

Abstract

Recently, supervised abstractive summarization using high-resource datasets, such as CNN/DailyMail and Xsum, has achieved significant performance improvements. However, most of the existing high-resource dataset is biased towards a specific domain like news, and annotating document-summary pairs for low-resource datasets is too expensive. Furthermore, the need for low-resource abstractive summarization task is emerging but existing methods for the task such as transfer learning still have domain shifting and overfitting problems. To address these problems, we propose a new framework for low-resource abstractive summarization using a meta-learning algorithm that can quickly adapt to a new domain using small data. For adaptive meta-learning, we introduce a lightweight module inserted into the attention mechanism of a pre-trained language model; the module is first meta-learned with high-resource task-related datasets and then is fine-tuned with the low-resource target dataset. We evaluate our model on 11 different datasets. Experimental results show that the proposed method achieves the state-of-the-art on 9 datasets in low-resource abstractive summarization.

References

[1]

Trapit Bansal, Rishikesh Jha, Tsendsuren Munkhdalai, and Andrew McCallum. 2020. Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 522--534.

[2]

Yi-Syuan Chen and Hong-Han Shuai. 2021. Meta-Transfer Learning for LowResource Abstractive Summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 12692--12700.

[3]

Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A Discourse-Aware Attention Model for Abstractive Summarization of Long Documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). 615--621.

[4]

Alexander Richard Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 1074--1084.

[5]

Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic metalearning for fast adaptation of deep networks. In International conference on machine learning. PMLR, 1126--1135.

[6]

Jiatao Gu, Yong Wang, Yun Chen, Victor OK Li, and Kyunghyun Cho. 2018. MetaLearning for Low-Resource Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3622--3631.

[7]

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems 28 (2015).

[8]

Byeongchang Kim, Hyunwoo Kim, and Gunhee Kim. 2019. Abstractive Summarization of Reddit Posts with Multi-level Memory Networks. In Proceedings of NAACL-HLT. 2519--2531.

[9]

Anastassia Kornilova and Vladimir Eidelman. 2019. BillSum: A Corpus for Automatic Summarization of US Legislation. In Proceedings of the 2nd Workshop on New Frontiers in Summarization. 48--56.

[10]

Mahnaz Koupaee and William Yang Wang. 2018. WikiHow: A Large Scale Text Summarization Dataset. CoRR abs/1810.09305 (2018). arXiv:1810.09305 http://arxiv.org/abs/1810.09305

[11]

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 7871--7880.

[12]

Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74--81.

[13]

Ilya Loshchilov and Frank Hutter. 2018. Decoupled Weight Decay Regularization. In International Conference on Learning Representations.

[14]

Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 1797--1807.

[15]

Cheonbok Park, Yunwon Tae, TaeHee Kim, Soyoung Yang, Mohammad Azam Khan, Lucy Park, and Jaegul Choo. 2021. Unsupervised Neural Machine Translation for Low-Resource Domains via Meta-Learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2888--2901.

[16]

Kun Qian and Zhou Yu. 2019. Domain Adaptive Dialog Generation via Meta Learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2639--2649.

[17]

Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A Neural Attention Model for Abstractive Sentence Summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. 379--389.

[18]

Eva Sharma, Chen Li, and Lu Wang. 2019. BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2204--2213.

[19]

Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning. PMLR, 11328--11339.

[20]

Rui Zhang and Joel Tetreault. 2019. This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 446--456.

Cited By

Chen XLi MGao SCheng XZhu QYan RGao XZhang XHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Flexible and Adaptable Summarization via Expertise SeparationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657789(2018-2027)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657789
Ragazzi LMoro GGuidi SFrisoni G(2024)LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdictsArtificial Intelligence and Law10.1007/s10506-024-09414-wOnline publication date: 9-Sep-2024
https://doi.org/10.1007/s10506-024-09414-w
Li YHuang YHuang WYu JHuang Z(2023)An Abstractive Summarization Model Based on Joint-Attention Mechanism and a Priori KnowledgeApplied Sciences10.3390/app1307461013:7(4610)Online publication date: 5-Apr-2023
https://doi.org/10.3390/app13074610
Show More Cited By

Index Terms

Lightweight Meta-Learning for Low-Resource Abstractive Summarization
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing

Recommendations

Abstractive Summarization Improved by WordNet-Based Extractive Sentences
Natural Language Processing and Chinese Computing
Abstract
Recently, the seq2seq abstractive summarization models have achieved good results on the CNN/Daily Mail dataset. Still, how to improve abstractive methods with extractive methods is a good research direction, since extractive methods have their ...
Abstractive summarization: An overview of the state of the art
Highlights
- AMR Graphs are based upon PropBanks which limits them.
- Deep Learning Models ...
Abstract
Summarization, is to reduce the size of the document while preserving the meaning, is one of the most researched areas among the Natural Language Processing (NLP) community. Summarization techniques, on the basis of whether the exact ...
Deep Learning-Based Abstractive Summarization for Brazilian Portuguese Texts
Intelligent Systems
Abstract
Automatic summarization captures the most relevant information and condenses it into an understandable text in natural language. Such a task can be classified as either extractive or abstractive summarization. Research on Brazilian Portuguese-...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2022

3569 pages

ISBN:9781450387323

DOI:10.1145/3477495

General Chairs:
Enrique Amigo
UNED
,
Pablo Castells
UAM and Amazon
,
Julio Gonzalo
UNED
,
Program Chairs:
Ben Carterette
Spotify
,
J. Shane Culpepper
RMIT University
,
Gabriella Kazai
Waseda University

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Institute of Information & Communications Technology Planning & Evaluation (IITP)
The National Research Foundation of Korea (NRF)
Institute of Information & communications Technology Planning & Evaluation (IITP)

Conference

SIGIR '22

Sponsor:

SIGIR

SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 11 - 15, 2022

Madrid, Spain

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
245
Total Downloads

Downloads (Last 12 months)38
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen XLi MGao SCheng XZhu QYan RGao XZhang XHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Flexible and Adaptable Summarization via Expertise SeparationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657789(2018-2027)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657789
Ragazzi LMoro GGuidi SFrisoni G(2024)LAWSUIT: a LArge expert-Written SUmmarization dataset of ITalian constitutional court verdictsArtificial Intelligence and Law10.1007/s10506-024-09414-wOnline publication date: 9-Sep-2024
https://doi.org/10.1007/s10506-024-09414-w
Li YHuang YHuang WYu JHuang Z(2023)An Abstractive Summarization Model Based on Joint-Attention Mechanism and a Priori KnowledgeApplied Sciences10.3390/app1307461013:7(4610)Online publication date: 5-Apr-2023
https://doi.org/10.3390/app13074610
Huh TKo Y(2023)Efficient framework for low-resource abstractive summarization by meta-transfer learning and pointer-generator networksExpert Systems with Applications10.1016/j.eswa.2023.121029234(121029)Online publication date: Dec-2023
https://doi.org/10.1016/j.eswa.2023.121029
Tang MWang CWang JChen CGao MQian W(2023)ParaSum: Contrastive Paraphrasing for Low-Resource Extractive Text SummarizationKnowledge Science, Engineering and Management10.1007/978-3-031-40289-0_9(106-119)Online publication date: 9-Aug-2023
https://doi.org/10.1007/978-3-031-40289-0_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten