Abstract
We live in a digital era - an era of technology, artificial intelligence, big data, and information. The data and information on which we depend to fulfil several daily tasks and decision-making can become overwhelming to deal with and requires effective processing. This can be achieved by designing improved and robust automatic text summarization systems. These systems reduce the size of text document while retaining the salient information. The resurgence of deep learning and its progress from the Recurrent Neural Networks to deep transformer based Pretrained Language Models (PLM) with huge parameters and ample world and common-sense knowledge have opened the doors for huge success and improvement of the Natural Language Processing tasks including Abstractive Text Summarization (ATS). This work surveys the scientific literature to explore and analyze recent research on pre-trained language models and abstractive text summarization utilizing these models. The pretrained language models on abstractive summarization tasks have been analyzed quantitatively based on ROUGE scores on four standard datasets while the analysis of state-of-the-art ATS models has been conducted qualitatively to identify some issues and challenges encountered on finetuning large PLMs on downstream datasets for abstractive summarization. The survey further highlights some techniques that can help boost the performance of these systems. The findings in terms of performance improvement reveal that the models with better performance use either one or a combination of these strategies: (1) Domain Adaptation, (2) Model Augmentation, (3) Stable finetuning, and (4) Data Augmentation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Klymenko, O., Braun, D., Matthes, F.: Automatic Text Summarization: a State-of-the-Art Review, vol. 1, pp. 648–655 (2020)
Han, X., et al.: Pre-Trained Models: Past, Present and Future. AI Open (2021). https://doi.org/10.1016/j.aiopen.2021.08.002
Hasan, T., et al.: XL-Sum: large-scale multilingual abstractive summarization for 44 languages. In: ACL-IJCNLP, pp. 4693–4703 (2021)
Cao, Y., Wan, X., Yao, J., Yu, D.: MultiSumm: Towards a Unified Model for Multi-Lingual Abstractive Summarization. Proc. AAAI 34(01), 11–18 (2020). https://doi.org/10.1609/aaai.v34i01.5328
Syed, A.A., Gaol, F.L., Matsuo, T.: A survey of the state-of-the-art models in neural abstractive text summarization. IEEE Access 9, 13248–13265 (2021)
Devlin, J., et al.: BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT 1, 4171–4186 (2019)
Peters, M.E., et al.: Improving language understanding by generative pre-training. In: OpenAI, pp. 1–10 (2018)
K. Song, X. Tan, T. Qin, J. Lu, and T. Y. Liu, “MASS: Masked sequence to sequence pre-training for language generation,” 36th ICML, pp. 10384–10394, 2019.
Dong, L., et al.: Unified language model pre-training for natural language understanding and generation. In: NIPS., vol. 32 (2019)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL, pp. 7871–7880 (2020)
Zhang, J., Zhao, Y., Saleh, M., Liu, P. J.: PEGASUS: pre-training with extracted gap-sentences for abstractive summarization. In: ICML, pp. 11328–11339 (2020)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1–67 (2020)
Beltagy, I., et al.: Longformer: The Long-Document Transformer (2020)
Zaheer, M., et al.: “Big bird: Transformers for longer sequences,” 2020.
Aksenov, D., et al.: Abstractive text summarization based on language model conditioning and locality modeling. In: 12th International Conference on Language Resources and Evaluation, pp. 6680–6689 (2020)
Zolotareva, E., et al.: Abstractive text summarization using transfer learning. CEUR Workshop Proc. 2718, 75–80 (2020)
Zhao, S., You, F., Liu, Z.: Leveraging pre-trained language model for summary generation on short text. IEEE Access 1–6 (2020)
Hoang, A., et al.: Efficient adaptation of pretrained transformers for abstractive summarization (2019)
Kim, S.: Using pre-Trained Transformer for Better Lay Summarization, pp. 328–335 (2020). https://doi.org/10.18653/v1/2020.sdp-1.38.
Bajaj, A., et al.: Long Document Summarization in a Low Resource Setting using Pretrained Language Models, pp. 71–80 (2021). https://doi.org/10.18653/v1/2021.acl-srw.7.
Xiao, L., Wang, L., He, H.: Modeling Content Importance for Summarization with Pre-trained Language Models, pp. 3606–3611 (2020)
Pilault, J., Li, R., Subramanian, S., Pal, C.: On Extractive and Abstractive Neural Document Summarization with Transformer Language Models, pp. 9308–9319 (2020)
Yu, T., Liu, Z., Fung, P.: AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization, pp. 5892–5904 (2021)
Aghajanyan, A., et al.: Better Fine-tuning by Reducing Representational Collapse (2021)
Liu, Y., Lapata, M: Text Summarization with Pretrained Encoders (2019)
Fabbri, A., et al.: Improving zero and few-shot abstractive summarization with intermediate fine-tuning and data augmentation. In: NAACL, pp. 704–717 (2021)
Gunel, B., Du, J., Conneau, A., Stoyanov, V.: Supervised contrastive learning for pre-trained language model fine-tuning. In: ICLR, pp. 1–21 (2021)
Gururangan, S., et al.: Don’t stop pretraining: adapt language models to domains and tasks. In: ACL, pp. 8342–8360 (2020)
Guo, H., et al.: Multi-source domain adaptation for text classification via DistanceNet-bandits. In: 34th AAAI Conference on Artifical Intelligence, pp. 7830–7838 (2020)
Khandelwal, U., Clark, K., Jurafsky, D., Brain, G.: Sample Efficient Text Summarization Using a Single Pre-Trained Transformer (2018)
Gu, J., Lu, Z., Li, H., Li, V.O.K.: Incorporating copying mechanism in sequence-to-sequence learning. ACL 3, 1631–1640 (2016)
See, A., Liu, P.J., Manning, C.D.: Get to the point: Summarization with pointer-generator networks. ACL 1, 1073–1083 (2017)
Chen, Q., et al.: Distraction-Based Neural Networks for Document Summarization (2016)
Li, S., Lei, D., Qin, P., Wang, W.Y.: Deep reinforcement learning with distributional semantic rewards for abstractive summarization. In: EMNLP, pp. 6038–6044 (2020)
Paulus, R., Xiong, C., Socher, R.: A deep reinforced model for abstractive summarization. In: 6th ICLR, pp. 1–12 (2018)
Li, W.: et al.: Improving neural abstractive document summarization with explicit information selection modeling. In: EMNLP, pp. 1787–1796 (2020)
Kong, L., Jiang, H., Zhuang, Y., Lyu, J., Zhao, T., Zhang, C.: Calibrated Language Model Fine-Tuning for In- and Out-of-Distribution Data, pp. 1326–1340 (2020). https://doi.org/10.18653/v1/2020.emnlp-main.102.
Laifa, A., et al.: Data augmentation impact on domain-specific text summarization Data augmentation impact on domain-specific text summarization (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Syed, A.A., Gaol, F.L., Boediman, A., Matsuo, T., Budiharto, W. (2022). A Survey of Abstractive Text Summarization Utilising Pretrained Language Models. In: Nguyen, N.T., Tran, T.K., Tukayev, U., Hong, TP., Trawiński, B., Szczerbicki, E. (eds) Intelligent Information and Database Systems. ACIIDS 2022. Lecture Notes in Computer Science(), vol 13757. Springer, Cham. https://doi.org/10.1007/978-3-031-21743-2_42
Download citation
DOI: https://doi.org/10.1007/978-3-031-21743-2_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-21742-5
Online ISBN: 978-3-031-21743-2
eBook Packages: Computer ScienceComputer Science (R0)