Abstract
Existing extractive summarization methods achieve state-of-the-art (SOTA) performance with pre-trained language models (PLMs) and sufficient training data. However, PLM-based methods are known to be data-hungry and often fail to deliver satisfactory results in low-resource scenarios. Constructing a high-quality summarization dataset with human-authored reference summaries is a prohibitively expensive task. To address these challenges, this paper proposes a novel paradigm for low-resource extractive summarization, called ParaSum. This paradigm reformulates text summarization as textual paraphrasing, aligning the text summarization task with the self-supervised Next Sentence Prediction (NSP) task of PLMs. This approach minimizes the training gap between the summarization model and PLMs, enabling a more effective probing of the knowledge encoded within PLMs and enhancing the summarization performance. Furthermore, to relax the requirement for large amounts of training data, we introduce a simple yet efficient model and align the training paradigm of summarization to textual paraphrasing to facilitate network-based transfer learning. Extensive experiments over two widely used benchmarks (i.e., CNN/DailyMail, Xsum) and a recent open-sourced high-quality Chinese benchmark (i.e., CNewSum) show that ParaSum consistently outperforms existing PLM-based summarization methods in all low-resource settings, demonstrating its effectiveness over different types of datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Xu, J., Gan, Z., Cheng, Y., Liu, J.: Discourse-aware neural extractive text summarization. In: ACL (2020)
Quatra, M., Cagliero, L.: End-to-end training for financial report summarization. In: COLING, pp. 118–123 (2020)
Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: EMNLP-IJCNLP, pp. 3730–3740 (2019)
Chen, Y.-C., Bansal, M.: Fast abstractive summarization with reinforce-selected sentence rewriting. In: ACL (2018)
Gu, N., Ash, E., Hahnloser, R.: MemSum: extractive summarization of long documents using multi-step episodic Markov decision processes. In: ACL, Ireland, Dublin, pp. 6507–6522 (2022)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL, pp. 4171–4186 (2019)
Liu, Y., Ott, M., Goyal, N., et al.: Roberta: a robustly optimized BERT pretraining approach, arXiv, vol. abs/1907.11692 (2019)
Zhong, M., Liu, P., Chen, Y., Wang, D., Qiu, X., Huang, X.: Extractive summarization as text matching. In: ACL, pp. 6197–6208 (2020)
Zhong, M., Liu, P., Wang, D., Qiu, X., Huang, X.: Searching for effective neural extractive summarization: what works and what’s next. In: ACL, pp. 1049–1058 (2019)
Schick, T., Schütze, H.: It’s not just size that matters: small language models are also few-shot learners. In: NAACL, pp. 2339–2352 (2021)
Hermann, K.M., et al.: Teaching machines to read and comprehend. In: NeuralIPS, pp. 1693–1701 (2015)
Narayan, S., Cohen, S.B., Lapata, M.: Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In: EMNLP (2018)
Chen, K., Fu, G., Chen, Q., Hu, B.: A large-scale Chinese long-text extractive summarization corpus. In: ICASSP, pp. 7828–7832 (2021)
Shafiq, N., et al.: Abstractive text summarization of low-resourced languages using deep learning. PeerJ Comput. Sci. 9, e1176 (2023)
Chen, Y.-S., Song, Y.-Z., Shuai, H.-H.: SPEC: summary preference decomposition for low-resource abstractive summarization. IEEE/ACM Trans. Audio Speech Lang. Process. 31, 603–618 (2022)
Huh, T., Ko, Y.: Lightweight meta-learning for low-resource abstractive summarization. In: SIGIR, pp. 2629–2633 (2022)
Zaken, E.B., Ravfogel, S., Goldberg, Y.: BitFit: simple parameter-efficient fine-tuning for transformer-based masked language-models. In: ACL, pp. 1–9 (2022)
Song, H., Dong, L., Zhang, W., Liu, T., Wei, F.: CLIP models are few-shot learners: empirical studies on VQA and visual entailment. In: ACL, pp. 6088–6100 (2022)
Wang, S., Fang, H., Khabsa, M., Mao, H., Ma, H.: Entailment as few-shot learner, CoRR (2021)
Gao, P., et al.: CLIP-Adapter: Better Vision-Language Models with Feature Adapters, arXiv (2021)
Zhang, R., et al.: Tip-adapter: training-free adaption of CLIP for few-shot classification. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) ECCV 2022. LNCS, vol. 13695, pp. 493–510. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19833-5_29
Houlsby, N., et al.: Parameter-efficient transfer learning for NLP. In: ICML, pp. 2790–2799 (2019)
Ganin, Y., et al.: Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17, 2096-2030 (2016)
Yosinski, J., Clune, J., Bengio, Y., Lipson, H.: How transferable are features in deep neural networks? In: NeurIPS (2014)
Gao, T., Fisch, A., Chen, D.: Making pre-trained language models better few-shot learners. In: ACL, pp. 3816–3830 (2021)
Lin, C.-Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
Radford, A., et al.: Learning transferable visual models from natural language supervision. In: ICML, pp. 8748–8763 (2021)
Liu, Y., Liu, P.: SimCLS: a simple framework for contrastive learning of abstractive summarization. In: ACL, pp. 1065–1072 (2021)
Liu, Y., Liu, P., Radev, D., Neubig, G.: BRIO: bringing order to abstractive summarization. In: ACL, pp. 2890–2903 (2022)
Wang, D., Chen, J., Wu, X., Zhou, H., Li, L.: CNewSum: a large-scale summarization dataset with human-annotated adequacy and deducibility level. In: Wang, L., Feng, Y., Hong, Yu., He, R. (eds.) NLPCC 2021. LNCS (LNAI), vol. 13028, pp. 389–400. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88480-2_31
Sharma, L., Graesser, L., Nangia, N., Evci, U.: Natural language understanding with the quora question pairs dataset, arXiv (2019)
Liu, X., et al.: LCQMC: a large-scale Chinese question matching corpus. In: COLING, pp. 1952–1962 (2018)
Hu, B., Chen, Q., Zhu, F.: LCSTS: a large scale Chinese short text summarization dataset. In: EMNLP, pp. 1967–1972 (2015)
Li, S., Zhao, Z., Hu, R., Li, W., Liu, T., Du, X.: Analogical reasoning on Chinese morphological and semantic relations. In: ACL, pp. 138–143 (2018)
Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant No. 62202170 and Alibaba Group through the Alibaba Innovation Research Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Tang, M., Wang, C., Wang, J., Chen, C., Gao, M., Qian, W. (2023). ParaSum: Contrastive Paraphrasing for Low-Resource Extractive Text Summarization. In: Jin, Z., Jiang, Y., Buchmann, R.A., Bi, Y., Ghiran, AM., Ma, W. (eds) Knowledge Science, Engineering and Management. KSEM 2023. Lecture Notes in Computer Science(), vol 14119. Springer, Cham. https://doi.org/10.1007/978-3-031-40289-0_9
Download citation
DOI: https://doi.org/10.1007/978-3-031-40289-0_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-40288-3
Online ISBN: 978-3-031-40289-0
eBook Packages: Computer ScienceComputer Science (R0)