Abstract
Summary generation using large language models (LLMs) is characterized by its flexibility, high quality, and efficiency. However, professional articles usually contain domain-specific background knowledge and many professional terminologies. It’s hard for the generated summary to maintain professionalism and good writing style using simple prompts coupled with LLMs, which is the common summarization method. While developing task-specific LLMs can improve the summary quality, it demands a high training cost. To enhance the summary quality cost-efficiently, we present ECR, a two-stage expertise-enriched conclude-then-refine summarization framework for professional articles. Firstly, the Key Information Conclusion Stage (KICS) distills the article content through elaborated prompt engineering. Subsequently, the Refinement and Term Enrichment Stage (RTES) enhances coherence, conciseness, and professionalism. Experimental results indicate that our approach offers superior summaries to the summaries generated by the common method across diverse domains. ECR shows an over 80% win rate in structural consistency evaluated by the LLM, alongside a 2x increase in terminology integration. The framework also provides sufficient flexibility to replace components within it. Furthermore, a test dataset comprising 100 articles and an evaluation method for assessing professionalism are proposed.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
Detail description of the data can be found on https://github.com/shauryr/ACL-anthology-corpus.
- 2.
- 3.
The three models’ weights can be downloaded from https://huggingface.co.
References
Bhaskar, A., Fabbri, A.R., Durrett, G.: Prompted opinion summarization with gpt-3.5. In: Annual Meeting of the Association for Computational Linguistics (2022)
Blog, A.: Anthropic claude 2 (2023). https://www.anthropic.com/index/claude-2, Accessed on 12 September 2023
Durmus, E., He, H., Diab, M.T.: Feqa: a question answering evaluation framework for faithfulness assessment in abstractive summarization. ArXiv arXiv: abs/2005.03754 (2020)
Floridi, L., Chiriatti, M.: Gpt-3: its nature, scope, limits, and consequences. Mind. Mach. 30, 681–694 (2020)
Goyal, T., Durrett, G.: Annotating and modeling fine-grained factuality in summarization. In: North American Chapter of the Association for Computational Linguistics (2021)
Goyal, T., Li, J.J., Durrett, G.: News summarization and evaluation in the era of gpt-3. ArXiv abs/ arXiv: 2209.12356 (2022)
Huang, Y., Sun, L., Han, C., Guo, J.: A high-precision two-stage legal judgment summarization. Mathematics 11(6) (2023). https://doi.org/10.3390/math11061320, https://www.mdpi.com/2227-7390/11/6/1320
Jain, D., Borah, M.D., Biswas, A.: Summarization of lengthy legal documents via abstractive dataset building: An extract-then-assign approach. Expert Syst. Appl. 237, 121571 (2024). https://doi.org/10.1016/j.eswa.2023.121571. https://www.sciencedirect.com/science/article/pii/S0957417423020730
Jain, S., et al.: Multi-dimensional evaluation of text summarization with in-context learning. ArXiv arXiv: abs/2306.01200 (2023)
Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Annual Meeting of the Association for Computational Linguistics (2004)
Liu, Y., Fabbri, A.R., Liu, P., Radev, D.R., Cohan, A.: On learning to summarize with large language models as references. ArXiv arXiv: abs/2305.14239 (2023)
Lu, G., Larcher, S.B., Tran, T.A.: Hybrid long document summarization using c2f-far and chatgpt: A practical study. ArXiv arXiv: 2306.01169 (2023)
Pu, X., Gao, M., Wan, X.: Summarization is (almost) dead. ArXiv arXiv: 2309.09558 (2023)
Rohatgi, S.: Acl anthology corpus with full text. Github (2022). https://github.com/shauryr/ACL-anthology-corpus
Veen, D.V., et al.: Adapted large language models can outperform medical experts in clinical text summarization. Nat. Med. (2023)
Wang, J., Liang, Y., Meng, F., Zou, B., Li, Z., Qu, J., Zhou, J.: Zero-shot cross-lingual summarization via large language models. In: Proceedings of the 4th New Frontiers in Summarization Workshop (2023)
Wang, Y., Zhang, Z., Wang, R.: Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. In: Annual Meeting of the Association for Computational Linguistics (2023)
Wu, N., Gong, M., Shou, L., Liang, S., Jiang, D.: Large language models are diverse role-players for summarization evaluation. In: Natural Language Processing and Chinese Computing (2023)
Yang, X., Li, Y., Zhang, X., Chen, H., Cheng, W.: Exploring the limits of chatgpt for query or aspect-based text summarization. ArXiv arxiv: abs/2302.08081 (2023)
van Zandvoort, D., Wiersema, L., Huibers, T., van Dulmen, S., Brinkkemper, S.: Enhancing summarization performance through transformer-based prompt engineering in automated medical reporting. arxiv: 2311.13274 (2023)
Zeng, Q., Sidhu, M., Chan, H.P., Wang, L., Ji, H.: Scientific opinion summarization: Meta-review generation with checklist-guided iterative introspection (2023)
Zhang, H., Liu, X., Zhang, J.: Extractive summarization via chatgpt for faithful summary generation. In: Conference on Empirical Methods in Natural Language Processing (2023)
Zhang, H., Liu, X., Zhang, J.: Summit: Iterative text summarization via chatgpt. ArXiv arXiv: abs/2305.14835 (2023)
Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. ArXiv arXiv: 1904.09675 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Liang, Z., Xie, K., Lu, S., Shi, Y., Yeerpan, T., Wang, Z. (2024). ECR: An Expertise-Enriched Conclude-Then-Refine Summarization Framework for Professional Articles. In: Rapp, A., Di Caro, L., Meziane, F., Sugumaran, V. (eds) Natural Language Processing and Information Systems. NLDB 2024. Lecture Notes in Computer Science, vol 14763. Springer, Cham. https://doi.org/10.1007/978-3-031-70242-6_10
Download citation
DOI: https://doi.org/10.1007/978-3-031-70242-6_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70241-9
Online ISBN: 978-3-031-70242-6
eBook Packages: Computer ScienceComputer Science (R0)