Abstract
Summarization models compress the source text without sacrificing the primary information. However, about 30% of summaries produced by state-of-the-art summarization models suffer from the factual inconsistencies between source text and summary, also known as hallucinations, making them less trustworthy. It has been challenging to reduce hallucinations, especially entity hallucinations. Most prior works use the entire source text for factual error correction, while input length is often limited. It prevents models from checking facts precisely and limits its applications on long document summarization. To address this issue, we propose a post-editing factual error correction method based on a Retriever-Reader pipeline. After the summarization model generates the summary, entities in the summary are examined and corrected iteratively by retrieving and reading only the most relevant parts of the source document. We validate the proposed approach on the CNN/DM and Gigaword datasets. Experiments show that the proposed approach outperforms all the baseline models in the consistency metric. Better results are also achieved on human evaluations.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Lin, C.Y.: Rouge: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
Kryściński, W., McCann, B., Xiong, C., Socher, R.: Evaluating the factual consistency of abstractive text summarization. In: EMNLP, pp. 9332–9346 (2020)
Cao, Z., Wei, F., Li, W., Li, S.: Faithful to the original: fact aware neural abstractive summarization. In: AAAI (2018)
Maynez, J., Narayan, S., Bohnet, B., McDonald, R.: On faithfulness and factuality in abstractive summarization. In: ACL, pp. 1906–1919 (2020)
Zhao, Z., Cohen, S.B., Webber, B.: Reducing quantity hallucinations in abstractive summarization. In: EMNLP(Findings), pp. 2237–2249 (2020)
Aralikatte, R., Narayan, S., Maynez, J., Rothe, S., McDonald, R.: Focus attention: promoting faithfulness and diversity in summarization. In: ACL-IJCNLP, pp. 6078–6095 (2021)
Nan, F., et al.: Improving factual consistency of abstractive summarization via question answering. In: ACL-IJCNLP, pp. 6881–6894 (2021)
Chen, S., Zhang, F., Sone, K., Roth, D.: Improving faithfulness in abstractive summarization with contrast candidate generation and selection. In: NAACL, pp. 5935–5941 (2021)
Huang, L., Wu, L., Wang, L.: Knowledge graph-augmented abstractive summarization with semantic-driven cloze reward. In: ACL, pp. 5094–5107 (2020)
Zhang, M., Zhou, G., Yu, W., Liu, W.: Far-ass: fact-aware reinforced abstractive sentence summarization. Inf. Process. Manage. 58(3), 102478 (2021)
Dong, Y., Wang, S., Gan, Z., Cheng, Y., Cheung, J.C.K., Liu, J.: Multi-fact correction in abstractive text summarization. In: EMNLP, pp. 9320–9331 (2020)
Cao, M., Dong, Y., Wu, J., Cheung, J.C.K.: Factual error correction for abstractive summarization models. In: EMNLP, pp. 6251–6258 (2020)
Lewis, M., et al.: Bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: ACL, pp. 7871–7880 (2020)
Sen, P., Saffari, A.: What do models learn from question answering datasets? In: EMNLP, pp. 2429–2438 (2020)
Ko, M., Lee, J., Kim, H., Kim, G., Kang, J.: Look at the first sentence: position bias in question answering. In: EMNLP, pp. 1109–1121 (2020)
Falke, T., Ribeiro, L.F., Utama, P.A., Dagan, I., Gurevych, I.: Ranking generated summaries by correctness: an interesting but challenging application for natural language inference. In: ACL, pp. 2214–2220 (2019)
Cao, S., Wang, L.: Cliff: contrastive learning for improving faithfulness and factuality in abstractive summarization. In: EMNLP, pp. 6633–6649 (2021)
Zhang, J., Zhao, Y., Saleh, M., Liu, P.: Pegasus: pre-training with extracted gap-sentences for abstractive summarization. In: ICML, pp. 11328–11339. PMLR (2020)
Zhu, C., et al.: Enhancing factual consistency of abstractive summarization. In: NAACL, pp. 718–733 (2021)
Lee, D., et al.: Capturing speaker incorrectness: speaker-focused post-correction for abstractive dialogue summarization. In: EMNLP(newsum), pp. 65–73 (2021)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: NAACL, pp. 4171–4186 (2019)
Xie, S., Tu, Z.: Holistically-nested edge detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1395–1403 (2015)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL, pp. 311–318 (2002)
Nallapati, R., Zhou, B., dos Santos, C., Gulçehre, Ç., Xiang, B.: Abstractive text summarization using sequence-to-sequence rnns and beyond. In: CoNLL, pp. 280–290 (2016)
Graff, D., Kong, J., Chen, K., Maeda, K.: English gigaword. Linguistic Data Consortium, Philadelphia 4(1), 34 (2003)
Gehrmann, S., Deng, Y., Rush, A.M.: Bottom-up abstractive summarization. In: EMNLP, pp. 4098–4109 (2018)
See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: ACL, pp. 1073–1083 (2017)
Song, K., et al.: Joint parsing and generation for abstractive summarization. In: AAAI, vol. 34, pp. 8894–8901 (2020)
Liu, Y., Lapata, M.: Text summarization with pretrained encoders. In: EMNLP-IJCNLP, pp. 3730–3740 (2019)
Laban, P., Hsi, A., Canny, J., Hearst, M.A.: The summary loop: learning to write abstractive summaries without examples. In: ACL, pp. 5135–5150 (2020)
Shah, D., Schuster, T., Barzilay, R.: Automatic fact-guided sentence modification. In: AAAI. 34, pp. 8791–8798 (2020)
Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: EMNLP, pp. 38–45 (2020)
Paszke, A., et al.: Pytorch: an imperative style, high-performance deep learning library. NIPS 32 (2019)
Rajpurkar, P., Zhang, J., Lopyrev, K., Liang, P.: Squad: 100,000+ questions for machine comprehension of text. In: EMNLP, pp. 2383–2392 (2016)
Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. arxiv 2017. arXiv preprint arXiv:1711.05101
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Li, W., Liu, J., Gao, H. (2023). Factual Error Correction in Summarization with Retriever-Reader Pipeline. In: Tanveer, M., Agarwal, S., Ozawa, S., Ekbal, A., Jatowt, A. (eds) Neural Information Processing. ICONIP 2022. Lecture Notes in Computer Science, vol 13623. Springer, Cham. https://doi.org/10.1007/978-3-031-30105-6_33
Download citation
DOI: https://doi.org/10.1007/978-3-031-30105-6_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-30104-9
Online ISBN: 978-3-031-30105-6
eBook Packages: Computer ScienceComputer Science (R0)