Skip to main content
SpringerLink
Log in

Document-level paraphrase generation base on attention enhanced graph LSTM

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Paraphrase generation is one of the long-standing and important tasks in natural language processing. Existing literature has mainly focused on the generation of sentence-level paraphrases, in which the relationship between sentences was ignored, such as sentence reordering, sentence splitting, and sentence merging. In this paper, while paying attention to the relationship within sentences, we also explore the relationship between sentences. For the task of document-level interpretation generation, we focus on reordering documents to enhance inter-sentence diversity. We use the attention-enhanced graph long short-term memory (LSTM) to encode the relationship graph between sentences, so that each sentence generates a coherent representation that conforms to the context. Based on the sentence-level paraphrase generation model, we constructed a pseudo-document-level paraphrase dataset. The automatic evaluation shows that our model achieves higher scores in terms of semantic relevance and diversity scores than other strong baseline models. In the manual evaluation, the validity of our model is also confirmed. Experiments show that our model retains the semantics of the source document, while generating paraphrase documents with high diversity. When we reorder the sentences, the output paraphrase documents can still preserve the coherence between sentences with higher scores.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://www.cs.cmu.edu/~jwieting/

  2. https://www.statmt.org/wmt20/translation-task.html

  3. The BERTScore tool is accessible at https://github.com/Tiiiger/bert_score

  4. The WER tool is accessible at https://github.com/belambert/asr-evaluationhttps://github.com/belambert/asr-evaluation

  5. The TER tool is accessible at https://github.com/jhclark/multeval

References

  1. Xu W, Li S, Lu Y (2021) Usr-mtl: an unsupervised sentence representation learning framework with multi-task learning. Appl Intell 51(6):3506–3521

    Article  Google Scholar 

  2. Aithal SG, Rao AB, Singh S (2021) Automatic question-answer pairs generation and question similarity mechanism in question answering system. Appl Intell 51:8484–8497

    Article  Google Scholar 

  3. Do P, Phan T H V (2022) Developing a bert based triple classification model using knowledge graph embedding for question answering system. Appl Intell 52(1):636–651

    Article  Google Scholar 

  4. Kang L, He S, Wang M (2022) Bilingual attention based neural machine translation. Appl Intell :1–14

  5. El-Kassas WS, Salama CR, Rafea AA, Mohamed HK (2021) Automatic text summarization: a comprehensive survey. Expert Syst Appl 165:113679

    Article  Google Scholar 

  6. Zhang D, Zhu Z, Kang S et al (2021) Syntactic and semantic analysis network for aspect-level sentiment classification. Appl Intell 51:6136–6147

    Article  Google Scholar 

  7. Karrar AE (2022) A proposed model for improving the performance of knowledge bases in real-world applications by extracting semantic information. Int J Adv Comput Sci Appl 13(2):116–123

    Google Scholar 

  8. Gong G, An X, Mahato N et al (2019) Research on short-term load prediction based on seq2seq model. Energies 12(16):3199

    Article  Google Scholar 

  9. Chen Z, Ren J (2021) Multi-label text classification with latent word-wise label information. Appl Intell 51(2):966–979

    Article  Google Scholar 

  10. Keneshloo Y, Shi T, Ramakrishnan N, Reddy CK (2019) Deep reinforcement learning for sequence-to-sequence models. IEEE Trans Neural Netw Learning Syst 31(7):2469–2489

    Google Scholar 

  11. Gao T, Cui Y, Ding F (2021) Seqvae: Sequence variational autoencoder with policy gradient. Appl Intell 51:9030–9037

    Article  Google Scholar 

  12. Li Y, Feng R, Rehg I, Zhang C (2020) Transformer-based neural text generation with syntactic guidance

  13. Kumar A, Ahuja K, Vadapalli R et al (2020) Syntax-guided controlled generation of paraphrases. Trans Assoc Comput Linguist 8(1):330–345

    Article  Google Scholar 

  14. Kubal D, Palivela H (2021) Unified model for paraphrase generation and paraphrase identification

  15. Prakash A, Hasan SA, Lee K et al (2016) Neural paraphrase generation with stacked residual lstm networks. In: Proceedings of the 26th international conference on computational linguistics, pp 2923–2934

  16. Hosking T, Lapata M (2021) Factorising meaning and form for intent-preserving paraphrasing. In: Proceedings of the 59th annual meeting of the association for computational linguistics, pp 1405–1418

  17. Li Z, Jiang X, Shang L et al (2019) Decomposable neural paraphrase generation. In: Proceedings of the 57th annual meeting of the association for computational linguistics, pp 3403–3414

  18. Goyal T, Durrett G (2020) Neural syntactic preordering for controlled paraphrase generation. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 238–252

  19. Fu Y, Feng Y, Cunningham JP (2019) Paraphrase generation with latent bag of words. Adv Neural Inf Process Syst 32:13645–13656

    Google Scholar 

  20. Kazemnejad A, Salehi M, Baghshah MS et al (2020) Paraphrase generation by learning how to edit from samples. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 6010–6021

  21. Huang K, Chang K (2021) Generating syntactically controlled paraphrases without using annotated parallel pairs. In: Proceedings of the 16th conference of the european chapter of the association for computational linguistics, pp 1022–1033

  22. Mallinson J, Sennrich R, Lapata M (2017) Paraphrasing revisited with neural machine translation. In: Proceedings of the 15th conference of the european chapter of the association for computational linguistics, pp 881–893

  23. Wieting J, Gimpel K (2018) Paranmt-50m: pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In: Proceedings of the 56th annual meeting of the association for computational linguistics, pp 451–463

  24. Moon HC, Mohiuddin T, Joty S et al (2019) A unified neural coherence model. In: Proceedings of the 56th annual meeting of the association for computational linguistics, pp 2262–2272

  25. Mohiuddin T, Jwalapuram P, Lin X et al (2020) Coheval: benchmarking coherence models

  26. Devlin J, Chang MW, Lee K et al (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational Linguistics, pp 4171–4186

  27. Lan Z, Chen M, Goodman S et al (2020) Albert: a lite bert for self-supervised learning of language representations. In: International conference on learning representations, pp 238–255

  28. Beck D, Haffari G, Cohn T et al (2018) Graph-to-sequence learning using gated graph neural networks. In: Proceedings of the 56th annual meeting of the association for computational linguistics, pp 273–283

  29. Sun Q, Zhang K, Lv L et al (2022) Joint extraction of entities and overlapping relations by improved graph convolutional networks. Appl Intell 52(5):5212–5224

    Article  Google Scholar 

  30. Guo Z, Zhang Y, Teng Z et al (2019) Densely connected graph convolutional networks for graph-to-sequence learning. Trans Assoc Comput Linguist 7:297–312

    Article  Google Scholar 

  31. Zhou J, Cui G, Hu S et al (2020) Graph neural networks: a review of methods and applications. AI Open 1:57–81

    Article  Google Scholar 

  32. Peng N, Poon H, Quirk C et al (2017) Cross-sentence n-ary relation extraction with graph lstms. Trans Assoc Comput Linguist 5:101–115

    Article  Google Scholar 

  33. Veličković P, Cucurull G, Casanova A et al (2018) Graph attention networks. Statistics 1050:20

    Google Scholar 

  34. Zhou A, Li Y (2021) Structural attention network for graph. Appl Intell 51:6255–6264

    Article  Google Scholar 

  35. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. In: Advances in neural information processing systems, pp 5998–6008

  36. Cao Y, Liu H, Wan X (2020) Jointly learning to align and summarize for neural cross-lingual summarization. In: Proceedings of the 58th annual meeting of the association for computational linguistics, pp 6220–6231

  37. Sagheer A, Hamdoun H, Youness H (2021) Deep lstm-based transfer learning approach for coherent forecasts in hierarchical time series. Sensors 21(13):4379

    Article  Google Scholar 

  38. Sagheer A, Hamdoun H, Youness H (2019) Understanding and improving layer normalization. Adv Neural Inf Process Syst: 32

  39. Zhang T, Kishore V, Wu F et al (2020) Bertscore: evaluating text generation with bert. In: International conference on learning representations, pp 26–30

  40. Weston J, Lenain R, Meepegama U et al (2022) Generative pretraining for paraphrase evaluation. In: Proceedings of the 60th annual meeting of the association for computational linguistics (Volume 1: Long Paper), pp 4052–4073

  41. Wolf T, Chaumond J, Debut L et al (2020) Transformers: state-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing, pp 38–45

  42. See A, Liu PJ, Manning CD (2017) Get to the point: summarization with pointer-generator networks. In: Proceedings of the 55th annual meeting of the association for computational linguistics, pp 1073–1083

  43. Fan A, Lewis M, Peng N (2018) Aesop: paraphrase generation with adaptive syntactic control. In: Proceedings of the 56th annual meeting of the association for computational linguistics, pp 889–898

  44. Sun J, Ma X, Dauphin Y (2021) Hierarchical neural story generation. In: Proceedings of the 2021 conference on empirical methods in natural language processing, pp 5176–5189

Download references

Funding

This work was supported by the National Natural Science Foundations of China (Grant no. 12171065 and 11671001 and 61901074) and the Science and Technology Research Program of Chongqing Municipal Education Commission (Grant No. KJQN201900636).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Nanan, 400065, Chongqing, China

    Dong Qiu, Lei Chen & Yang Yu

  2. College of Science, Chongqing University of Posts and Telecommunications, Nanan, 400065, Chongqing, China

    Dong Qiu

Authors
  1. Dong Qiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dong Qiu.

Ethics declarations

Ethics approval and consent to participate

This article does not contain any stulies with human participants or animals perfornedi by any of the authors.

Consent for Publication

No human or animal subjects were used in the study.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Qiu, D., Chen, L. & Yu, Y. Document-level paraphrase generation base on attention enhanced graph LSTM. Appl Intell 53, 10459–10471 (2023). https://doi.org/10.1007/s10489-022-04031-z

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-022-04031-z

Keywords

Search

Navigation