Skip to main content
SpringerLink
Log in

Improving PLMs for Graph-to-Text Generation by Relational Orientation Attention

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Pretrained language models (PLMs) with impressive performances in graph-to-text generation have recently been employed. However, linearized graph data will lead to the loss of triplet structure information and the problem of insufficient syntactic information when graph data are converted into sequence data by PLMs. These defects prevent PLMs from absorbing all the information that knowledge graphs hold and exerting their full potential in graph-to-text generation. To address these issues, we propose two targeted solutions. First, a relational orientation attention (ROA) module is proposed to incorporate triplet structure information into knowledge graph representations. During graph encoding, ROA establishes structural associations between entities and relations by fusing relevant relation information into entity representations. Second, the (knowledge subgraph, text) pairs are used to pretrain PLMs in text and triplet reconstruction tasks. Pretraining tasks with linearized graph data will enable PLMs to transfer learning more seamlessly between graphs and texts. The experiments with the WebNLG, WebQuestions, and PathQuestions datasets demonstrate that relational orientation attention and pretraining tasks (text and triplet reconstruction) can be implemented to capture triplet structure information and boost the learning ability of PLMs on structured data. Additional research reveals that PLMs equipped with designed approaches have superior few-shot learning capability.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Data Availability

The datasets generated and/or analysed during the current study will be made available on reasonable request.

Notes

  1. https://huggingface.co/.

References

  1. Huang X, Zhang J, Li D, et al (2019) Knowledge graph embedding based question answering. In: Proceedings of the twelfth ACM international conference on web search and data mining, pp 105–113

  2. Ji S, Pan S, Cambria E et al (2021) A survey on knowledge graphs: representation, acquisition, and applications. IEEE Trans Neural Netw Learn Syst 33(2):494–514

    Article  MathSciNet  Google Scholar 

  3. Tiddi I, Schlobach S (2022) Knowledge graphs as tools for explainable machine learning: a survey. Artif Intell 302(103):627

    MathSciNet  MATH  Google Scholar 

  4. Wang Y, Zhang H, Liu Y, et al (2019) Kg-to-text generation with slot-attention and link-attention. In: CCF International conference on natural language processing and Chinese computing, Springer, pp 223–234

  5. Zhou H, Young T, Huang M, et al (2018) Commonsense knowledge aware conversation generation with graph attention. In: IJCAI, pp 4623–4629

  6. Koncel-Kedziorski R, Bekal D, Luan Y, et al (2019) Text generation from knowledge graphs with graph transformers. In: 2019 annual conference of the north american chapter of the association for computational linguistics, association for computational linguistics (ACL), pp 2284–2293

  7. Ji H, Ke P, Huang S, et al (2020) Language generation with multi-hop reasoning on commonsense knowledge graph. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 725–736

  8. Gardent C, Shimorina A, Narayan S, et al (2017) The webnlg challenge: generating text from RDF data. In: Proceedings of the 10th international conference on natural language generation, pp 124–133

  9. Wang T, Wan X, Jin H (2020) Amr-to-text generation with graph transformer. Trans Assoc Comput Linguist 8:19–33

    Article  Google Scholar 

  10. Schmitt M, Ribeiro LF, Dufter P, et al (2021) Modeling graph structure via relative position for text generation from knowledge graphs. In: Proceedings of the fifteenth workshop on graph-based methods for natural language processing (TextGraphs-15), pp 10–21

  11. Li L, Geng R, Li B, et al (2022) Graph-to-text generation with dynamic structure pruning. In: Proceedings of the 29th international conference on computational linguistics, pp 6115–6127

  12. Wang Q, Yavuz S, Lin XV, et al (2021) Stage-wise fine-tuning for graph-to-text generation. In: Proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: student research workshop, pp 16–22

  13. Hoyle AM, Marasović A, Smith NA (2021) Promoting graph awareness in linearized graph-to-text generation. Find Assoc Comput Linguist ACL-IJCNLP 2021:944–956

    Article  Google Scholar 

  14. Radford A, Wu J, Child R et al (2019) Language models are unsupervised multitask learners. OpenAI Blog 1(8):9

    Google Scholar 

  15. Lewis M, Liu Y, Goyal N, et al (2019) Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461

  16. Raffel C, Shazeer N, Roberts A et al (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. J Mach Learn Res 21(140):1–67

    MathSciNet  Google Scholar 

  17. Ribeiro LF, Schmitt M, Schütze H, et al (2020) Investigating pretrained language models for graph-to-text generation. arXiv preprint arXiv:2007.08426

  18. Qiu X, Sun T, Xu Y et al (2020) Pre-trained models for natural language processing: a survey. Sci China Technol Sci 63(10):1872–1897

    Article  Google Scholar 

  19. Wang T, Wan X, Yao S (2021) Better amr-to-text generation with graph structure reconstruction. In: Proceedings of the twenty-ninth international conference on international joint conferences on artificial intelligence, pp 3919–3925

  20. Ribeiro LF, Zhang Y, Gardent C et al (2020) Modeling global and local node contexts for text generation from knowledge graphs. Trans Assoc Comput Linguist 8:589–604

    Article  Google Scholar 

  21. Ke P, Ji H, Ran Y, et al (2021) Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. arXiv preprint arXiv:2106.10502

  22. Berant J, Chou A, Frostig R, et al (2013) Semantic parsing on freebase from question-answer pairs. In: Proceedings of the 2013 conference on empirical methods in natural language processing, pp 1533–1544

  23. Zhou M, Huang M, Zhu X (2018) An interpretable reasoning network for multi-relation question answering. In: Proceedings of the 27th international conference on computational linguistics, pp 2010–2022

  24. Vaswani A, Shazeer N, Parmar N, et al (2017) Attention is all you need. In: Advances in neural information processing systems, p 30

  25. Veličković P, Cucurull G, Casanova A, et al (2017) Graph attention networks. arXiv preprint arXiv:1710.10903

  26. Cai D, Lam W (2020) Graph transformer for graph-to-sequence learning. In: Proceedings of the AAAI conference on artificial intelligence, pp 7464–7471

  27. Edunov S, Baevski A, Auli M (2019) Pre-trained language model representations for language generation. In: Proceedings of NAACL-HLT, pp 4052–4059

  28. Li J, Tang T, Zhao WX, et al (2021) Few-shot knowledge graph-to-text generation with pretrained language models. arXiv preprint arXiv:2106.01623

  29. Schlichtkrull M, Kipf TN, Bloem P, et al (2018) Modeling relational data with graph convolutional networks. In: European semantic web conference, Springer, pp 593–607

  30. Zhang Q, Wang R, Yang J et al (2022) Knowledge graph embedding by reflection transformation. Knowl Based Syst 238(107):861

    Google Scholar 

  31. Vrandečić D, Krötzsch M (2014) Wikidata: a free collaborative knowledgebase. Commun ACM 57(10):78–85

    Article  Google Scholar 

  32. Chen W, Su Y, Yan X, et al (2020) Kgpt: Knowledge-grounded pre-training for data-to-text generation. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pp 8635–8648

  33. Auer S, Bizer C, Kobilarov G, et al (2007) Dbpedia: A nucleus for a web of open data. In: The semantic web. Springer, pp 722–735

  34. Bollacker K, Evans C, Paritosh P, et al (2008) Freebase: a collaboratively created graph database for structuring human knowledge. In: Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pp 1247–1250

  35. Papineni K, Roukos S, Ward T, et al (2002) Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th annual meeting of the association for computational linguistics, pp 311–318

  36. Lin CY (2004) Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out, pp 74–81

  37. Banerjee S, Lavie A (2005) Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp 65–72

Download references

Acknowledgements

This work was supported by the National Key Research and Development Program of China (grant number 2021YFC3300204).

Author information

Authors and Affiliations

  1. School of Electronic and Information Engineering, Beijing Jiaotong University, Beijing, 100044, China

    Tao Wang, Bo Shen, Jinglin Zhang & Yu Zhong

  2. Key Laboratory of Communication and Information Systems, Beijing Municipal Commission of Education, Beijing, 100044, China

    Tao Wang, Bo Shen, Jinglin Zhang & Yu Zhong

Authors
  1. Tao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bo Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jinglin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yu Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bo Shen.

Ethics declarations

Conflict of interest

The authors declare that they have no competing interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, T., Shen, B., Zhang, J. et al. Improving PLMs for Graph-to-Text Generation by Relational Orientation Attention. Neural Process Lett 55, 7967–7983 (2023). https://doi.org/10.1007/s11063-023-11292-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-023-11292-3

Keywords

Search

Navigation