skip to main content
10.1145/3488560.3502214acmconferencesArticle/Chapter ViewAbstractPublication PageswsdmConference Proceedingsconference-collections
extended-abstract

Improving Text Generation via Neural Discourse Planning

Published: 15 February 2022 Publication History

Abstract

Recent Transformer-based approaches to NLG like GPT-2 can generate syntactically coherent original texts. However, these generated texts have serious flaws. One of them is a global discourse incoherence. We present an approach to estimate the quality of discourse structure. Empirical results confirm that the discourse structure of currently generated texts is inaccurate. We propose the research directions to plan it and fill in the text in its leaves using the pipeline consisting of two GPT-based generation models. The suggested approach is universal and can be applied to different languages.

Supplementary Material

MP4 File (wsdm22-dc04.mp4)
This talk describes a universal approach to improve the discourse coherence in the texts generated by Transformer-based models. I present an automatic approach to estimate the quality of discourse structure and demonstrate that the discourse structure of currently generated texts is inaccurate. Apart from that, a neural pipeline for generation is proposed by planning not only words but also discourse substructures.

References

[1]
Or Biran and Kathleen McKeown. 2015. Discourse Planning with an N-gram Model of Relations. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing . Association for Computational Linguistics, Lisbon, Portugal, 1973--1977. https://doi.org/10.18653/v1/D15--1230
[2]
Antoine Bosselut, Asli Celikyilmaz, Xiaodong He, Jianfeng Gao, Po-Sen Huang, and Yejin Choi. 2018. Discourse-Aware Neural Rewards for Coherent Text Generation. CoRR, Vol. abs/1805.03766 (2018). arxiv: 1805.03766 https://arxiv.org/abs/1805.03766
[3]
Alexander Chernyavskiy and Dmitry Ilvovsky. 2020. Recursive Neural Text Classification Using Discourse Tree Structure for Argumentation Mining and Sentiment Analysis Tasks . 90--101. https://doi.org/10.1007/978--3-030--59491--6_9
[4]
Giovanni Luca Ciampaglia, Prashant Shiralkar, Luis Mateus Rocha, Johan Bollen, Filippo Menczer, and Alessandro Flammini. 2015. Computational Fact Checking from Knowledge Networks. PLoS ONE, Vol. 10 (2015).
[5]
Vrindavan Harrison, Lena Reed, Shereen Oraby, and Marilyn A. Walker. 2019. Maximizing Stylistic Control and Semantic Accuracy in NLG: Personality Variation and Discourse Contrast. CoRR, Vol. abs/1907.09527 (2019). arxiv: 1907.09527 http://arxiv.org/abs/1907.09527
[6]
Yangfeng Ji, Gholamreza Haffari, and Jacob Eisenstein. 2016. A Latent Variable Recurrent Neural Network for Discourse Relation Language Models. CoRR, Vol. abs/1603.01913 (2016). arxiv: 1603.01913 http://arxiv.org/abs/1603.01913
[7]
Wei-Jen Ko and Junyi Jessy Li. 2020. Assessing Discourse Relations in Language Generation from GPT-2. In Proceedings of the 13th International Conference on Natural Language Generation . Association for Computational Linguistics, Dublin, Ireland, 52--59. https://aclanthology.org/2020.inlg-1.8
[8]
William Mann and Sandra Thompson. 1987. Rhetorical Structure Theory: A Theory of Text Organization. (01 1987).
[9]
Nanyun Peng, Marjan Ghazvininejad, Jonathan May, and Kevin Knight. 2018. Towards Controllable Story Generation. In Proceedings of the First Workshop on Storytelling. Association for Computational Linguistics, New Orleans, Louisiana, 43--49. https://doi.org/10.18653/v1/W18--1505
[10]
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019. Data-to-Text Generation with Content Selection and Planning. In AAAI .
[11]
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving Language Understanding by Generative Pre-Training. https://www.semanticscholar.org/paper/Improving-Language-Understanding-by-Generative-Radford-Narasimhan/cd18800a0fe0b668a1cc19f2ec95b5003d0a5035
[12]
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
WSDM '22: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining
February 2022
1690 pages
ISBN:9781450391320
DOI:10.1145/3488560
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 February 2022

Check for updates

Author Tags

  1. discourse planning
  2. discourse structure
  3. neural text generation

Qualifiers

  • Extended-abstract

Funding Sources

  • HSE University
  • Government of the Russian Federation

Conference

WSDM '22

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 192
    Total Downloads
  • Downloads (Last 12 months)5
  • Downloads (Last 6 weeks)0
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media