Adversarial text generation with context adapted global knowledge and a self-attentive discriminator

Highlights

• A word sequence-based adversarial network that exploits the semantics of the corpus by adapting global word embeddings to the context of analysis.

• Self-attentive discriminator to map the semantics of the generated text with real-world text.

• Evaluation framework based on quantitative and qualitative analyses.

• A word sequence-based adversarial network that balances both generator and discriminator towards reaching the Nash equilibrium.

Abstract

Text generation is a challenging task for intelligent agents. Numerous research attempts have investigated the use of adversarial networks with word sequence-based generators. However, these approaches suffer from an unbalance between generator and discriminator causing overfitting due to the strength that the discriminator acquires by getting too precise in distinguishing what the generator is producing and what instead comes from the real dataset. In this paper, we investigate how to balance both generator and discriminator of a sequence-based text adversarial network exploiting: i) the contribution of global knowledge in the input of the adversarial network encoded by global word embeddings that are adapted to the context of the datasets in which they are utilized, and ii) the use of a self-attentive discriminator that slowly minimizes its loss function and thus enables the generator to get valuable feedback during the training process. Through an extensive evaluation on three datasets of short-, medium- and long-length text documents, the results computed using word-overlapping metrics show that our model outperforms four baselines. We also discuss the results of our model using readability metrics and the human perceived quality of the generated documents.

Introduction

Text generation is the computational task of producing text in natural language. Over the last few years, it has become popular thanks to the many applications in which it can be used such as dialogue generation, text summarization and neural machine translation.

Text is a means of communication consisting of the use of words in structured and conventional ways (syntax) to deliver multifarious meanings (semantics). Generating text autonomously is an outstanding challenge in today’s intelligent agents, whose results are far from human-like level of the generated text. Numerous approaches address the task by building language models that aim to represent the probability distribution of words in a text used as a reference in order to predict the next word given those that precede it, thus casting the text generation task as a sequence prediction problem. The current state of the art of language models are developed using neural networks. Recurrent neural networks (RNNs) are frequently used learning methodologies that maximize the likelihood to predict the observed data (Karafiàt, Burget, C̆ernocký, & Khudanpur, 2010). A major limitation of RNNs in addressing this task is the exposure bias (Bengio, Vinyals, Jaitly, & Shazeer, 2015). During the training process, the network processes training data, but during the inference stage it uses its own predictions as input. As a result when the network makes a prediction mistake the error is accumulated and propagated throughout the sequence, causing the model to being unable to work properly. In Bengio et al. (2015), authors proposed the Scheduled Sampling technique, which consists of randomly alternating the input during the training process by feeding real data or its own predictions, but, in Huszár (2015), authors demonstrated that this method is inconsistent. Such an attempt investigates the impact of real data utilized during the inference stage, a methodology that is later exploited by generative adversarial networks successfully.

The complexity of language is manifold since there are several aspects to take into account, from syntax, to semantics, to how to express a coherent meaning within a sentence, diversity, and improvisation. Syntax structures can be learned by statistical models, for instance how to structure a sentence with a subject, a predicate and an object.

Readability and discourse coherence are two challenges addressed by automated text generation approaches. Even though numerous advancements have been made in this field, the approaches discussed so far (Bosselut et al. (2018), Shi, Chen, Qiu, and Huang (2018)), still have limitations that are inherited from the machine learning field: an inability to make agents really understanding but rather replicate an observed pattern of a given data series. As observed in Li, Galley, Brockett, Gao, and Dolan (2016), this outputs a strong ability to reproduce a syntax structure, but it presents weaknesses in generating articulated semantic structures and in text diversity.

The introduction of Generative Adversarial Networks (GANs) (Goodfellow et al., 2014) paved the way to how generative tasks can be addressed. Compared to previous computational attempts that are based on having an explicit likelihood function, GANs’ likelihood function is implied by having two components with two different objective functions (Danihelka, Lakshminarayanan, Uria, Wierstra, & Dayan, 2017). For this reason these networks have fundamental characteristics to alleviate the exposure bias problem and to provide further diversity in the generated outputs. These are investigated in this work.

GANs were designed to deal with continuous data such as image characteristics, while natural language is based on discrete tokens and therefore a different implementation is required. SeqGAN (Yu, Zhang, Wang, & Yu, 2017) extends GANs to be applied for a discrete sequence generation task by considering the problem of predicting the next word as a reinforcement learning problem where i) the initial state is the current sequence at a given timestep, ii) the action is to select the next word and iii) the reward is computed according to the distance between the expected word and the predicted one. However, these approaches suffer from an unbalance between generator and discriminator causing overfitting due to the strength that the discriminator acquires by getting too precise in distinguishing what the generator is producing and what instead comes from the real dataset. In this paper, we investigate the contribution of contextual information to SeqGAN as inputs encoded by global word embeddings built from common knowledge that are adapted to the context of the datasets in which they are utilized. We also experiment with a sequence-based self-attentive neural network as a discriminator and experimentally assessed whether the model is able to encode knowledge into the generated text while contributing to stabilize the loss function in the adversarial training. In fact, a self-attentive network slowly minimizes its loss function while exploiting the semantics of text and thus enabling the generator to get valuable feedback during the training process. We investigate the following two research questions:

• what is the impact of utilizing supervised context adaptation of global word embeddings in a SeqGAN model for text generation?

• which is the most suitable discriminator in a SeqGAN model with context adaptation for text generation?

We perform an experiment based on three datasets of short-, medium- and long-length² text documents and we propose a quantitative analysis of the performance of our approach on a set of n-gram matching metrics that measure word and part-of-speech overlaps, and readability metrics. We also discuss the human perceived quality of the generated sentences.

The remainder of this paper is organized as follows: Section 2 presents a background on probabilistic language modeling for text generation and, then, reports state-of-the-art approaches in this research field; Section 4 illustrates the use of word embeddings by exploiting contextual knowledge; Section 5 introduces a sequence-based self-attentive discriminator that contributes to give valuable feedback to the generator; Section 6 reports the algorithm utilized for training our sequence-based adversarial network; Section 7 reports the datasets and metrics utilized in our experimentation; Section 8 illustrates the quantitative and qualitative results, and we conclude the paper with a discussion in Section 9.