Abstract
Human language has the ability to convey the speaker’s emotions, such as happiness, sadness, or anger. Existing text generation methods mainly focus on the sequence-to-sequence (Seq2Seq) model that applied an encoder to transform the input text into latent representation and a decoder to generate texts from the latent representation. To control the sentiment of the generated text, these models usually concatenate a disentangled feature into the latent representation. However, such a method is only suitable for short texts, since the sentiment information may gradually dissipate as the text becomes longer. To address this issue, a variational autoencoder with interactive variation attention was proposed in this study. Unlike the previous method of directly connecting sentiment information with the latent variables to control text generation, the proposed model adds the sentiment information into variational attention with a dynamic update mechanism. At each timestep, the model leverage both the variational attention and hidden representation to decode and predict the target word and then uses the generated results to update the emotional information in attention. It can keep track of the attention history, which encourages the attention-based VAE to control better the sentiment and content of generating text. The empirical experiments were conducted using the SST dataset to evaluate the generation performance of the proposed model. The comparative results showed that the proposed method outperformed the other methods for affective text generation. In addition, it can still maintain accurate sentiment information and sentences smoothness even in the longer text.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bahuleyan, H., Mou, L., Vechtomova, O., Poupart, P.: Variational attention for sequence-to-sequence models. In: Proceedings of the 27th International Conference on Computational Linguistics, COLING, pp. 1672–1682 (2018)
Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Józefowicz, R., Bengio, S.: Generating sentences from a continuous space. In: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10–21 (2016)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, NAACL, pp. 4171–4186 (2019)
Hochreiter, S., Schmidhuber, J., Elvezia, C.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Hu, Z., Yang, Z., Liang, X., Salakhutdinov, R., Xing, E.P.: Toward controlled generation of text. In: Proceedings of the International Conference on Machine Learning, ICML, pp. 1587–1596 (2017)
Jain, P., Agrawal, P., Mishra, A., Sukhwani, M., Laha, A., Sankaranarayanan, K.: Story generation from sequence of independent short descriptions (2017)
Kingma, D.P., Rezende, D.J., Mohamed, S., Welling, M.: Semi-supervised learning with deep generative models. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, ICNIPS, pp. 3581–3589 (2014)
Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. In: Proceedings of the 2nd International Conference on Learning Representations, ICLR (2014)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: Proceedings of the 7th International Conference on Learning Representations, ICLR. OpenReview.net (2019)
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1532–1543 (2014)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Socher, R., et al.: Recursive deep models for semantic compositionality over a sentiment treebank. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1631–1642 (2013)
Stolcke, A.: SRILM - an extensible language modeling toolkit. In: Proceedings of the 7th International Conference on Spoken Language Processing, ICSLP (2002)
Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Proceedings of Advances in Neural Information Processing Systems, NIPS, pp. 3104–3112 (2014)
Wang, J., Yu, L.C., Lai, K.R., Zhang, X.: Tree-structured regional CNN-LSTM model for dimensional sentiment analysis. IEEE/ACM Trans. Audio Speech Lang. Process. 28, 581–591 (2019)
Wang, K., Wan, X.: Automatic generation of sentimental texts via mixture adversarial networks. Artif. Intell. 275, 540–558 (2019)
Zhang, R., Wang, Z., Yin, K., Huang, Z.: Emotional text generation based on cross-domain sentiment transfer. IEEE Access 7, 100081–100089 (2019)
Zhang, Y., Wang, J., Zhang, X.: Learning sentiment sentence representation with multiview attention model. Inf. Sci. 571, 459–474 (2021)
Zhang, Y., Wang, J., Zhang, X.: Personalized sentiment classification of customer reviews via an interactive attributes attention model. Knowl. Based Syst. 226, 107135 (2021)
Zhao, J.J., Kim, Y., Zhang, K., Rush, A.M., LeCun, Y.: Adversarially regularized autoencoders. In: Proceedings of the 35th International Conference on Machine Learning, ICML, vol. 80, pp. 5897–5906 (2018)
Acknowledgement
This work was supported by the National Natural Science Foundation of China (NSFC) under Grants Nos. 61702443, 61966038 and 61762091.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Chen, R., Wang, J., Zhang, X. (2021). Variational Autoencoder with Interactive Attention for Affective Text Generation. In: Wang, L., Feng, Y., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2021. Lecture Notes in Computer Science(), vol 13029. Springer, Cham. https://doi.org/10.1007/978-3-030-88483-3_9
Download citation
DOI: https://doi.org/10.1007/978-3-030-88483-3_9
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-88482-6
Online ISBN: 978-3-030-88483-3
eBook Packages: Computer ScienceComputer Science (R0)