Skip to main content

Advertisement

Springer Nature Link
Log in

Memory-enhanced text style transfer with dynamic style learning and calibration

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Text style transfer aims to rephrase a sentence to match the desired style while retaining the original content. As a controllable text generation task, mainstream approaches use content-independent style embedding as control variables to guide stylistic generation. Nonetheless, stylistic properties are context-sensitive even under the same style. For example, “delicious” and “helpful” convey positive sentiments, although they are more likely to describe food and people, respectively. Therefore, desired style signals must vary with the content. To this end, we propose a memory-enhanced transfer method, which learns fine-grained style representation concerning content to assist transfer. Rather than employing static style embedding or latent variables, our method abstracts linguistic characteristics from training corpora and memorizes subdivided content with the corresponding style representations. The style signal is dynamically retrieved from memory using the content as a query, providing a more expressive and flexible latent style space. To address the imbalance between quantity and quality in different content, we further introduce a calibration method to augment memory construction by modeling the relationship between candidate styles. Experimental results obtained using three benchmark datasets confirm the superior performance of our model compared to competitive approaches. The evaluation metrics and case study also indicate that our model can generate diverse stylistic phrases matching context.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

References

  1. Xu J, Sun X, Zeng Q, et al. Unpaired sentiment-to-sentiment translation: a cycled reinforcement learning approach. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, 2018. 979–988

  2. Li J, Jia R, He H, et al. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Louisiana, 2018. 1865–1874

  3. Fan A, Grangier D, Auli M. Controllable abstractive summarization. In: Proceedings of the 2nd Workshop on Neural Machine Translation and Generation, Melbourne, 2018. 45–54

  4. Cao Y, Shui R, Pan L, et al. Expertise style transfer: a new task towards better communication between experts and laymen. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, 2020. 1061–1071

  5. Hu Z, Yang Z, Liang X, et al. Toward controlled generation of text. In: Proceedings of International Conference on Machine Learning, Sydney, 2017. 1587–1596

  6. Shen T, Lei T, Barzilay R, et al. Style transfer from non-parallel text by cross-alignment. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, Long Beach, 2017. 6833–6844

  7. Zhang Z, Ren S, Liu S, et al. Style transfer as unsupervised machine translation. 2018. ArXiv:1808.07894

  8. John V, Mou L, Bahuleyan H, et al. Disentangled representation learning for non-parallel text style transfer. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, 2019. 424–434

  9. Lee J. Stable style transformer: delete and generate approach with encoder-decoder for text style transfer. In: Proceedings of the 13th International Conference on Natural Language Generation, Dublin, 2020. 195–204

  10. Tian Y, Hu Z, Yu Z. Structured content preservation for unsupervised text style transfer. 2018. ArXiv:1810.06526

  11. Dai N, Liang J, Qiu X, et al. Style transformer: unpaired text style transfer without disentangled latent representation. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, 2019. 5997–6007

  12. Yi X, Liu Z, Li W, et al. Text style transfer via learning style instance supported latent space. In: Proceedings of the 29th International Conference on International Joint Conferences on Artificial Intelligence, Yokohama, 2021. 3801–3807

  13. Lample G, Subramanian S, Smith E, et al. Multiple-attribute text rewriting. In: Proceedings of the International Conference on Learning Representations, New Orleans, 2018

  14. Jin D, Jin Z, Hu Z, et al. Deep learning for text style transfer: a survey. 2021. ArXiv:2011.00416

  15. Bordes A, Usunier N, Garcia-Duran A, et al. Translating embeddings for modeling multi-relational data. In: Proceedings of the 27th International Conference on Neural Information Processing Systems, Nevada, 2013. 2787–2795

  16. Gatys L A, Ecker A S, Bethge M. Image style transfer using convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, 2016. 2414–2423

  17. Fu Z, Tan X, Peng N, et al. Style transfer in text: exploration and evaluation. In: Proceedings of the AAAI Conference on Artificial Intelligence, Louisiana, 2018. 663–670

  18. Jin Z, Jin D, Mueller J, et al. IMaT: unsupervised text attribute transfer via iterative matching and translation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, Hong Kong, 2019. 3097–3109

  19. Luo F, Li P, Zhou J, et al. A dual reinforcement learning framework for unsupervised text style transfer. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence, Macao, 2019. 5116–512

  20. Zhang Y, Xu J, Yang P, et al. Learning sentiment memories for sentiment modification without parallel data. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Brussels, 2018. 1103–1108

  21. Malmi E, Severyn A, Rothe S. Unsupervised text style transfer with padded masked language models. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Online, 2020. 8671–8680

  22. Prabhumoye S, Tsvetkov Y, Salakhutdinov R, et al. Style transfer through back-translation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, 2018. 866–876

  23. Wang K, Hua H, Wan X. Controllable unsupervised text attribute transfer via editing entangled latent representation. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, 2019. 11036–11046

  24. He J, Wang X, Neubig G, et al. A probabilistic formulation of unsupervised text style transfer. In: Proceedings of International Conference on Learning Representations, Addis Ababa, 2020

  25. Liu Y, Neubig G, Wieting J. On learning text style transfer with direct rewards. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, 2021. 4262–4273

  26. Xiao F, Pang L, Lan Y, et al. Transductive learning for unsupervised text style transfer. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Punta Cana, 2021. 2510–2521

  27. Wu C, Socher R, Xiong C. Global-to-local memory pointer networks for task-oriented dialogue. In: Proceedings of the 7th International Conference on Learning Representations, 2019

  28. Yang P, Li L, Luo F, et al. Enhancing topic-to-essay generation with external commonsense knowledge. In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, 2019. 2002–2012

  29. Ayana, Wang Z Y, Xu L, et al. Topic-sensitive neural headline generation. Sci China Inf Sci, 2020, 63: 182103

    Article  MathSciNet  Google Scholar 

  30. Yu W, Zhu C, Li Z, et al. A survey of knowledge-enhanced text generation. 2020. ArXiv:2010.04389

  31. Rousseeuw P J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J Comput Appl Math, 1987, 20: 53–65

    Article  Google Scholar 

  32. Rao S, Tetreault J. Dear sir or madam, may I introduce the GYAFC dataset: corpus, benchmarks and metrics for formality style transfer. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Louisiana, 2018. 129–140

  33. Liu D, Fu J, Zhang Y, et al. Revision in continuous space: unsupervised text style transfer without adversarial learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, New York, 2020. 8376–8383

  34. Devlin J, Chang M W, Lee K, et al. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, 2019. 4171–4186

  35. Loper E, Bird S. NLTK: the natural language toolkit. In: Proceedings of the International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics, Sydney, 2002. 63–70

  36. Heafield K. KenLM: faster and smaller language model queries. In: Proceedings of the 6th Workshop on Statistical Machine Translation, 2011. 187–197

  37. Li J, Galley M, Brockett C, et al. A diversity-promoting objective function for neural conversation models. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2016. 110–119

  38. Pennington J, Socher R, Manning C D. Glove: global vectors for word representation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, Doha, 2014. 1532–1543

  39. Kingma D P, Ba J. Adam: a method for stochastic optimization. In: Proceedings of the International Conference on Learning Representations, San Diego, 2015

  40. Guo Q P, Qiu X P, Xue X Y, et al. Syntax-guided text generation via graph neural network. Sci China Inf Sci, 2021, 64: 152102

    Article  MathSciNet  Google Scholar 

  41. Bai G R, He S Z, Liu K, et al. Example-guided stylized response generation in zero-shot setting. Sci China Inf Sci, 2022, 65: 149103

    Article  Google Scholar 

  42. dos Santos C, Melnyk I, Padhi I. Fighting offensive language on social media with unsupervised text style transfer. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, 2018. 189–194

  43. Tran M, Zhang Y, Soleymani M. Towards a friendly online community: an unsupervised style transfer framework for profanity redaction. In: Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, 2020. 2107–2114

Download references

Acknowledgements

This work was supported by National Natural Science Foundation of China (Grant No. 62106275).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, National University of Defense Technology, Changsha, 410073, China

    Fuqiang Lin, Yiping Song, Wangqun Chen, Diwen Dong & Bo Liu

  2. Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, 999077, China

    Zhiliang Tian

  3. Strategic Assessments and Consultation Institute, Academy of Military Sciences, Beijing, 100097, China

    Bo Liu

Authors
  1. Fuqiang Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yiping Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhiliang Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wangqun Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Diwen Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bo Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yiping Song or Bo Liu.

Ethics declarations

Our work focuses on TST, which controls the stylistic properties of generated text while retaining content semantics. Such methods have a broad impact in the field of controllable natural language generation [40] and can provide strong support for potential real-world applications, e.g., stylized response generation [41], stylistic summarization [3], text simplification [4], and offensive language transfer [42, 43]. Nonetheless, as with all TST methods, our method can also potentially be used maliciously with concealed intentions, including possible content manipulation and forgery issues, e.g., fake review generation. For this reason, we restrict the proposed method to academic use only, and it must be coupled with strict misrepresentation, offensiveness, and bias checks. Furthermore, with increasing attention to shared ethical issues in text generation models, we encourage future studies to address such cases.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lin, F., Song, Y., Tian, Z. et al. Memory-enhanced text style transfer with dynamic style learning and calibration. Sci. China Inf. Sci. 67, 142105 (2024). https://doi.org/10.1007/s11432-022-3726-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-022-3726-0

Keywords

Search

Navigation