Abstract
The positive text reframing (PTR) task, where the goal is to generate a text that gives a positive perspective to a reader while preserving the original sense of the input text, has attracted considerable attention as one of the natural language generation (NLG). In the PTR task, large annotated pairs of datasets are not available and would be expensive and time-consuming to create. Therefore, how to interpret a diversity of contexts and generate a positive perspective from a small size of the training dataset is still an open problem. In this work, we propose a simple but effective Framework for Decoupling the sentiment Style from the Contents of the text (FDSC) for the PTR task. Different from the previous work on the PTR task that utilizes Pre-trained Language Models (PLM) to directly fine-tune the task-specific labeled dataset such as Positive Psychology Frames (PPF), our FDSC fine-tunes the model for the input sequence with two special symbols to decouple style from the contents. We apply contrastive learning to enhance the model that learns a more robust contextual representation. The experimental results on the PPF dataset, show that our approach outperforms baselines by fine-turning two popular Seq2Seq PLMs, BART and T5, and can achieve better text reframing. Our codes are available online (https://github.com/codesedoc/FDSC).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607 (July 2020)
Fu, Z., Tan, X., Peng, N., Zhao, D., Yan, R.: Style transfer in text: exploration and evaluation. Proc. AAAI Conf. Artif. Intell. 32(1), 663–670 (2018)
Hovy, E.: Generating natural language under pragmatic constraints. J. Pragmat. 11(6), 689–719 (1987)
Jin, D., Jin, Z., Hu, Z., Vechtomova, O., Mihalcea, R.: Deep learning for text style transfer: a survey. Comput. Linguist. 48(1), 155–205 (2022)
Lee, S., Lee, D.B., Hwang, S.J.: Contrastive learning with adversarial perturbations for conditional text generation. In: International Conference on Learning Representations (2021)
Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871–7880 (2020)
Li, J., Jia, R., He, H., Liang, P.: Delete, retrieve, generate: a simple approach to sentiment and style transfer. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1865–1874. Association for Computational Linguistics (2018)
Lin, C.Y.: ROUGE: a package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81 (2004)
Liu, D., Fu, J., Zhang, Y., Pal, C., Lv, J.: Revision in continuous space: unsupervised text style transfer without adversarial learning. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 8376–8383 (2020)
Loria, S.: textblob documentation. Release 0.16 2 (2018)
McDonald, D.D., Pustejovsky, J.D.: A computational theory of prose style for natural language generation. In: Second Conference of the European Chapter of the Association for Computational Linguistics, pp. 187–193 (1985)
Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311–318 (2002)
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners. OpenAI Blog 1(8), 9 (2019)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020)
Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020)
Rao, S., Tetreault, J.: Dear sir or madam, may I introduce the GYAFC dataset: corpus, benchmarks and metrics for formality style transfer. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 129–140 (2018)
Reif, E., Ippolito, D., Yuan, A., Coenen, A., Callison-Burch, C., Wei, J.: A recipe for arbitrary text style transfer with large language models. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 837–848. Association for Computational Linguistics (2022)
Su, Y., Lan, T., Wang, Y., Yogatama, D., Kong, L., Collier, N.: A contrastive framework for neural text generation. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022). https://openreview.net/forum?id=V88BafmH9Pj
Tan, W., Heffernan, K., Schwenk, H., Koehn, P.: Multilingual representation distillation with contrastive learning. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 1477–1490 (May 2023)
Touvron, H., et al.: Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)
Wei, J., et al.: Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022)
Xu, R., Ge, T., Wei, F.: Formality style transfer with hybrid textual annotations. arXiv preprint arXiv:1903.06353 (2019)
Yang, Z., Hu, Z., Dyer, C., Xing, E.P., Berg-Kirkpatrick, T.: Unsupervised text style transfer using language models as discriminators. In: Advances in Neural Information Processing Systems, vol. 31 (2018)
Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: evaluating text generation with bert. In: International Conference on Learning Representations, pp. 74–81 (2020)
Zhang, Y., Ge, T., Sun, X.: Parallel data augmentation for formality style transfer. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 3221–3228 (2020)
Ziems, C., Li, M., Zhang, A., Yang, D.: Inducing positive perspectives with text reframing. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3682–3700 (2022)
Acknowledgments
We would like to thank anonymous reviewers for their comments and suggestions. This work is supported by SCAT, JKA, Kajima Foundation’s Support Program, and JSPS KAKENHI (No. 21K12026, 22K12146, and 23H03402). The first author is supported by JST, the establishment of university fellowships towards the creation of science technology innovation, Grant Number JPMJFS2117.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Xu, S., Suzuki, Y., Li, J., Fukumoto, F. (2024). Decoupling Style from Contents for Positive Text Reframing. In: Luo, B., Cheng, L., Wu, ZG., Li, H., Li, C. (eds) Neural Information Processing. ICONIP 2023. Communications in Computer and Information Science, vol 1967. Springer, Singapore. https://doi.org/10.1007/978-981-99-8178-6_6
Download citation
DOI: https://doi.org/10.1007/978-981-99-8178-6_6
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-8177-9
Online ISBN: 978-981-99-8178-6
eBook Packages: Computer ScienceComputer Science (R0)