research-article

Constraint-embedded paraphrase generation for commercial tweets

Authors:

Rajiv RamnathAuthors Info & Claims

ASONAM '21: Proceedings of the 2021 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

Pages 369 - 376

https://doi.org/10.1145/3487351.3490974

Published: 19 January 2022 Publication History

Abstract

Automated generation of commercial tweets has become a useful and important tool in the use of social media for marketing and advertising. In this context, paraphrase generation has emerged as an important problem. This type of paraphrase generation has the unique requirement of requiring certain elements to be kept in the result, such as the product name or the promotion details. To address this need, we propose a Constraint-Embedded Language Modeling (CELM) framework, in which hard constraints are embedded in the text content and learned through a language model. This embedding helps the model learn not only paraphrase generation but also constraints in the content of the paraphrase specific to commercial tweets. In addition, we apply knowledge learned from a general domain to the generation task of commercial tweets. Our model is shown to outperform general paraphrase generation models as well as the state-of-the-art CopyNet model, in terms of paraphrase similarity, diversity, and the ability to conform to hard constraints.

References

[1]

T. L. Tuten, Social media marketing. SAGE Publications Limited, 2020.

[2]

Y. Wang and Y. Yang, "Dialogic communication on social media: How organizations use twitter to build dialogic relationships with their publics," Computers in Human Behavior, vol. 104, p. 106183, 2020.

[3]

W. J. Hutchins and H. L. Somers, An introduction to machine translation. Academic Press London, 1992, vol. 362.

[4]

M. Maybury, Advances in automatic text summarization. MIT press, 1999.

[5]

M. Shardlow, "A survey of automated text simplification," International Journal of Advanced Computer Science and Applications, vol. 4, no. 1, pp. 58--70, 2014.

[6]

N. Duan, D. Tang, P. Chen, and M. Zhou, "Question generation for question answering," in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017, pp. 866--874.

[7]

A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," OpenAI blog, vol. 1, no. 8, p. 9, 2019.

[8]

D. Bahdanau, K. Cho, and Y. Bengio, "Neural machine translation by jointly learning to align and translate," arXiv preprint arXiv:1409.0473, 2014.

[9]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," in Advances in neural information processing systems, 2014, pp. 2672--2680.

Digital Library

[10]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in neural information processing systems, 2017, pp. 5998--6008.

[11]

Z. Hu, Z. Yang, X. Liang, R. Salakhutdinov, and E. P. Xing, "Toward controlled generation of text," arXiv preprint arXiv:1703.00955, 2017.

[12]

K.-H. Zeng, M. Shoeybi, and M.-Y. Liu, "Style example-guided text generation using generative adversarial transformers," arXiv preprint arXiv:2003.00674, 2020.

[13]

Z. Cao, C. Luo, W. Li, and S. Li, "Joint copying and restricted generation for paraphrase," arXiv preprint arXiv:1611.09235, 2016.

[14]

Y. Zhang, G. Wang, C. Li, Z. Gan, C. Brockett, and B. Dolan, "Pointer: Constrained text generation via insertion-based generative pre-training," arXiv preprint arXiv:2005.00558, 2020.

[15]

M. Damonte and S. B. Cohen, "Structural neural encoders for amr-to-text generation," arXiv preprint arXiv:1903.11410, 2019.

[16]

L. Song, A. Wang, J. Su, Y. Zhang, K. Xu, Y. Ge, and D. Yu, "Structural information preserving for graph-to-text generation," in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7987--7998.

[17]

C. Hokamp and Q. Liu, "Lexically constrained decoding for sequence generation using grid beam search," arXiv preprint arXiv:1704.07138, 2017.

[18]

M. Post and D. Vilar, "Fast lexically constrained decoding with dynamic beam allocation for neural machine translation," arXiv preprint arXiv:1804.06609, 2018.

[19]

N. Nayak, D. Hakkani-Tür, M. A. Walker, and L. P. Heck, "To plan or not to plan? discourse planning in slot-value informed sequence to sequence models for language generation." in INTERSPEECH, 2017, pp. 3339--3343.

[20]

S. Sharma, J. He, K. Suleman, H. Schulz, and P. Bachman, "Natural language generation in dialogue using lexicalized and delexicalized data," arXiv preprint arXiv:1606.03632, 2016.

[21]

J. Gu, Z. Lu, H. Li, and V. O. Li, "Incorporating copying mechanism in sequence-to-sequence learning," arXiv preprint arXiv:1603.06393, 2016.

[22]

K. McKeown, "Paraphrasing questions using given and new information," American Journal of Computational Linguistics, vol. 9, no. 1, pp. 1--10, 1983.

Digital Library

[23]

S. Zhao, X. Lan, T. Liu, and S. Li, "Application-driven statistical paraphrase generation," in Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, 2009, pp. 834--842.

[24]

M. Ellsworth and A. Janin, "Mutaphrase: Paraphrasing with framenet," in Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, 2007, pp. 143--150.

[25]

R. Kozlowski, K. F. McCoy, and K. Vijay-Shanker, "Generation of single-sentence paraphrases from predicate/argument structure using lexico-grammatical resources," in Proceedings of the second international workshop on Paraphrasing, 2003, pp. 1--8.

[26]

A. Prakash, S. A. Hasan, K. Lee, V. Datla, A. Qadir, J. Liu, and O. Farri, "Neural paraphrase generation with stacked residual lstm networks," arXiv preprint arXiv:1610.03098, 2016.

[27]

J. Mallinson, R. Sennrich, and M. Lapata, "Paraphrasing revisited with neural machine translation," in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, 2017, pp. 881--893.

[28]

L. Dong, J. Mallinson, S. Reddy, and M. Lapata, "Learning to paraphrase for question answering," arXiv preprint arXiv:1708.06022, 2017.

[29]

S. Ma, X. Sun, W. Li, S. Li, W. Li, and X. Ren, "Query and output: Generating words by querying distributed word representations for paraphrase generation," arXiv preprint arXiv:1803.01465, 2018.

[30]

Z. Li, X. Jiang, L. Shang, and H. Li, "Paraphrase generation with deep reinforcement learning," arXiv preprint arXiv:1711.00279, 2017.

[31]

A. Gupta, A. Agarwal, P. Singh, and P. Rai, "A deep generative framework for paraphrase generation," arXiv preprint arXiv:1709.05074, 2017.

[32]

S. R. Bowman, L. Vilnis, O. Vinyals, A. M. Dai, R. Jozefowicz, and S. Bengio, "Generating sentences from a continuous space," arXiv preprint arXiv:1511.06349, 2015.

[33]

S. Witteveen and M. Andrews, "Paraphrasing with large language models," arXiv preprint arXiv:1911.09661, 2019.

[34]

E. Egonmwan and Y. Chali, "Transformer and seq2seq model for paraphrase generation," in Proceedings of the 3rd Workshop on Neural Generation and Translation, 2019, pp. 249--255.

[35]

S. Wang, R. Gupta, N. Chang, and J. Baldridge, "A task in a suit and a tie: paraphrase generation with semantic augmentation," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 7176--7183.

Digital Library

[36]

M. Chen, Q. Tang, S. Wiseman, and K. Gimpel, "Controllable paraphrase generation with a syntactic exemplar," arXiv preprint arXiv:1906.00565, 2019.

[37]

T. Wang, X. Wang, Y. Qin, B. Packer, K. Li, J. Chen, A. Beutel, and E. Chi, "Cat-gen: Improving robustness in nlp models via controlled adversarial text generation," arXiv preprint arXiv:2010.02338, 2020.

[38]

N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, "Ctrl: A conditional transformer language model for controllable generation," arXiv preprint arXiv:1909.05858, 2019.

[39]

O. Vinyals, M. Fortunato, and N. Jaitly, "Pointer networks," in Advances in neural information processing systems, 2015, pp. 2692--2700.

[40]

N. Miao, H. Zhou, L. Mou, R. Yan, and L. Li, "Cgmh: Constrained sentence generation by metropolis-hastings sampling," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 6834--6842.

Digital Library

[41]

R. H. Susanto, S. Chollampatt, and L. Tan, "Lexically constrained neural machine translation with levenshtein transformer," arXiv preprint arXiv:2004.12681, 2020.

[42]

J. E. Hu, H. Khayrallah, R. Culkin, P. Xia, T. Chen, M. Post, and B. Van Durme, "Improved lexically constrained decoding for translation and monolingual rewriting," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019, pp. 839--850.

[43]

G. Dinu, P. Mathur, M. Federico, and Y. Al-Onaizan, "Training neural machine translation to apply terminology constraints," arXiv preprint arXiv:1906.01105, 2019.

[44]

G. Chen, Y. Chen, Y. Wang, and V. O. Li, "Lexical-constraint-aware neural machine translation via data augmentation," in Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, 2020, pp. 3587--3593.

[45]

M. R. Parvez, S. Chakraborty, B. Ray, and K.-W. Chang, "Building language models for text with named entities," arXiv preprint arXiv:1805.04836, 2018.

[46]

C. Hegde and S. Patil, "Unsupervised paraphrase generation using pre-trained language models," arXiv preprint arXiv:2006.05477, 2020.

[47]

N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," The journal of machine learning research, vol. 15, no. 1, pp. 1929--1958, 2014.

[48]

A. Fan, M. Lewis, and Y. Dauphin, "Hierarchical neural story generation," arXiv preprint arXiv:1805.04833, 2018.

[49]

A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, "The curious case of neural text degeneration," arXiv preprint arXiv:1904.09751, 2019.

[50]

W. B. Dolan and C. Brockett, "Automatically constructing a corpus of sentential paraphrases," in Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.

[51]

W. Xu, A. Ritter, C. Callison-Burch, W. B. Dolan, and Y. Ji, "Extracting lexically divergent paraphrases from twitter," Transactions of the Association for Computational Linguistics, vol. 2, pp. 435--448, 2014.

[52]

W. Lan, S. Qiu, H. He, and W. Xu, "A continuously growing dataset of sentential paraphrases," arXiv preprint arXiv:1708.00391, 2017.

[53]

J. Wieting and K. Gimpel, "Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations," arXiv preprint arXiv:1711.05732, 2017.

[54]

O. Bojar, O. Dušek, T. Kocmi, J. Libovickỳ, M. Novák, M. Popel, R. Sudarikov, and D. Variš, "Czeng 1.6: enlarged czech-english parallel corpus with processing tools dockered," in International Conference on Text, Speech, and Dialogue. Springer, 2016, pp. 231--238.

[55]

M. Honnibal and I. Montani, "spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing," To appear, vol. 7, no. 1, 2017.

[56]

C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Text summarization branches out, 2004, pp. 74--81.

[57]

K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: a method for automatic evaluation of machine translation," in Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311--318.

[58]

N. Reimers and I. Gurevych, "Sentence-bert: Sentence embeddings using siamese bert-networks," arXiv preprint arXiv:1908.10084, 2019.

Cited By

Babakov NDale DGusev IKrotova IPanchenko A(2023)Don’t Lose the Message While Paraphrasing: A Study on Content Preserving Style TransferNatural Language Processing and Information Systems10.1007/978-3-031-35320-8_4(47-61)Online publication date: 14-Jun-2023
https://doi.org/10.1007/978-3-031-35320-8_4

Index Terms

Constraint-embedded paraphrase generation for commercial tweets
1. Applied computing
  1. Arts and humanities
    1. Language translation
  2. Document management and text processing
2. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Language resources
  2. Machine learning
    1. Learning paradigms

Index terms have been assigned to the content through auto-classification.

Recommendations

Paraphrase Generation and Identification at Paragraph-Level
Generative Intelligence and Intelligent Tutoring Systems
Abstract
The availability and growth of tools and natural language generation (NLG) models that are used to paraphrase text could be helping to improve students’ writing and comprehension skills or a threat to intellectual property and educational ...
Exploiting lexical conceptual structure for paraphrase generation
IJCNLP'05: Proceedings of the Second international joint conference on Natural Language Processing

Lexical Conceptual Structure (LCS) represents verbs as semantic structures with a limited number of semantic predicates. This paper attempts to exploit how LCS can be used to explain the regularities underlying lexical and syntactic paraphrases, such as ...
Linguistic resources for paraphrase generation in portuguese: a lexicon-grammar approach
Abstract
This paper presents a new linguistic resource for the generation of paraphrases in Portuguese, based on the lexicon-grammar framework. The resource components include: (i) a lexicon-grammar based dictionary of 2100 predicate nouns co-occurring ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASONAM '21: Proceedings of the 2021 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining

November 2021

693 pages

ISBN:9781450391283

DOI:10.1145/3487351

Editors:
Michele Coscia
IT University of Copenhagen, Denmark
,
Alfredo Cuzzocrea
University of Calabria, Italy
,
Kai Shu
Illinois Institute of Technology
,
General Chairs:
Ralf Klamma
RWTH Aachen University, Germany
,
Sharyn O'Halloran
Columbia University
,
Jon Rokne
University of Calgary, Canada

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGKDD: ACM Special Interest Group on Knowledge Discovery in Data

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 January 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

ASONAM '21

Sponsor:

SIGKDD

ASONAM '21: International Conference on Advances in Social Networks Analysis and Mining

November 8 - 11, 2021

Virtual Event, Netherlands

Acceptance Rates

ASONAM '21 Paper Acceptance Rate 22 of 118 submissions, 19%;

Overall Acceptance Rate 116 of 549 submissions, 21%

Upcoming Conference

KDD '25

Sponsor:
sigkdd
sigkdd

The 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining

August 3 - 7, 2025

Toronto , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
54
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Babakov NDale DGusev IKrotova IPanchenko A(2023)Don’t Lose the Message While Paraphrasing: A Study on Content Preserving Style TransferNatural Language Processing and Information Systems10.1007/978-3-031-35320-8_4(47-61)Online publication date: 14-Jun-2023
https://doi.org/10.1007/978-3-031-35320-8_4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten