Skip to main content
Springer Nature Link
Log in

A Submodular Optimization-Based VAE-Transformer Framework for Paraphrase Generation

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12430))

  • 3439 Accesses

Abstract

Paraphrase plays an important role in various Natural Language Processing (NLP) problems, such as question answering, information retrieval, conversation systems, etc. Previous approaches mainly concentrate on producing paraphrases with similar semantics, namely fidelity, while recent ones begin to focus on the diversity of generated paraphrases. However, most of the existing models fail to explicitly emphasize on both metrics above. To fill this gap, we propose a submodular optimization-based VAE-transformer model to generate more consistent and diverse phrases. Through extensive experiments on datasets like Quora and Twitter, we demonstrate that our proposed model outperforms state-of-the-art baselines on BLEU, METEOR, TERp and n-distinct grams. Furthermore, through ablation study, our results suggest that incorporating VAE and submodularity functions could effectively promote fidelity and diversity respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.kaggle.com/c/quora-question-pairs.

References

  1. Barzilay, R., Lee, L.: Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In: HLT-NAACL. The Association for Computational Linguistics (2003)

    Google Scholar 

  2. Bolshakov, I.A., Gelbukh, A.: Synonymous paraphrasing using WordNet and Internet. In: Meziane, F., Métais, E. (eds.) NLDB 2004. LNCS, vol. 3136, pp. 312–323. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-27779-8_27

    Chapter  Google Scholar 

  3. Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Józefowicz, R., Bengio, S.: Generating sentences from a continuous space. In: CoNLL, pp. 10–21. ACL (2016)

    Google Scholar 

  4. Elhamifar, E., Sapiro, G., Sastry, S.S.: Dissimilarity-based sparse subset selection. IEEE Trans. Pattern Anal. Mach. Intell. 38(11), 2182–2197 (2016)

    Article  Google Scholar 

  5. Fu, Y., Feng, Y., Cunningham, J.P.: Paraphrase generation with latent bag of words. In: NeurIPS, pp. 13623–13634 (2019)

    Google Scholar 

  6. Gimpel, K., Batra, D., Dyer, C., Shakhnarovich, G.: A systematic exploration of diversity in machine translation. In: EMNLP, pp. 1100–1111. ACL (2013)

    Google Scholar 

  7. Gupta, A., Agarwal, A., Singh, P., Rai, P.: A deep generative framework for paraphrase generation. In: AAAI, pp. 5149–5156. AAAI Press (2018)

    Google Scholar 

  8. Huang, S., Wu, Y., Wei, F., Luan, Z.: Dictionary-guided editing networks for paraphrase generation. In: AAAI, pp. 6546–6553. AAAI Press (2019)

    Google Scholar 

  9. Iyyer, M., Wieting, J., Gimpel, K., Zettlemoyer, L.: Adversarial example generation with syntactically controlled paraphrase networks. In: NAACL-HLT, pp. 1875–1885. Association for Computational Linguistics (2018)

    Google Scholar 

  10. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: ICLR (2014)

    Google Scholar 

  11. Kulesza, A., Taskar, B.: Determinantal point processes for machine learning. Found. Trends Mach. Learn. 5(2–3), 123–286 (2012)

    Article  Google Scholar 

  12. Kumar, A., Bhattamishra, S., Bhandari, M., Talukdar, P.P.: Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In: NAACL-HLT (1), pp. 3609–3619. Association for Computational Linguistics (2019)

    Google Scholar 

  13. Lan, W., Qiu, S., He, H., Xu, W.: A continuously growing dataset of sentential paraphrases. In: EMNLP, pp. 1224–1234. Association for Computational Linguistics (2017)

    Google Scholar 

  14. Li, J., Galley, M., Brockett, C., Gao, J., Dolan, B.: A diversity-promoting objective function for neural conversation models. In: HLT-NAACL, pp. 110–119. The Association for Computational Linguistics (2016)

    Google Scholar 

  15. Li, Z., Jiang, X., Shang, L., Li, H.: Paraphrase generation with deep reinforcement learning. In: EMNLP, pp. 3865–3878. Association for Computational Linguistics (2018)

    Google Scholar 

  16. Liu, D., Liu, G.: A transformer-based variational autoencoder for sentence generation. In: IJCNN, pp. 1–7. IEEE (2019)

    Google Scholar 

  17. McKeown, K.R.: Paraphrasing questions using given and new information. Am. J. Comput. Linguist. 9(1), 1–10 (1983)

    Google Scholar 

  18. Nemhauser, G.L., Wolsey, L.A., Fisher, M.L.: An analysis of approximations for maximizing submodular set functions - I. Math. Program. 14(1), 265–294 (1978)

    Article  MathSciNet  Google Scholar 

  19. Parikh, A.P., Täckström, O., Das, D., Uszkoreit, J.: A decomposable attention model for natural language inference. In: EMNLP, pp. 2249–2255. The Association for Computational Linguistics (2016)

    Google Scholar 

  20. Prakash, A., et al.: Neural paraphrase generation with stacked residual LSTM networks. In: COLING, pp. 2923–2934. ACL (2016)

    Google Scholar 

  21. Quirk, C., Brockett, C., Dolan, W.B.: Monolingual machine translation for paraphrase generation. In: EMNLP, pp. 142–149. ACL (2004)

    Google Scholar 

  22. See, A., Liu, P.J., Manning, C.D.: Get to the point: summarization with pointer-generator networks. In: ACL (1), pp. 1073–1083. Association for Computational Linguistics (2017)

    Google Scholar 

  23. Song, Y., Yan, R., Feng, Y., Zhang, Y., Zhao, D., Zhang, M.: Towards a neural conversation model with diversity net using determinantal point processes. In: AAAI, pp. 5932–5939. AAAI Press (2018)

    Google Scholar 

  24. Stobbe, P., Krause, A.: Efficient minimization of decomposable submodular functions. In: NIPS, pp. 2208–2216. Curran Associates, Inc. (2010)

    Google Scholar 

  25. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: NIPS, pp. 3104–3112 (2014)

    Google Scholar 

  26. Vaswani, A., et al.: Attention is all you need. In: NIPS, pp. 5998–6008 (2017)

    Google Scholar 

  27. Vijayakumar, A.K., et al.: Diverse beam search for improved description of complex scenes. In: AAAI, pp. 7371–7379. AAAI Press (2018)

    Google Scholar 

  28. Zhang, X., Lapata, M.: Sentence simplification with deep reinforcement learning. In: EMNLP, pp. 584–594. Association for Computational Linguistics (2017)

    Google Scholar 

Download references

Acknowledgement

This research work has been funded by the National Natural Science Foundation of China (Grant No. 61772337, U1736207) and the National Key R&D Program of China (2018YFC0832004).

Author information

Authors and Affiliations

  1. School of Electronic Information and Electrical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China

    Xiaoning Fan, Danyang Liu, Yiding Liu, Gongshen Liu & Bo Su

  2. Heinz College, Carnegie Mellon University, Pittsburgh, 15213, USA

    Xuejian Wang

Authors
  1. Xiaoning Fan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Danyang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xuejian Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yiding Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gongshen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Bo Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Gongshen Liu or Bo Su .

Editor information

Editors and Affiliations

  1. ECE & Ingenuity Labs Research Institute, Queen’s University, Kingston, ON, Canada

    Xiaodan Zhu

  2. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Min Zhang

  3. School of Computer Science and Technology, Soochow University, Suzhou, China

    Yu Hong

  4. College of Intelligence and Computing, Tianjin University, Tianjin, China

    Ruifang He

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fan, X., Liu, D., Wang, X., Liu, Y., Liu, G., Su, B. (2020). A Submodular Optimization-Based VAE-Transformer Framework for Paraphrase Generation. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60450-9_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60449-3

  • Online ISBN: 978-3-030-60450-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships