Skip to main content
SpringerLink
Log in

Deep Text Generation – Using Hierarchical Decomposition to Mitigate the Effect of Rare Data Points

  • Conference paper
  • First Online:
Language, Data, and Knowledge (LDK 2017)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 10318))

Included in the following conference series:

Abstract

Deep learning has recently been adopted for the task of natural language generation (NLG) and shown remarkable results. However, learning can go awry when the input dataset is too small or not well balanced with regards to the examples it contains for various input sequences. This is relevant to naturally occurring datasets such as many that were not prepared for the task of natural language processing but scraped off the web and originally prepared for a different purpose. As a mitigation to the problem of unbalanced training data, we therefore propose to decompose a large natural language dataset into several subsets that “talk about” the same thing. We show that the decomposition helps to focus each learner’s attention during training. Results from a proof-of-concept study show 73% times faster learning over a flat model and better results.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://www.weather.gov/.

References

  1. Angeli, G., Liang, P., Klein, D.: A simple domain-independent probabilistic approach to generation. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Cambridge, Massachusetts (2010)

    Google Scholar 

  2. Belz, A., Gatt, A.: Intrinsic vs. extrinsic evaluation measures for referring expression generation. In: Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics (ACL), Columbus, OH, USA (2008)

    Google Scholar 

  3. Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)

    Article  Google Scholar 

  4. Cho, K., van Merrienboer, B., Gülçehre, Ç., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Doha, Qatar (2014)

    Google Scholar 

  5. Cuayáhuitl, H., Dethlefs, N., Hastie, H., Liu, X.: Training a statistical surface realiser from automatic slot labelling. In: Proceedings of the IEEE Workshop on Spoken Language Technology (SLT), South Lake Tahoe, USA (2014)

    Google Scholar 

  6. Dethlefs, N., Cuayáhuitl, H.: Hierarchical reinforcement learning and hidden markov models for task-oriented natural language generation. In: Proceedings of the 49th Annual Conference of the Association for Computational Linguistics (ACL-HLT), Short Papers, Portland, OR, USA (2011)

    Google Scholar 

  7. Dethlefs, N., Cuayáhuitl, H.: Hierarchical reinforcement learning for situated natural language generation. Natl. Lang. Eng. 21, 391–435 (2015)

    Article  Google Scholar 

  8. Dethlefs, N., Hastie, H., Cuayáhuitl, H., Lemon, O.: Conditional random fields for responsive surface realisation. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), Sofia, Bulgaria (2013)

    Google Scholar 

  9. Dusek, O., Jurcicek, F.: Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), Berlin, Germany (2016)

    Google Scholar 

  10. Graves, A.: Generating sequences with recurrent neural networks. CoRR abs/1308.0850 (2013). http://arxiv.org/abs/1308.0850

  11. Konstas, I., Lapata, M.: Unsupervised concept-to-text generation with hypergraphs. In: Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), Montreal, Canada (2012)

    Google Scholar 

  12. Liang, P., Jordan, M., Klein, D.: Learning semantic correspondences with less supervision. In: Proceedings of the 47th Annual Meeting of the Association for Computational Linguistics (ACL), Singapore (2009)

    Google Scholar 

  13. Mairesse, F., Jurčíček, F., Keizer, S., Thomson, B., Yu, K., Young, S.: Phrase-based statistical language generation using graphical models and active learning. In: Proceedings of the 48th Annual Meeting of the Association of Computational Linguistics (ACL), Uppsala, Sweden (2010)

    Google Scholar 

  14. Mei, H., Bansal, M., Walker, M.: What to talk about and how? Selective Generation using LSTMs with coarse-to-fine alignment. In: Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), San Diego, CA, USA (2016)

    Google Scholar 

  15. Morin, F., Bengio, Y.: Hierarchical probabilistic neural network language model. In: Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 246–252 (2005)

    Google Scholar 

  16. Novikova, J., Rieser, V.: The aNALoGuE Challenge: Non Aligned Language GEneration. In: Proceedings of the 9th International Natural Language Generation Conference (INLG) (2016)

    Google Scholar 

  17. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL), Association for Computational Linguistics, pp. 311–318 (2001)

    Google Scholar 

  18. Reiter, E., Dale, R.: Building Natural Language Generation Systems. Cambridge University Press, New York (2000)

    Book  Google Scholar 

  19. Snyder, B., Barzilay, R.: Database-text alignment via structured multilabel classification. In: Proceedings of 20th International Joint Conference on Artificial Intelligence (IJCAI), Hyderabad, India (2007)

    Google Scholar 

  20. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. Adv. Neural Inf. Process. Syst. (NIPS) 27, 3104–3112 (2014)

    Google Scholar 

  21. Turner, A.P., Caves, L.S., Stepney, S., Tyrrell, A.M., Lones, M.A.: Artificial epigenetic networks: automatic decomposition of dynamical control tasks using topological self-modification. IEEE Trans. Neural Netw. Learn. Syst. (2016)

    Google Scholar 

  22. Turner, A.P., Lones, M.A., Fuente, L.A., Stepney, S., Caves, L.S., Tyrrell, A.M.: The artificial epigenetic network. In: 2013 IEEE International Conference on Evolvable Systems (ICES), pp. 66–72. IEEE (2013)

    Google Scholar 

  23. Walker, M., Stent, A., Mairesse, F., Prasad, R.: Individual and domain adaptation in sentence planning for dialogue. J. Artif. Intell. Res. 30(1), 413–456 (2007)

    MATH  Google Scholar 

  24. Wen, T.H., Gašić, M., Mrkšić, N., Su, P.H., Vandyke, D., Young, S.: Semantically conditioned LSTM-based natural language generation for spoken dialogue systems. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2015)

    Google Scholar 

Download references

Acknowledgements

We acknowledge the VIPER high-performance computing facility of the University of Hull and its support team. We are also grateful for Nvidia’s donation of a Titan X Pascal graphics card for our work on deep learning.

Author information

Authors and Affiliations

  1. School of Engineering and Computer Science, University of Hull, Cottingham Road, Hull, HU7 6RX, UK

    Nina Dethlefs & Alexander Turner

Authors
  1. Nina Dethlefs
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander Turner
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nina Dethlefs .

Editor information

Editors and Affiliations

  1. Universidad Politécnica de Madrid, Madrid, Spain

    Jorge Gracia

  2. Nanyang Technological University, Singapore, Singapore

    Francis Bond

  3. Insight Centre for Data Analytics, National University of Ireland, Galway, Galway, Ireland

    John P. McCrae

  4. Insight Centre for Data Analytics, National University of Ireland, Galway, Ireland

    Paul Buitelaar

  5. Goethe-University Frankfurt, Frankfurt, Germany

    Christian Chiarcos

  6. University of Leipzig, Leipzig, Germany

    Sebastian Hellmann

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Dethlefs, N., Turner, A. (2017). Deep Text Generation – Using Hierarchical Decomposition to Mitigate the Effect of Rare Data Points. In: Gracia, J., Bond, F., McCrae, J., Buitelaar, P., Chiarcos, C., Hellmann, S. (eds) Language, Data, and Knowledge. LDK 2017. Lecture Notes in Computer Science(), vol 10318. Springer, Cham. https://doi.org/10.1007/978-3-319-59888-8_25

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-59888-8_25

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-59887-1

  • Online ISBN: 978-3-319-59888-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation