Abstract
Recurrent neural networks are increasingly used to classify text data, displacing feed-forward networks. This article is a demonstration of how to classify text using Long Term Term Memory (LSTM) network and their modifications, i.e. Bidirectional LSTM network and Gated Recurrent Unit. We present the superiority of this method over other algorithms for text classification on the example of three sets: Spambase Data Set, Farm Advertisement and Amazon book reviews. The results of the first two datasets were compared with AdaBoost ensemble of feedforward neural networks. In the case of the last database, the result is compared to the bag-of-words algorithm. In this article, we focus on classifying two groups in the first two collections, since we are only interested in whether something is classified into a SPAM or an eligible message. In the last dataset, we distinguish three classes.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Auli, M., Galley, M., Quirk, C., Zweig, G.: Joint language and translation modeling with recurrent neural networks. In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, pp. 1044–1054, October 2013 Association for Computational Linguistics (2013)
Bas, E.: The training of multiplicative neuron model based artificial neural networks with differential evolution algorithm for forecasting. J. Artif. Intell. Soft Comput. Res. 6(1), 5–11 (2016)
Bengio, Y., Simard, P., Frasconi, P.: Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5(2), 157–166 (1994)
Bertini Junior, J.R., Nicoletti, M.D.C.: Enhancing constructive neural network performance using functionally expanded input data. J. Artif. Intell. Soft Comput. Res. 6(2), 119–131 (2016)
Britz, D.: Recurrent neural network tutorial, part 4 - implementing a GRU/LSTM RNN with python and theano. http://www.wildml.com/. Accessed 27 Oct 2015
Chen, C., Xia, L.: Recurrent neural network and long short-term memory. http://ace.cs.ohiou.edu/~razvan/courses/dl6890/presentations/lichen-lijie.pdf
Cho, K., Van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014). arXiv preprint arXiv:1406.1078
Gers, F.A., Schmidhuber, E.: LSTM recurrent networks learn simple context-free and context-sensitive languages. IEEE Trans. Neural Netw. 12(6), 1333–1340 (2001)
Gers, F.A., Schmidhuber, J., Cummins, F.: Learning to forget: continual prediction with LSTM. Neural Comput. 12(10), 2451–2471 (2000)
Gers, F.A., Schraudolph, N.N., Schmidhuber, J.: Learning precise timing with LSTM recurrent networks. J. Mach. Learn. Res. 3, 115–143 (2002)
Graves, A.: Supervised sequence labelling. In: Supervised Sequence Labelling with Recurrent Neural Networks, pp. 5–13. Springer (2012)
Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional lstm and other neural network architectures. Neural Netw. 18(5), 602–610 (2005)
Hochreiter, S.: The vanishing gradient problem during learning recurrent neural nets and problem solutions. Int. J. Uncertainty Fuzz. Knowl. Based Syst. 6(02), 107–116 (1998)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRFmodels for sequence tagging (2015). arXiv preprint arXiv:1508.01991
Murata, M., Ito, S., Tokuhisa, M., Ma, Q.: Order estimation of Jaanese paragraphs by supervised machine learning and various textual features. J. Artif. Intell. Soft Comput. Res. 5(4), 247–255 (2015)
Olah, C.: Understanding LSTM networks, August 2015. http://colah.github.io/posts/2015-08-Understanding-LSTMs
Patgiri, C., Sarma, M., Sarma, K.K.: A class of neuro-computational methods for assamese fricative classification. J. Artif. Intell. Soft Comput. Res. 5(1), 59–70 (2015)
Shuang Bi, W.Z.: Cs294-1 final project algorithms comparison deep learning neural network — adaboost — random forest. http://bid.berkeley.edu/cs294-1-spring13/images/0/0d/ProjectReport(Shuang_and_Wenchang).pdf. Accessed 15 May 2013
Taspinar, A.: Sentiment analysis with bag-of-words. http://ataspinar.com/2016/01/21/sentiment-analysis-with-bag-of-words/. Accessed 21 Jan 2016
Trask, A.: Anyone can learn to code an LSTM-RNN in python. https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/. Accessed 15 Nov 2015
Trofimovich, J.: Comparison of neural network architectures for sentiment analysis of Russian tweets, 1–4 June 2016
Williams, R.J., Zipser, D.: Gradient-based learning algorithms for recurrent networks and their computational complexity. In: Backpropagation: Theory, Architectures, and Applications, vol. 1, pp. 433–486 (1995)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Nowak, J., Taspinar, A., Scherer, R. (2017). LSTM Recurrent Neural Networks for Short Text and Sentiment Classification. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L., Zurada, J. (eds) Artificial Intelligence and Soft Computing. ICAISC 2017. Lecture Notes in Computer Science(), vol 10246. Springer, Cham. https://doi.org/10.1007/978-3-319-59060-8_50
Download citation
DOI: https://doi.org/10.1007/978-3-319-59060-8_50
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-59059-2
Online ISBN: 978-3-319-59060-8
eBook Packages: Computer ScienceComputer Science (R0)