Abstract
Text classification task includes total sentence sentimental classification as well as target-based sentimental classification. Target-based sentimental analysis and classification is aiming at locating sentimental classes of given sentences over different opinion aspects. Recurrent neural network is perfectly suitable for this kind of assignment, and it does achieve the state-of-the-art (SOTA) performance by now. Most of the previous works model target and context words with Recurrent Neural Network (RNN) with attention mechanism. However, RNN can hardly parallelize to train and cause too much memory occupancy. What’s more, for this task, long-term memory may cause confusion. For example, the food is delicious but the service is frustrating, where the model may think the food is good while the service is bad. Convolutional neural network (CNN) seems vital in this situation as it can learn the local n-grams information while RNN cannot make it. To address these issues, this paper comes up with an Attention Transformer Network (ATNet) which can perfectly address issues above. Our model employs attention mechanism and transformer component to generate target-orient representation, along with CNN layers to extract N-grams information. On open benchmark datasets, our proposed models achieve state-of-art results, namely, 70.3%, 72.1% and 83.4% in three benchmarks. Also, this paper applies pretrained BERT in the encoder part and acquires SOTA achievement. We performed many contrast experiments to elaborate effectiveness of our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Jiang, L., Yu, M., Zhou, M., et al.: Target-dependent twitter sentiment classification. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pp. 151–160. Association for Computational Linguistics (2011)
Mikolov, T., Karafiát, M., Burget, L., et al.: Recurrent neural network based language model. In: Eleventh Annual Conference of the International Speech Communication Association (2010)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Li, X., Bing, L., Lam, W., et al.: Transformation networks for target-oriented sentiment classification. arXiv preprint arXiv:1805.01086 (2018)
Mohammad, S., Kiritchenko, S., Sobhani, P., et al.: A dataset for detecting stance in tweets. In: LREC (2016)
Suykens, J.A.K., Vandewalle, J.: Least squares support vector machine classifiers. Neural Process. Lett. 9(3), 293–300 (1999)
Scholkopf, B., Smola, A.J.: Learning with Kernels: Support Vector Machines, Regularization, Optimization, and Beyond. MIT Press, Cambridge (2001)
Tang, D., Qin, B., Liu, T.: Aspect level sentiment classification with deep memory network. arXiv preprint arXiv:1605.08900 (2016)
Wang, Y., Huang, M., Zhao, L.: Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 606–615 (2016)
Yang, M., Tu, W., Wang, J., et al.: Attention based LSTM for target dependent sentiment classification. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)
Ma, D., Li, S., Zhang, X., et al.: Interactive attention networks for aspect-level sentiment classification. arXiv preprint arXiv:1709.00893 (2017)
Chen, P., Sun, Z., Bing, L., et al.: Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 452–461 (2017)
Dey, R., Salemt, F.M.: Gate-variants of gated recurrent unit (GRU) neural networks. In: 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), pp. 1597–1600. IEEE (2017)
Fan, F., Feng, Y., Zhao, D.: Multi-grained attention network for aspect-level sentiment classification. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 3433–3442 (2018)
Huang, B., Ou, Y., Carley, K.M.: Aspect level sentiment classification with attention-over-attention neural networks. In: Thomson, R., Dancy, C., Hyder, A., Bisgin, H. (eds.) SBP-BRiMS 2018. LNCS, vol. 10899, pp. 197–206. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-93372-6_22
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014)
Devlin, J., Chang, M.W., Lee, K., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Song, Y., Wang, J., Jiang, T., et al.: Attentional encoder network for targeted sentiment classification. arXiv preprint arXiv:1902.09314 (2019)
Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Liu, Q., Zhang, H., Zeng, Y., et al.: Content attention model for aspect based sentiment analysis. In: Proceedings of the 2018 World Wide Web Conference on World Wide Web, pp. 1023–1032. International World Wide Web Conferences Steering Committee (2018)
Zhang, L., Wang, S., Liu, B.: Deep learning for sentiment analysis: a survey. Wiley Interdisc. Rev.: Data Min. Knowl. Discovery 8(4), e1253 (2018)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Zhou, Z., Zhang, W., Wang, J.: Inception score, label smoothing, gradient vanishing and-log (d(x)) alternative. arXiv preprint arXiv:1708.01729 (2017)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Jozefowicz, R., Zaremba, W., Sutskever, I.: An empirical exploration of recurrent network architectures. Int. Conf. Mach. Learn., 2342–2350 (2015)
Kim, Y.: Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882 (2014)
Tang, D., Qin, B., Feng, X., et al.: Effective LSTMs for target-dependent sentiment classification. arXiv preprint arXiv:1512.01100 (2015)
Acknowledgement
This work is sponsored by National Key Research and Development Project (No. 213), the National Nature Science Foundation of China (No. 61573259, No. 61673299, No. 61673301, No. 61573255) and the Special Project of the Ministry of Public Safety (No. 20170004). Supported by Key Laboratory of Information Network Safety, Ministry of Public Safety No. C18608. It is also supported by Shanghai Health and Family Planning Commission Chinese Medicine Science and Technology Innovation Project (ZYKC201702005).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Tong, J., Chen, W., Wei, Z. (2019). Attentional Transformer Networks for Target-Oriented Sentiment Classification. In: Jin, H., Lin, X., Cheng, X., Shi, X., Xiao, N., Huang, Y. (eds) Big Data. BigData 2019. Communications in Computer and Information Science, vol 1120. Springer, Singapore. https://doi.org/10.1007/978-981-15-1899-7_19
Download citation
DOI: https://doi.org/10.1007/978-981-15-1899-7_19
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-1898-0
Online ISBN: 978-981-15-1899-7
eBook Packages: Computer ScienceComputer Science (R0)
