Skip to main content
SpringerLink
Log in

Memory network with hierarchical multi-head attention for aspect-based sentiment analysis

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Aspect-based sentiment analysis is a challenging subtask of sentiment analysis, which aims to identify the sentiment polarities of the given aspect terms in sentences. Previous studies have demonstrated the remarkable progress achieved by memory networks. However, current memory-network-based models cannot fully exploit long-term semantic relationships to the given aspect terms in sentences, which may lead to the loss of aspect information. In this paper, we propose a novel memory network with hierarchical multi-head attention (MNHMA) for aspect-based sentiment analysis. First, we introduce a semantic information extraction strategy based on the rotational unit of memory to acquire long-term semantic information in context and build memory for the memory network. Second, we propose a hierarchical multi-head attention mechanism to preserve aspect information and enable MNHMA to focus on the critical context words to the given aspect terms in sentences. Third, we employ a fully connected layer in each attention layer of the hierarchical multi-head attention layer to simulate the nonlinear transformation of sentiments, thereby acquiring a comprehensive context representation for aspect-level sentiment classification. Experimental results on three commonly used benchmark datasets demonstrate that our MNHMA model outperforms other state-of-the-art models for aspect-based sentiment analysis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Song Y, Wang J, Jiang T et al (2019) Attentional encoder network for targeted sentiment classification. arXiv:1902.09314

  2. Wagner J, Arora P, Cortes S et al (2014) DCU: aspect-based polarity classification for SemEval task 4. In: Proceedings of the 8th international workshop on semantic evaluation (SemEval-2014). Association for computational linguistics, pp 312–317

  3. Lakkaraju H, Bhattacharyya C, Bhattacharya I et al (2011) Exploiting coherence for the simultaneous discovery of latent facets and associated sentiments. In: Proceedings of the 11th SIAM international conference on data mining, SDM, vol 2011, pp 498–509

  4. Prasojo RE, Kacimi M, Nutt W (2015) Entity and aspect extraction for organizing news comments. In: Proceedings of the 24th ACM international conference on information and knowledge management, pp 233–243

  5. Titov I, Mcdonald R (2008) Modeling online reviews with multigrain topic models. In: Proceedings of international conference on World Wide Web, WWW ’08. ACM, Beijing, pp 111–120

  6. Zhao WX, Jiang J, Yan H, Li X (2010) Jointly modeling aspects and opinions with a MaxEnt-LDA hybrid. In: Proceedings of the 2010 conference on empirical methods in natural language processing. association for computational linguistics, pp 56–65

  7. Wang S, Chen Z, Liu B (2016) Mining aspect-specific opinion using a holistic lifelong topic model. In: Proceedings of international conference on World Wide Web, WWW, vol 2016. ACM, Montreal, pp 167–176

  8. Poria S, Chaturvedi I, Cambria E et al (2016) Sentic LDA: improving on LDA with semantic similarity for aspect-based sentiment analysis. In: Proceedings of the 2016 international joint conference on neural networks, pp 4465–4473

  9. Bansal B, Srivastava S (2019) Hybrid attribute based sentiment classification of online reviews for consumer intelligence. Appl Intell 49(1):137–149

    Article  Google Scholar 

  10. Marcheggiani D, Oscar T, Esuli A et al (2014) Hierarchical multi-label conditional random fields for aspect-oriented opinion mining. In: Proceedings of the 36th european conference on information retrieval, pp 273–285

  11. Zainuddin N, Selamat A, Ibrahim R (2018) Hybrid sentiment classification on twitter aspect-based sentiment analysis. Appl Intell 48(5):1218–1232

    Google Scholar 

  12. Vo AD, Nguyen QP, Ock CY (2020) Semantic and syntactic analysis in learning representation based on a sentiment analysis mode. Appl Intell 50(3):663–680

    Article  Google Scholar 

  13. Hayashi T, Fujita H (2019) Word embeddings-based sentence-level sentiment analysis considering word importance. Acta Polytech Hung 16:7–24

    Article  Google Scholar 

  14. Appel O, Chiclana F, Carter J, Fujita H (2017) Cross-ratio uninorms as an effective aggregation mechanism in sentiment analysis. Knowl Based Syst 124:16–22

    Article  Google Scholar 

  15. Pham DH, Le AC (2018) Exploiting multiple word embeddings and one-hot character vectors for aspect-based sentiment analysis. Acoust Bull 103:1–10

    MathSciNet  Google Scholar 

  16. Huang B, Carley KM (2018) Parameterized convolutional neural networks for aspect level sentiment classification. In: Proceedings of the 2018 conference on empirical methods in natural language processing. association for computational linguistics , pp 1091–1096

  17. Xue W, Li T (2018) Aspect based sentiment analysis with gated convolutional networks. In: Proceedings of the 56th annual meeting of the association for computational linguistics (vol. 1: Long Papers). Association for computational linguistics, pp 2514–2523

  18. Tang D, Qin B, Liu T (2016) Effective LSTMs for target-dependent sentiment classification. In: Proceedings of the 26th international conference on computational linguistics. Association for computational linguistics, pp 3298–3307

  19. Wang Y, Huang M, Zhao L, Zhu X (2016) Attention-based LSTM for aspect-level sentiment classification. In: Proceedings of the 2016 conference on empirical methods in natural language processing. association for computational linguistics , pp 606–615

  20. Yang C, Zhang H, Jiang B, Li K (2019) Aspect-based sentiment analysis with alternating coattention networks. Inf Process Manage 56(3):463–478

    Article  Google Scholar 

  21. Ma D, Li S, Zhuang X, Wang H (2017) Interactive attention networks for aspect-level sentiment classification. In: Proceedings of the 26th international joint conference on artificial intelligence, pp 4068–4074

  22. Li X, Bing L, Lam W, Shi B (2018) Transformation networks for target-oriented sentiment classification. In: Proceedings of the 56th annual meeting of the association for computational linguistics (vol. 1: Long Papers). Association for computational linguistics, pp 946–956

  23. Gu S, Zhang L, Hou Y, Song Y (2018) Position-aware bidirectional attention network for aspect-level sentiment analysis. In: Proceedings of the 27th international conference on computational linguistics. Association for computational linguistics, pp 774–784

  24. Li X, Bing L, Li P et al (2018) Aspect term extraction with history attention and selective transformation. In: Proceedings of the 27th international joint conference on artificial intelligence, pp 4194–4200

  25. Shuang K, Ren X, Yang Q et al (2019) AELA-DLSTMS: attention-enabled and location-aware double LSTMs for aspect-level sentiment classification. Neurocomputing 334:25–34

    Article  Google Scholar 

  26. Akhtar MS, Gupta D, Ekbal A, et al. (2017) Feature selection and ensemble construction: a two-step method for aspect based sentiment analysis. Knowl Based Syst 125:116–135

    Article  Google Scholar 

  27. Peng H, Ma Y, Li Y, et al. (2018) Learning multi-grained aspect target sequence for Chinese sentiment analysis. Knowl Based Syst 148:167–176

    Article  Google Scholar 

  28. Jiang T, Wang J, Song Y et al (2019) A Position-aware transformation network for aspect-level sentiment classification. In: Proceedings of the 2019 international joint conference on neural networks

  29. Weston J, Chopra S, Bordes A (2014) Memory Networks. arXiv:1410.3916

  30. Tang D, Qin B, Liu T (2016) Aspect Level Sentiment Classification with Deep Memory Network. arXiv:1605.08900

  31. Chen P, Sun Z, Bing L, Yang W (2017) Recurrent attention network on memory for aspect sentiment analysis. In: Proceedings of the 2017 conference on empirical methods in natural language processing. association for computational linguistics , pp 452–461

  32. Han H, Liu J, Liu G (2018) Attention-Based Memory network for text sentiment classification. IEEE Access 6:68302–68310

    Article  Google Scholar 

  33. Tay Y, Tuan LA, Hui SC (2017) Dyadic memory networks for aspect-based sentiment analysis. In: Proceedings of the 26th ACM international conference on information and knowledge management, pp 107–116

  34. Zhu P, Qian T (2018) Enhanced aspect level sentiment classification with auxiliary memory. In: Proceedings of the 27th international conference on computational linguistics. Association for computational linguistics, pp 1077–1087

  35. Dangovski R, Jing L, Nakov P et al (2019) Rotational unit of memory: a novel representation unit for RNNs with scalable applications. Trans Assoc Comput Linguist 7:121–138

    Article  Google Scholar 

  36. Huang B, Carley KM (2019) Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks. (2019). In: Proceedings of the 2019 conference on empirical methods in natural language processing. association for computational linguistics, pp 5469–5477

  37. Zhang C, Li Q, Song D (2019) Aspect-based sentiment classification with aspect-specific graph convolutional networks. In: Proceedings of the 2019 conference on empirical methods in natural language processing. association for computational linguistics, pp 4568–4578

  38. Vaswani A, Shazeer N, Parmar N et al (2017) Attention is all you need. arXiv:1706.03762

  39. Pontiki M, Galanis D, Pavlopoulos J et al (2014) Semeval-2014 task 4: aspect based sentiment analysis. In: Proceedings of the 8th international workshop on semantic evaluation (SemEval-2014). Association for computational linguistics, pp 27–35

  40. Dong L, Wei F, Tan C et al (2014) Adaptive recursive neural network for target-dependent twitter sentiment classification. In: Proceedings of the 52nd annual meeting of the association for computational linguistics (vol. 2: Short Papers). Association for computational linguistics, pp 49–54

  41. Pennington J, Socher R, Manning CD (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing. association for computational linguistics, pp 1532–1543

  42. Glorot X, Bengio Y (2010) Understanding the difficulty of training deep feedforward neural networks. J Mach Learn Res 9:249– 256

    Google Scholar 

  43. Kingma DP, Ba J (2014) Adam: a Method for Stochastic Optimization. arXiv:1412.6980

  44. Mao Q, Li J, Wang S et al (2019) Aspect-based sentiment classification with attentive neural turing machines. In: Proceedings of the 28th international joint conference on artificial intelligence, pp 5139–5145

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 61672158, 61502105, 61672159, and 61502104, in part by the Industry-Academy Cooperation Project under Grant 2018H6010, in part by the Natural Science Foundation of Fujian Province under Grant 201801795.

Author information

Authors and Affiliations

  1. Fujian Key Laboratory of Network Computing and Intelligent Information Processing, College of Mathematics and Computer Science, Fuzhou University, Fuzhou, 350116, China

    Yuzhong Chen, Tianhao Zhuang & Kun Guo

  2. Key Laboratory of Spatial Data Mining and Information Sharing, Ministry of Education, Fuzhou, 350116, China

    Yuzhong Chen, Tianhao Zhuang & Kun Guo

Authors
  1. Yuzhong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tianhao Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Kun Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kun Guo.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Zhuang, T. & Guo, K. Memory network with hierarchical multi-head attention for aspect-based sentiment analysis. Appl Intell 51, 4287–4304 (2021). https://doi.org/10.1007/s10489-020-02069-5

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-02069-5

Keywords

Search

Navigation