Skip to main content

Advertisement

Springer Nature Link
Log in

A dual deep neural network with phrase structure and attention mechanism for sentiment analysis

An ablation experiment on Chinese short financial texts

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Sentiment analysis of short texts is difficult for their simplicity and compactness. This goes a step further when it comes to the Chinese texts. Although deep learning achieved better accuracy in sentiment analysis, there is a lack of explain-ability. Thus, this paper evaluates the effectiveness of techniques for sentiment analysis of Chinese short financial texts with deep learning. For this, we built a Chinese short financial texts corpus (CSFC) and designed an ablation experiment. Beside the CFSC, we used a Chinese review collection and an English short-text repository in the experiment for comparison. There are five techniques involved. They are the Pinyin, the segmentation, the lexical analysis, the phrase structure and the attention mechanism. As results, we found that the phrase structure and the attention mechanism are two of the best. Therefore, the best model in the experiment is called a Phrase Structure and Attention-based Deep network model (PhraSAD). Moreover, to improve the classification accuracy on neutral data, we use a dual classifier strategy for 3-class problems. Experimental results showed that PhraSAD outperformed all other compared models on all experimental datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. The corpus will be available online after publication.

  2. A social networking website: http://stocktwits.com.

  3. https://github.com/fxsjy/jieba.

  4. For example, every sentence in the CSFC corpus is annotated by three experts with a positive, negative or neutral label. Hence, we can use the CSFC to train the classifiers.

  5. https://github.com/mozillazg/python-pinyin.

  6. https://pypi.org/project/jieba/.

  7. http://code.google.com/archive/p/word2vec/.

  8. https://stanfordnlp.github.io/CoreNLP/.

  9. https://github.com/fip-lab/Sentiment Analysis.

  10. http://guba.eastmoney.com/.

  11. https://github.com/SophonPlus/ChineseNlpCorpus.

  12. http://thinknook.com/wp-content/uploads/2012/09/Sentiment-Analysis-Dataset.zip.

  13. http://thuctc.thunlp.org/.

  14. https://github.com/yoonkim/CNN sentence.

  15. We fine-tuned the BERT-based, Chinese [6] model with our corpus (i.e., the CFSC). After 30 epochs, the accuracy is 74.5%. However, given more resources and times, the result could be better. This is not listed in the table because one motivation of this paper is to discover explanations.

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv: 1409.0473 (2014)

  2. Batra, R., Daudpota, S.M.: Integrating StockTwits with sentiment analysis for better prediction of stock price movement. In: Computing, Mathematics and Engineering Technologies (iCoMET). pp. 1–5. IEEE (2018)

  3. Chen J, Yan S, Wong KC (2018) Verbal aggression detection on twitter comments: Convolutional neural network for short-text sentiment analysis. Neural Comput Appl 32:1–10

    Google Scholar 

  4. Chen T, Xu R, He Y, Xia Y, Wang X (2016) Learning user and product distributed representations using a sequence model for sentiment analysis. IEEE Comput Intell Mag 11(3):34–44

    Article  Google Scholar 

  5. Cho, K., Van Merrie¨nboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint:1406.1078 (2014)

  6. Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. CoRR abs/1810.04805 (2018), arXiv: 1810.04805

  7. Dong L, Wei F, Liu S, Zhou M, Xu K (2015) A statistical parsing framework for sentiment classification. Comput Linguist 41(2):293–336

    Article  MathSciNet  Google Scholar 

  8. Du Y, Zhao X, He M, Guo W (2019) A novel capsule based hybrid neural network for sentiment classification. IEEE Access 7:39321–39328

    Article  Google Scholar 

  9. Kaplanski G, Levy H (2010) Sentiment and stock prices: The case of aviation disasters. J Financ Econ 95(2):174–201

    Article  Google Scholar 

  10. Kim, Y.: Convolutional neural networks for sentence classification. In EMNLP pp.1746–1751 (2014)

  11. La Su, Y., Liu, W.W., et al.: Research on the LSTM Mongolian and Chinese machine translation based on morpheme encoding. Neural Computing and Applications pp. 1–9 (2018)

  12. Lazaridou, A., Titov, I., Sporleder, C.: A Bayesian model for joint unsupervised induction of sentiment, aspect and discourse representations. In: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). vol. 1, pp. 1630–1639 (2013)

  13. Li, J., Sun, M., Zhang, X.: A comparison and semi-quantitative analysis of words and character-bigrams as features in Chinese text categorization. In: Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. pp. 545–552 (2006)

  14. Li, L., Goh, T.T., Jin, D.: How textual quality of online reviews affect classification performance: a case of deep learning sentiment analysis. Neural Computing and Applications pp. 1–29 (2018)

  15. Li W, Liu P, Zhang Q, Liu W (2019) An improved approach for text sentiment classification based on a deep neural network via a sentiment attention mechanism. Future Internet 11(4):96

    Article  Google Scholar 

  16. Li, X., Meng, Y., Sun, X., Han, Q., Yuan, A., Li, J.: Is word segmentation necessary for deep learning of Chinese representations? In: Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 3242–3252. Association for Computational Linguistics, Florence, Italy (Jul 2019), https://www.aclweb.org/anthology/P19-1314

  17. Lin, Z., Feng, M., Santos, C.N.d., Yu, M., Xiang, B., Zhou, B., Bengio, Y.: A structured self-attentive sentence embedding. arXiv preprint arXiv: 1703.03130 (2017)

  18. Liu, P., Qiu, X., Huang, X.: Dynamic compositional neural networks over tree structure. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19–25, 2017. pp. 4054–4060 (2017), https://doi.org/10.24963/ijcai.2017/566

  19. Liu, Y., Chen, Y.: Research on Chinese micro-blog sentiment analysis based on deep learning. In: Computational Intelligence and Design (ISCID). vol. 1, pp. 358–361. IEEE (2015)

  20. Long W, Tang Yr, Tian Yj (2018) Investor sentiment identification based on the universum SVM. Neural Comput Appl 30(2):661–670

    Article  Google Scholar 

  21. Luong MT, Frank MC, Johnson M (2013) Parsing entire discourses as very long strings: Capturing topic continuity in grounded language learning. Transactions of the Association of Computational Linguistics 1:315–326

    Article  Google Scholar 

  22. Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D.: The Stanford CoreNLP natural language processing toolkit. In: Association for Computational Linguistics (ACL) System Demonstrations. pp. 55–60 (2014), http://www.aclweb.org/anthology/P/P14/P14-5010

  23. Mnih, V., Heess, N., Graves, A., et al.: Recurrent models of visual attention. In: Advances in neural information processing systems. pp. 2204–2212 (2014)

  24. Nagarajan SM, Gandhi UD (2019) Classifying streaming of twitter data based on sentiment analysis using hybridization. Neural Comput Appl 31(5):1425–1433

    Article  Google Scholar 

  25. Ouyang, X., Zhou, P., Li, C.H., Liu, L.: Sentiment analysis using convolutional neural network. In: Computer and Information Technology. pp. 2359–2364. IEEE (2015) Peng, H.: Linguistic-inspired Chinese sentiment analysis: from characters to radicals and phonetics. Ph.D. thesis (2019)

  26. Ruan, X., Wilson, S., Mihalcea, R.: Finding optimists and pessimists on twitter. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). vol. 2, pp. 320–325 (2016)

  27. Silhavy R, Senkerik R, Oplatkova ZK, Silhavy P, Prokopova Z (2016) Artificial Intelligence Perspectives in Intelligent Systems. Springer, Berlin

    Book  Google Scholar 

  28. Tai, K.S., Socher, R., Manning, C.D.: Improved semantic representations from tree-structured long short-term memory networks. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26–31, 2015, Beijing, China, Volume 1: Long Papers. pp. 1556–1566 (2015), http://aclweb.org/anthology/P/P15/P15–1150.pdf

  29. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems. pp. 5998–6008 (2017)

  30. Vilares D, Alonso MA, Gomez-Rodrıguez C (2015) A syntactic approach for opinion mining on Spanish reviews. Natural Language Engineering 21(1):139–163

    Article  Google Scholar 

  31. Wang, J., Wang, Z., Zhang, D., Yan, J.: Combining knowledge with deep convolutional neural networks for short text classification. In: Proceedings of IJCAI. vol. 350 (2017)

  32. Wang L, Niu J, Song H, Atiquzzaman M (2018) Sentirelated: A cross-domain sentiment classification algorithm for short texts through sentiment related index. J Netw Comput Appl 101:111–119

    Article  Google Scholar 

  33. Wu L, Hoi SC, Yu N (2010) Semantics-preserving bag-of-words models and applications. IEEE Trans Image Process 19(7):1908–1920

    Article  MathSciNet  Google Scholar 

  34. Wu L, Morstatter F, Liu H (2018) Slangsd: building, expanding and using a sentiment dictionary of slang words for short-text sentiment classification. Language Resour Eval 52(3):839–852. https://doi.org/10.1007/s10579-018-9416-0

    Article  Google Scholar 

  35. Yang Q, Rao Y, Xie H, Wang J, Wang FL, Chan WH, Cambria C (2019) Segment-level joint topic-sentiment model for online review analysis. IEEE Intell Syst 34(1):43–50

    Article  Google Scholar 

  36. Yenter, A., Verma, A.: Deep CNN-LSTM with combined kernels from multiple branches for imdb review sentiment analysis. In: Ubiquitous Computing, Electronics and Mobile Communication Conference (UEMCON). pp. 540–546. IEEE (2017)

  37. Yogatama, D., Smith, N.A.: Linguistic structured sparsity in text categorization. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 786–796 (2014)

  38. Yu Y, Duan W, Cao Q (2013) The impact of social and conventional media on firm equity value: A sentiment analysis approach. Decis Support Syst 55(4,SI):919–926

    Article  Google Scholar 

  39. Zeng, J., Li, J., Song, Y., Gao, C., Lyu, M.R., King, I.: Topic memory networks for short text classification. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 3120–3131 (2018), https: //aclanthology.info/papers/D18–1351/d18–1351

  40. Zhang H, Huang W, Liu L, Chow TW. Learning to match clothing from textual feature-based compatible relationships. IEEE Transactions on Industrial Informatics. 2019 Jun 24.

  41. Zhang H, Li J, Ji Y, Yue H (2016) Understanding subtitles by character-level sequence-to-sequence learning. IEEE Trans Industr Inf 13(2):616–624

    Article  Google Scholar 

  42. Zhu, X., Sobhani, P., Guo, H.: Dag-structured long short-term memory for semantic compositionality. In: NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12–17, 2016. pp. 917–926 (2016), http://aclweb.org/anthology/N/N16/N16-1106.pdf

Download references

Author information

Authors and Affiliations

  1. School of Computer, Guangdong University of Technology, Guangzhou, 510006, People’s Republic of China

    Dongning Rao & Sihong Huang

  2. Department of Computer Science, Jinan University, Guangzhou, 510632, People’s Republic of China

    Zhihua Jiang

  3. Department of Computer Science Engineering, Chandigarh University, Mohali, 140413, Punjab, India

    Ganesh Gopal Deverajan

  4. Department of Computer Science and Engineering, Velagapudi Ramakrishna Siddhartha Engineering College, Vijayawada, 520007, Andhra Pradesh, India

    Rizwan Patan

Authors
  1. Dongning Rao
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Sihong Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Zhihua Jiang
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Ganesh Gopal Deverajan
    View author publications

    You can also search for this author inPubMed Google Scholar

  5. Rizwan Patan
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Rizwan Patan.

Ethics declarations

Conflict of interest

There is no conflict of interest between the authors to publishing this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rao, D., Huang, S., Jiang, Z. et al. A dual deep neural network with phrase structure and attention mechanism for sentiment analysis. Neural Comput & Applic 33, 11297–11308 (2021). https://doi.org/10.1007/s00521-020-05652-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05652-6

Keywords