Skip to main content
Springer Nature Link
Log in

Novel Sentiment Analysis from Twitter for Stock Change Prediction

  • Conference paper
  • First Online:
Data Mining and Big Data (DMBD 2022)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1745))

Included in the following conference series:

  • 914 Accesses

Abstract

Literature in behavioral economics and socioeconomics tells us that the public’s sentiment expression affects individual decision-making and hence the market collective decision-making. In this paper, we investigate whether public sentiment drives stock market performance. To be specific, we look at whether there is an association between changes in the Dow Jones Industrial Average (DJIA) and sentiment expression by using a large-scale comprehensive dataset of emotional state swings obtained from Twitter. We analyze relevant textual content on daily Twitter feeds using two sentiment quantification tools: FinBert, which is a categorical indicator that captures positive, neutral, and negative sentiment, and XLNet, which quantifies public sentiment from three types of moods (Positive, Neutral and Negative). Based on the time series dataset of the sentiment indicators, the relationship between public sentiment and DJIA index value is studied through Granger causal analysis and self-organizing fuzzy neural network. In addition, the changes in DJIA closing prices are predicted. Our results show that the accuracy of DJIA predictions can be significantly improved by including information on public sentiment. We have achieved state-of-the-art accuracy when predicting the daily up and down movement of the Dow Jones Industrial Average closing prices.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Fama, E.F.: Efficient capital markets: a review of theory and empirical. J. Finan. 25(2), 383–417 (1970)

    Article  Google Scholar 

  2. Kendall, M.G., Hill, A.B.: The analysis of economic time-series-part I: prices. J. Roy. Stat. Soc. 116(1), 11–34 (1953)

    Article  Google Scholar 

  3. Larson, A.: Measurement of a random process in futures prices. Proc. Annual Meet. 33, 101–112 (1960)

    Google Scholar 

  4. Fama, E.F.: Efficient capital markets: II. J. Finan. 46(5), 1575 (1991). https://doi.org/10.2307/2328565

    Article  Google Scholar 

  5. Sewell, M. (2022) History of the Efficient Market Hypothesis. Research Note, 11 (04): 04, UCL Computer Science. https://www.ucl.ac.uk/computer-science/. Accessed 30 Oct 2022

  6. Shiller, R.J.: From efficient markets theory to behavioral finance. J. Econ. Perspect. 17(1), 83–104 (2003)

    Article  Google Scholar 

  7. Butler, K.C., Malaikah, S.J.: Efficiency and inefficiency in thinly traded stock markets: Kuwait and Saudi Arabia. J. Bank. Finan. 16(1), 197–210 (1992). https://doi.org/10.1016/0378-4266(92)90085-E

    Article  Google Scholar 

  8. Kavussanos, M.G., Dockery, E.: A multivariate test for stock market efficiency: the case of ASE. Appl. Finan. Econ. 11(5), 573–579 (2001). https://doi.org/10.1080/09603100010013006

    Article  Google Scholar 

  9. Schwert, G.W.: Analomies and market efficiency. Handb. Econ. Finan. 1, 939–974 (2003). http://www.nber.org/papers/w9277. Accessed 30 Oct 2022

  10. Bouman, S., Jacobsen, B.: The halloween indicator, “Sell in May and Go Away’’: another puzzle. Am. Econ. Rev. 92(5), 1618–1635 (2002)

    Article  Google Scholar 

  11. Bailey, W., Kumar, A., Ng, D.: Behavioral biases of mutual fund investors. J. Finan. Econ. 102(1), 1–27 (2011). https://doi.org/10.1016/j.jfineco.2011.05.002

    Article  Google Scholar 

  12. Bartram, S.M., Grinblatt, M.: Global market inefficiencies. J. Finan. Econ. 139(1), 234–259 (2021). https://doi.org/10.1016/j.jfineco.2020.07.011

    Article  Google Scholar 

  13. Gruhl, D., Guha, R., Kumar, R., et al.: The predictive power of online chatter. Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 78–87 (2005). https://doi.org/10.1145/1081870.1081883

  14. Mishne, G., Glance, N.: Predicting movie sales from blogger sentiment. In: AAAI Spring Symposium - Technical Report, SS-06-03, pp. 155–158 (2006)

    Google Scholar 

  15. Asur, S., Huberman, B.A.: Predicting the future with social media. In: Proceedings - 2010 IEEE/WIC/ACM International Conference on Web Intelligence, WI 2010, vol. 1, pp. 492–499 (2010). https://doi.org/10.1109/WI-IAT.2010.63

  16. Choi, H., Varian, H.: Predicting the present with google trends. Econ. Rec. 88(SUPPL.1), 2–9 (2012). https://doi.org/10.1111/j.1475-4932.2012.00809.x

  17. Gilbert, E., Karahalios, K.: Widespread worry and the stock market. In: ICWSM 2010 - Proceedings of the 4th International AAAI Conference on Weblogs and Social Media, pp. 58–65 (2010)

    Google Scholar 

  18. Bollen, J., Mao, H., Zeng, X.: Twitter mood predicts the stock market. J. Comput. Sci. 2(1), 1–8 (2011). https://doi.org/10.1016/j.jocs.2010.12.007

    Article  Google Scholar 

  19. Goel, A., Mittal, A.: Stock prediction using twitter sentiment analysis. Cs229.Stanford.Edu, (December), pp. 1–5 (2012). http://cs229.stanford.edu/proj2011/GoelMittal-StockMarketPredictionUsingTwitterSentimentAnalysis.pdf. Accessed 30 Oct 2022

  20. Gokulakrishnan, B., Priyanthan, P., Ragavan, T., Prasath, N., Perera, A.: Opinion mining and sentiment analysis on a Twitter data stream. In: International Conference on Advances in ICT for Emerging Regions (ICTer2012), pp. 182–188 (2012). https://doi.org/10.1109/ICTer.2012.6423033

  21. Choy, M.: Effective listings of function stop words for Twitter. Int. J. Adv. Comput. Sci. Appl. 3(6) (2012). https://doi.org/10.14569/ijacsa.2012.030602

  22. Kharde, V., Sonawane, S.S.: Sentiment analysis of twitter data: a survey of techniques. Int. J. Comput. Appl. 139(11), 5–15 (2016). https://doi.org/10.5120/ijca2016908625

    Article  Google Scholar 

  23. Yang, Z., et al.: XLNet: generalized autoregressive pretraining for Language Understanding (2020). arXiv.org. https://arxiv.org/abs/1906.08237?source=techstories.org. Accessed 30 Oct 2022

  24. Mustapha, M., Krasnashchok, K., Al Bassit, A., Skhiri, S.: Privacy policy classification with XLNet (Short Paper). In: Garcia-Alfaro, J., Navarro-Arribas, G., Herrera-Joancomarti, J. (eds.) DPM/CBT -2020. LNCS, vol. 12484, pp. 250–257. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-66172-4_16

    Chapter  Google Scholar 

  25. Malo, P., et al.: Good debt or bad debt: detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol. 65(4), 782–796 (2013). https://doi.org/10.1002/asi.23062

    Article  Google Scholar 

  26. Babu, N.V., Rawther, F.A.: Multiclass sentiment analysis in text and emoticons of twitter data: a review. Trans. Comput. Sci. Comput. Intell., 61–68 (2021). https://doi.org/10.1007/978-3-030-49500-8_6

  27. Adoma, A.F., Henry, N.-M., Chen, W.: Comparative analyses of Bert, Roberta, Distilbert, and xlnet for text-based emotion recognition. In: 2020 17th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) (2020). https://doi.org/10.1109/iccwamtip51612.2020.9317379

  28. Hussein, S.: Twitter Sentiments Dataset. Mendeley Data, V. 1 (2021). https://doi.org/10.17632/z9zw7nt5h2.1

  29. Colacicchi, L.: Comparison and fine-tuning of methods for financial sentiment analysis (2022). https://dke.maastrichtuniversity.nl/jan.niehues/wp-content/uploads/2022/01/Colacicchi-Thesis.pdf. Accessed 30 Oct 2022

  30. Gao, Z., Feng, A., Song, X., Wu, X.: Target-dependent sentiment classification with BERT. IEEE Access 7, 154290–154299 (2019). https://doi.org/10.1109/ACCESS.2019.2946594

    Article  Google Scholar 

  31. Liu, Z., et al.: Finbert: A pre-trained financial language representation model for financial text mining. In: Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence [Preprint] (2020). https://doi.org/10.24963/ijcai.2020/622

  32. Gadri, S., et al.: Sentiment analysis: developing an efficient model based on machine learning and deep learning approaches. In: Intelligent Computing & Optimization, pp. 237–247 (2022). https://doi.org/10.1007/978-3-030-93247-3_24

  33. Dusane, P., Sujatha, G.: Events of interest extraction from forensic timeline using Natural Language Processing (NLP). In: Proceedings of International Conference on Deep Learning, Computing and Intelligence, pp. 83–94 (2022). https://doi.org/10.1007/978-981-16-5652-1_7

  34. Hiemstra, C., Jones, J.D.: Testing for linear and nonlinear granger causality in the stock price-volume. J. Finan. 49(5), 1639–1664 (1994)

    Google Scholar 

  35. Granger, C.W.J.: Investigating causal relations by econometric models and cross-spectral methods. Econometrica 37, 424–38 (1969)

    Article  MATH  Google Scholar 

  36. Shojaie, A., Fox, E.B.: Granger causality: a review and recent advances. Ann. Rev. Stat. Appl. 9, 289–319 (2022). https://doi.org/10.1146/annurev-statistics-040120-010930

    Article  Google Scholar 

  37. Kuremoto, T., Obayashi, M., Kobayashi, K.: Forecasting time series by SOFNN with reinforcement learning (2015). https://www.semanticscholar.org/paper/Forecasting-Time-Series-by-SOFNN-with-Reinforcement-Kuremoto-Obayashi/8a5ce65e52077303b8dcbe39a3953219e910ca3f/figure/12. Accessed 30 Oct 2022

  38. Leng, G., Ray, A., Mcginnity, T.M., Coleman, S., Maguire, L., Vance, P.: An Interval Type-2 Fuzzy Neural Network for Cognitive Decisions (2014). https://www.researchgate.net/publication/266849980_An_Interval_Type-2_Fuzzy_Neural_Network_for_Cognitive_Decisions. Accessed 30 Oct 2022

  39. Leng, G., Prasad, G., McGinnity, T.M.: An on-line algorithm for creating self-organizing fuzzy neural networks. Neural Netw. 17(10), 1477–1493 (2004). https://doi.org/10.1016/j.neunet.2004.07.009

    Article  MATH  Google Scholar 

  40. Nofer, M., Hinz, O.: Using twitter to predict the stock market. Bus. Inf. Syst. Eng. 57(4), 229–242 (2015). https://doi.org/10.1007/s12599-015-0390-4

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. VisionX LLC., San Jose, USA

    Yang Cui & Haisong Gu

  2. Birmingham Business School, University of Birmingham, Birmingham, UK

    Yucen Jiang

Authors
  1. Yang Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yucen Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haisong Gu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yang Cui .

Editor information

Editors and Affiliations

  1. Peking University, Beijing, China

    Ying Tan

  2. Southern University of Science and Technology, Shenzhen, China

    Yuhui Shi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Cui, Y., Jiang, Y., Gu, H. (2022). Novel Sentiment Analysis from Twitter for Stock Change Prediction. In: Tan, Y., Shi, Y. (eds) Data Mining and Big Data. DMBD 2022. Communications in Computer and Information Science, vol 1745. Springer, Singapore. https://doi.org/10.1007/978-981-19-8991-9_13

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-8991-9_13

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-8990-2

  • Online ISBN: 978-981-19-8991-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics