Skip to main content
SpringerLink
Log in

Predicting stock market trends using machine learning algorithms via public sentiment and political situation analysis

  • Focus
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Stock market trends can be affected by external factors such as public sentiment and political events. The goal of this research is to find whether or not public sentiment and political situation on a given day can affect stock market trends of individual companies or the overall market. For this purpose, the sentiment and situation features are used in a machine learning model to find the effect of public sentiment and political situation on the prediction accuracy of algorithms for 7 days in future. Besides, interdependencies among companies and stock markets are also studied. For the sake of experimentation, stock market historical data are downloaded from Yahoo! Finance and public sentiments are obtained from Twitter. Important political events data of Pakistan are crawled from Wikipedia. The raw text data are then pre-processed, and the sentiment and situation features are generated to create the final data sets. Ten machine learning algorithms are applied to the final data sets to predict the stock market future trend. The experimental results show that the sentiment feature improves the prediction accuracy of machine learning algorithms by 0–3%, and political situation feature improves the prediction accuracy of algorithms by about 20%. Furthermore, the sentiment attribute is most effective on day 7, while the political situation attribute is most effective on day 5. SMO algorithm is found to show the best performance, while ASC and Bagging show poor performance. The interdependency results indicate that stock markets in the same industry show a medium positive correlation with each other.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

Notes

  1. https://www.wikipedia.org.

  2. http://www.stocktwits.com.

  3. https://www.finance.yahoo.com.

  4. https://www.google.com/finance.

  5. http://www.nlp.stanford.edu/.

References

  • Ahuja R, Rastogi H, Choudhuri A, Garg B (2015) Stock market forecast using sentiment analysis. In: IEEE 2nd international conference on computers for sustainable global development, pp 1008–1010

  • Beaulieu MC, Cosset JC, Essaddam N (2005) The impact of political risk on the volatility of stock returns: the case of Canada. J Int Bus Stud 36(6):701–718

    Article  Google Scholar 

  • Billsus D, Pazzani MJ (2000) User modelling for adaptive news access. User Modell User Adapt Interact 10(2–3):147–180

    Article  Google Scholar 

  • Bing L, Chan KC, Ou C (2014) Public sentiment analysis in Twitter data for prediction of a company’s stock price movements. In: 2014 IEEE 11th international conference on e-Business engineering (ICEBE), pp 232–239

  • Bollen J, Mao H, Pepe A (2011) Modeling public mood and emotion: twitter sentiment and socio-economic phenomena. In: 5th international AAAI conference on weblogs and social media

  • Bollerslev T (1986) Generalized autoregressive conditional heteroscedasticity. J Econ China 31(3):307–327

    MATH  Google Scholar 

  • Cardoso B, Almeida R, Dias M, Coelho G (2008) Structural reliability analysis using Monte Carlo simulation and neural networks. Adv Eng Soft 39(6):505–513

    Article  Google Scholar 

  • Chen DH, Bin FS, Chen CD (2005) The impacts of political events on foreign institutional investors and stock returns: emerging market evidence from Taiwan. Int J Bus 10(2)

  • Chou J, Lin C (2012) Predicting disputes in public-private partnership projects: classification and ensemble models. J Comput Civil Eng 27(1):51–60

    Article  Google Scholar 

  • Chouliaras A (2015) High frequency newswire textual sentiment analysis: evidence from international stock markets during the European financial crisis. Available at SSRN 2572597

  • Deng W, Zhao H, Zou L, Li G, Yang X, Wu D (2017a) A novel collaborative optimization algorithm in solving complex optimization problems. J Soft Comput 21(15):4387–4398. https://doi.org/10.1007/s00500-016-2071-8

    Article  Google Scholar 

  • Deng W, Zhao H, Yang X, Xiong J, Sun M, Li B (2017b) Study on an improved adaptive PSO algorithm for solving multi-objective gate assignment. Appl Soft Comput 59:288–302. https://doi.org/10.1016/j.asoc.2017.06.004

    Article  Google Scholar 

  • Deng W, Xu J, Zhao H (2019a) An improved ant colony optimization algorithm based on hybrid strategies for scheduling problem. IEEE Access 7:20281–20292. https://doi.org/10.1109/ACCESS.2019.2897580

    Article  Google Scholar 

  • Deng W, Yao R, Zhao H, Yang X, Li G (2019b) A novel intelligent diagnosis method using optimal LS-SVM with improved PSO algorithm. J Soft Comput 23(7):2445–2462. https://doi.org/10.1007/s00500-017-2940-9

    Article  Google Scholar 

  • Dey A, Miyani G, Sil A (2019) Application of artificial neural network (ANN) for estimating reliable service life of reinforced concrete (RC) structure bookkeeping factors responsible for deterioration mechanism. Soft Comput. https://doi.org/10.1007/s00500-019-04042-y

    Article  Google Scholar 

  • Egeli B., Badur B, Ozturan M, Badur B (2003) Stock market prediction using artificial neural networks, In: Proceedings of the 3rd Hawaii international conference on business, pp 1–8

  • Fan Y, Ying SJ, Wang BH, Wei YM (2009) The effect of investor psychology on the complexity of stock market: an analysis based on cellular automaton model. Comput Ind Eng 56(1):63–69. https://doi.org/10.1016/j.cie.2008.03.015

    Article  Google Scholar 

  • Frank E, Hall MA, Witten IH (2016) The WEKA workbench. Online appendix for data mining: practical machine learning tools and techniques, 4th edn. Morgan Kaufmann, Burlington

    Google Scholar 

  • Gidofalvi G, Elkan C (2001) Using news articles to predict stock price movements. Department of Computer Science and Engineering, University of California, San Diego

    Google Scholar 

  • Go A, Bhayani R, Huang L (2009) Twitter sentiment classification using distant supervision. CS224 N project report, Stanford 1(1)

  • Hagenau M, Liebmann M, Neumann D (2013) Automated news reading: stock price prediction based on financial news using context-capturing features. Decis Support Syst 55(3):685–697

    Article  Google Scholar 

  • Hegazy O, Soliman OS, Salam MA (2014) A machine learning model for stock market prediction. Int J Comput Sci Telecom 4(12):16–23

    Google Scholar 

  • Jeffrey B (2011) Twitter text mining. https://www.jeffreybreen.wordpress.com/2011/07/04/twitter-text-mining-r-slides/. Accessed 23 June 2018

  • Kara Y, Boyacioglu MA, Baykan OK (2011) Predicting direction of stock price index movement using artificial neural networks and support vector machines: the sample of the Istanbul stock exchange. Exp Syst Appl 38(5):5311–5319

    Article  Google Scholar 

  • Kazem A, Sharifi E, Hussain FK, Saberi M, Hussain OK (2013) Support vector regression with chaos-based firefly algorithm for stock market price forecasting. Appl Soft Comput 13(2):947–958

    Article  Google Scholar 

  • Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. Ijcai 14(2):1137–1145

    Google Scholar 

  • Lakshmi V, Harika K, Bavishya H, Sri Harsha C (2017) Sentiment analysis of twitter data. Int Res J Eng Technol 4(2):2224–2227

    Google Scholar 

  • Lee H, Surdeanu M, MacCartney B, Jurafsky D (2014) On the importance of text analysis for stock price prediction. LREC 2014:1170–1175

    Google Scholar 

  • Li Q, Wang T, Li P, Liu L, Gong Q, Chen Y (2014a) The effect of news and public mood on stock movements. Inf Sci 278:826–840

    Article  Google Scholar 

  • Li X, Xie H, Chen L, Wang J, Deng X (2014b) News impact on stock price return via sentiment analysis. Knowl Based Syst 69:14–23

    Article  Google Scholar 

  • Lu N (2016) A machine learning approach to automated trading. M.S. thesis. Department of Computer Science, Boston College, Boston, USA

  • Makrehchi M, Shah S, Liao W (2013) Stock prediction using event-based sentiment analysis. In: 2013 IEEE/WIC/ACM international joint conference on WI and IAT 1, pp 337–342

  • Malik S, Hussain S, Ahmed S (2009) Impact of political event on trading volume and stock returns: the case of KSE. Int Rev Bus Res Papers 5(4):354–364

    Google Scholar 

  • Mostafa MM (2010) Forecasting stock exchange movements using neural networks: empirical evidence from Kuwait. Exp Syst Appl 37(9):6302–6309

    Article  Google Scholar 

  • Murphy JJ (1999) Technical analysis of the financial markets: a comprehensive guide to trading methods and applications. Penguin, London

    Google Scholar 

  • Naderpour H, Mirrashid M (2019) Shear failure capacity prediction of concrete beam–column joints in terms of ANFIS and GMDH. Pract Period Struct Des Const 24(2):04019006. https://doi.org/10.1061/(ASCE)SC.1943-5576.0000417

    Article  Google Scholar 

  • Naderpour H, Mirrashid M, Nagai K (2019) An innovative approach for bond strength modelling in FRP strip-to-concrete joints using adaptive neuro–fuzzy inference system. Eng Comput. https://doi.org/10.1007/s00366-019-00751-y

    Article  Google Scholar 

  • Nguyen TH, Shirai K, Velcin J (2015) Sentiment analysis on social media for stock movement prediction. Exp Syst Appl 42(24):9603–9611

    Article  Google Scholar 

  • Oh C, Chong, Sheng O (2011) Investigating predictive power of stock micro blog sentiment in forecasting future stock price directional movement. In: ICIS, Shanghai, China

  • Olaniyi SAS, Adewole KS, Jimoh RG (2011) Stock trend prediction using regression analysis–a data mining approach. ARPN J Syst Softw 1(4):154–157

    Google Scholar 

  • Oliveira N, Cortez P, Areal N (2013) On the predictability of stock market behavior using stocktwits sentiment and posting volume. Portuguese conference on artificial intelligence. Springer, Berlin, pp 355–365

    Google Scholar 

  • Ou P, Wang H (2009) Prediction of stock market index movement by ten data mining techniques. Mod Appl Sci 3(12):28

    Article  Google Scholar 

  • Pak A, Paroubek P (2010) Twitter as a corpus for sentiment analysis and opinion mining. LREc 10(2010):1320–1326

    Google Scholar 

  • Papadrakakis M, Lagaros D (2002) Reliability-based structural optimization using neural networks and Monte Carlo simulation. Comput Methods Appl Mech Eng 191(32):3491–3507

    Article  Google Scholar 

  • R Core Team (2018) R: a language and environment for statistical computing. R foundation for statistical computing, Vienna, Austria. https://www.R-project.org/ Accessed 7 May 2018

  • Rahman A, Ali A (2016) Sentiment analysis on Twitter data. B.S. thesis. Department of Computer Science and Engineering, East West University, Dhaka, Bangladesh

  • Revelle, W (2018) psych: procedures for personality and psychological research, Northwestern University, Evanston, Illinois, USA. https://CRAN.R-project.org/package=psychversion=1.8.4. Accessed 10 May 2018

  • Sadhukhan S, Dhadekar M, Bhonar S (2016) Stock market prediction using artificial neural networks. Imp J Interdiscip Res 2(5)

  • Schumaker RP, Chen H (2009) Textual analysis of stock market prediction using breaking financial news: the AZFin text system. ACM Trans Inf Syst (TOIS) 27(2):12

    Article  Google Scholar 

  • Shahbaz P, Ahmad B, Reza EA, Jalal JM (2014) Stock market forecasting using artificial neural networks. Eur Online J Nat Soc Sci 2(3):2404–2411

    Google Scholar 

  • Shen S, Jiang H, Zhang T (2012) Stock market forecasting using machine learning algorithms. Department of Electrical Engineering, Stanford University, Stanford

    Google Scholar 

  • Socher R, Perelygin A, Wu J, Chuang J, Manning CD, Ng A, Potts C (2013) Recursive deep models for semantic compositionality over a sentiment Treebank. In: Proceedings of the 2013 conference on empirical methods in natural language processing, pp 1631–1642

  • Suleman MT (2012) Stock market reaction to good and bad political news. Asian J Finance Acc 4(1):299–312

    Google Scholar 

  • Taimur M, Khan S (2015) Impact of political and catastrophic events on stock returns. VFAST Trans Edu Soc Sci 6(1):21–32

    Google Scholar 

  • Tang X, Yang C, Zhou J (2009) Stock price forecasting by combining news mining and time series analysis. IEEE/WIC/ACM Int Jt Conf Web Intel Intel Agent Technol 1:279–282

    Google Scholar 

  • Tayal D, Komaragiri S (2009) Comparative analysis of the impact of blogging and micro-blogging on market performance. Int J Comput Sci Eng 1(3):176–182

    Google Scholar 

  • Turner T, (2007) A beginner’s guide to day trading online. 2nd edn, Adams Media

  • Yuan B (2016) Sentiment analysis of Twitter data. M.S. thesis, Department of Computer Science, Rensselaer Polytechnic Institute, New York

  • Zhao H, Sun M, Deng W, Yang X (2016) A new feature extraction method based on EEMD and multi-scale fuzzy entropy for motor bearing. Entropy 19(1):14. https://doi.org/10.3390/e19010014

    Article  Google Scholar 

  • Zhao H, Yao R, Xu L, Yuan Y, Li G, Deng W (2018) Study on a novel fault damage degree identification method using high-order differential mathematical morphology gradient spectrum entropy. Entropy 20(9):682. https://doi.org/10.3390/e20090682

    Article  Google Scholar 

  • Zhou Z, Zhao J, Xu K (2016) Can online emotions predict the stock market in China?. In: International conference on web information systems engineering, pp 328–342

Download references

Acknowledgements

The authors would like to thank all the reviewers for their constructive comments.

Author information

Authors and Affiliations

  1. Department of Software Engineering, University of Engineering and Technology, Taxila, 47050, Pakistan

    Wasiat Khan

  2. LITIS Lab, Normandy University, INSA Rouen, 76000, Rouen, France

    Usman Malik

  3. School of Architecture, Computing and Engineering, University of East London, London, UK

    Mustansar Ali Ghazanfar

  4. Department of Computer Engineering, University of Engineering and Technology, Taxila, 47050, Pakistan

    Muhammad Awais Azam

  5. Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah, Saudi Arabia

    Khaled H. Alyoubi & Ahmed S. Alfakeeh

Authors
  1. Wasiat Khan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Usman Malik
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mustansar Ali Ghazanfar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Muhammad Awais Azam
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Khaled H. Alyoubi
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Ahmed S. Alfakeeh
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wasiat Khan.

Ethics declarations

Conflict of interest

Authors declare that he/she has no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Additional information

Communicated by M. D. Lytras.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Khan, W., Malik, U., Ghazanfar, M.A. et al. Predicting stock market trends using machine learning algorithms via public sentiment and political situation analysis. Soft Comput 24, 11019–11043 (2020). https://doi.org/10.1007/s00500-019-04347-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-019-04347-y

Keywords

Search

Navigation