Skip to main content
SpringerLink
Log in

Gated three-tower transformer for text-driven stock market prediction

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Effective stock market prediction can significantly assist individual and institutional investors to make better trading decisions and help government stabilize the market. Therefore, a variety of methods have been proposed to tackle the issue of stock market prediction recently. However, it is still quite challenging to effectively extract the correlations and temporal information from multivariate time series of market data and integrate various kinds of features as well as auxiliary information, which is important for improving the performance of stock market prediction. This paper proposes an entirely Transformer based model, namely Gated Three-Tower Transformer (GT3), to incorporate numerical market information and social text information for accurate stock market prediction. Firstly, we devise a Channel-Wise Tower Encoder (CWTE) to capture the channel-wise features from transposed numerical data embeddings. Secondly, we design a Shifted Window Tower Encoder (SWTE) with Multi-Temporal Aggregation to extract and aggregate the multi-scale temporal features from the original numerical data embeddings. Then we adopt the encoder of vanilla Transformer as a Text Tower Encoder (TTE) to obtain the high-level textual features. Furthermore, we design a Cross-Tower Attention mechanism to assist the model to learn the trend-relevant significance of each daily text representation by leveraging the temporal features from SWTE. Finally, we unify CWTE, SWTE, and TTE as the GT3 model through a self-adaptive gate layer to perform end-to-end text-driven stock market prediction by fusing three types of features effectively and efficiently. Extensive experimental results on a real-world dataset show that the proposed model outperforms state-of-the-art baselines.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Notes

  1. https://github.com/chenboluo/GT3

  2. https://docs.quandl.com/docs/data-organization

References

  1. Akita R, Yoshihara A, Matsubara T, Uehara K (2016) Deep learning for stock prediction using numerical and textual information. In: 2016 IEEE/ACIS 15Th international conference on computer and information science (ICIS), IEEE, pp 1–6

  2. Arroyo-Fernández I, Méndez-Cruz C-F, Sierra G, Torres-Moreno J-M, Sidorov G (2019) Unsupervised sentence representations as word information series: revisiting tf–idf. Comput Speech Language 56:107–129

    Article  Google Scholar 

  3. Bagnall A, Lines J, Bostrom A, Large J, Keogh E (2017) The great time series classification bake off: a review and experimental evaluation of recent algorithmic advances. Data Min Knowl Disc 31(3):606–660

    Article  MathSciNet  Google Scholar 

  4. Basak S, Kar S, Saha S, Khaidem L, Dey SR (2019) Predicting the direction of stock market prices using tree-based classifiers. North Amer J Econ Finance 47:552–567

    Article  Google Scholar 

  5. Bhalla VK (2008) Investment Management (Security Analysis and Portfolio Management). S. Chand Publishing

  6. Bollen J, Mao H, Zeng X (2011) Twitter mood predicts the stock market. J Comput Sci 2(1):1–8

    Article  Google Scholar 

  7. Breiman L, Friedman JH, Olshen RA, Stone CJ (1984) Classification and regression tree. wadsworth & brooks Cole Statistics/Probability Series

  8. Butler KC, Jamal Malaikah S (1992) Efficiency and inefficiency in thinly traded stock markets: Kuwait and Saudi Arabia. J Banking Finance 16(1):197–210

    Article  Google Scholar 

  9. Dami S, Esterabi M (2021) Predicting stock returns of tehran exchange using lstm neural network and feature engineering technique. Multimed Tools Appl 80(13):19947–19970

    Article  Google Scholar 

  10. Devlin J, Chang Ming-Wei, Lee K, Toutanova K (2019) BERT: Pre-training Of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: Human language technologies, Volume 1 (Long and Short papers), Minneapolis, Minnesota, Association for Computational Linguistics, pp 4171–4186

  11. Dey R, Salem F (2017) Gate-variants of gated recurrent unit (gru) neural networks. 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS) pages 1597–1600

  12. Ding X, Zhang Y, Liu T, Duan J (2015) Deep learning for event-driven stock prediction. In: Yang Q, Wooldridge MJ (eds) IJCAI, AAAI Press, pp 2327–2333

  13. Franses PH, Ghijsels H (1999) Additive outliers, garch and forecasting volatility. Int J Forecast 15(1):1–9

    Article  Google Scholar 

  14. Gallagher LA, Taylor MP (2002) Permanent and temporary components of stock prices: Evidence from assessing macroeconomic shocks. South Econ J, pp 345–362

  15. Gardner MW, Dorling SR (1998) Artificial neural networks (the multilayer perceptron)—a review of applications in the atmospheric sciences. Atmos Environ 32(14-15):2627–2636

    Article  Google Scholar 

  16. Gervais S, Kaniel R, Mingelgrin DH (2001) The high-volume return premium. J Finance 56(3):877–919

    Article  Google Scholar 

  17. Geurts P (2001) Pattern extraction for time series classification. In: De Raedt L, Siebes A (eds) Principles of data mining and knowledge discovery. Springer, Berlin, pp 115–127

  18. Goodfellow I, Bengio Y, Courville A (2016) Deep Learning. MIT Press. http://www.deeplearningbook.org

  19. Gunduz H, Yaslan Y, Cataltepe Z (2017) Intraday prediction of borsa istanbul using convolutional neural networks and feature correlations. Knowl-Based Syst 137:138–148

    Article  Google Scholar 

  20. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  21. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  22. Hsieh T-Y, Wang S, Sun Y, Honavar V (2021) Explainable multivariate time series classification: a deep neural network which learns to attend to important variables as well as time intervals. In: Proceedings of the 14th ACM international conference on web search and data mining, pp 607–615

  23. Huizhe W, Zhang W, Shen W, Wang J (2018) Hybrid deep sequential modeling for social text-driven stock prediction. In: Proceedings of the 27th ACM international conference on information and knowledge management, pp 1627–1630

  24. Jin X, Zheng WZ, Kong JL, Wang XY, Lin S (2021) Deep-learning forecasting method for electric power load via attention-based encoder-decoder with bayesian optimization. Energies 14(6):1596

    Article  Google Scholar 

  25. Kara Y, Boyacioglu MA, Baykan ÖK (2011) Predicting direction of stock price index movement using artificial neural networks and support vector machines: the sample of the istanbul stock exchange. Expert Syst Appl 38(5):5311–5319

    Article  Google Scholar 

  26. Karim F, Majumdar S, Darabi H, Chen S (2017) Lstm fully convolutional networks for time series classification. IEEE Access 6:1662–1669

    Article  Google Scholar 

  27. Kavussanos MG, Dockery E (2001) A multivariate test for stock market efficiency: the case of ase. Appl Financ Econ 11(5):573–579

    Article  Google Scholar 

  28. Kim T, Kim HY (2019) Forecasting stock prices with a feature fusion lstm-cnn model using different representations of the same data. PloS One 14 (2):e0212320

    Article  Google Scholar 

  29. Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang Y-X, Yan X (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In: Wallach H, Larochelle H, Beygelzimer A, d’Alché-Buc F, Fox E, Garnett R (eds) Advances in neural information processing systems, vol 32. Curran Associates, Inc.

  30. Li S, Jin X, Xuan Y, Zhou X, Chen W, Wang Yu-Xiang, Yan X (2019) Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In: Advances in neural information processing systems, vol 32. Curran Associates, Inc.

  31. Lines J, Bagnall A (2015) Time series classification with ensembles of elastic distance measures. Data Min Knowl Disc 29(3):565–592

    Article  MathSciNet  Google Scholar 

  32. Liu J, Lin H, Liu X, Bo X u, Ren Y, Diao Y, Yang L (2019) Transformer-based capsule network for stock movement prediction. In: Proceedings of the First workshop on financial technology and natural language processing, pp 66–73

  33. Liu M, Ren S, Ma S, Jiao J, Chen Y, Wang Z, Song W (2021) Gated transformer networks for multivariate time series classification. arXiv:2103.14438

  34. Ni H, Wang S, Cheng P (2021) A hybrid approach for stock trend prediction based on tweets embedding and historical prices. World Wide Web, pp 1–20

  35. Oguiza I (2020) tsai - a state-of-the-art deep learning library for time series and sequential data. Github

  36. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: machine learning in Python. J Mach Learn Res 12:2825–2830

    MathSciNet  MATH  Google Scholar 

  37. Rotman M, Wolf L (2021) Shuffling recurrent neural networks. Proceedings of the AAAI Conference on Artificial Intelligence 35 (11):9428–9435

    Google Scholar 

  38. Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323(6088):533–536

    Article  Google Scholar 

  39. Sarantis N (2001) Nonlinearities, cyclical behaviour and predictability in stock markets: international evidence. Int J Forecast 17(3):459–482

    Article  Google Scholar 

  40. Serrà J, Pascual S, Karatzoglou A (2018) Towards a universal neural network encoder for time series. In: CCIA, pp 120–129

  41. Sharaf M, El-Din Hemdan E, El-Sayed A, El-Bahnasawy NA (2021) Stockpred: a framework for stock price prediction. Multimed Tools Appl 80(12):17923–17954

    Article  Google Scholar 

  42. Singh R, Srivastava S (2017) Stock prediction using deep learning. Multimed Tools Appl 76(18):18569–18584

    Article  Google Scholar 

  43. Sun T, Wang J, Ni J, Cao Y u, Liu B (2019) Predicting futures market movement using deep neural networks. In: 2019 18Th IEEE international conference on machine learning and applications (ICMLA), pp 118–125

  44. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. arXiv:1409.3215

  45. Thakkar A, Chaudhari K (2021) A comprehensive survey on deep neural networks for stock market: the need, challenges, and future directions. Expert Syst Appl 177(2):114800

    Article  Google Scholar 

  46. Torres José F, Hadjout D, Sebaa A, Martinez-Alvarez F, Troncoso A (2021) Deep learning for time series forecasting: a survey. Big Data 9 (1):3–21

    Article  Google Scholar 

  47. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. CoRR, arXiv:abs/1706.03762

  48. Virtanen I, Yli-Olli P (1987) Forecasting stock market prices in a thin security market. Omega 15(2):145–155

    Article  Google Scholar 

  49. Wang Z, Yan W, Oates T (2017) Time series classification from scratch with deep neural networks: a strong baseline. In: 2017 International joint conference on neural networks (IJCNN), IEEE, pp 1578–1585

  50. Yong H u, Feng B, Zhang X, Ngai EWT, Liu M (2015) Stock trading rule discovery with an evolutionary trend following model. Expert Syst Appl 42(1):212–222

    Article  Google Scholar 

  51. Yong BX, Rahim MRA, Abdullah AS (2017) A stock market trading system using deep neural network. In: Communications in computer and information science. Springer, Singapore, pp 356–364

  52. Yusen L, Xue J, Raschid L (2021) Predicting the behavior of dealers in over-the-counter corporate bond markets

  53. Zhang L, Aggarwal C, Qi G-J (2017) Stock price prediction via discovering multifrequency trading patterns. In: Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining, KDD ?17. Association for Computing Machinery, New York, pp 2141–2149

  54. Zhou F, Zhou Hao-min, Yang Z, Yang L (2019) Emd2fnn: a strategy combining empirical mode decomposition and factorization machine based neural network for stock market trend prediction. Expert Syst Appl 115:136–151

    Article  Google Scholar 

Download references

Funding

This research is supported by Natural Science Foundation of Zhejiang Province under No.LQ21F020015 and No.LQ20F020015.

Author information

Author notes

    Authors and Affiliations

    1. School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China

      Jia Chen, Tao Chen, Mengqi Shen, Dongjing Wang & Xin Zhang

    2. School of Media and Design, Hangzhou Dianzi University, Hangzhou, China

      Yunhai Shi

    Authors
    1. Jia Chen
      View author publications

      You can also search for this author in PubMed Google Scholar

    2. Tao Chen
      View author publications

      You can also search for this author in PubMed Google Scholar

    3. Mengqi Shen
      View author publications

      You can also search for this author in PubMed Google Scholar

    4. Yunhai Shi
      View author publications

      You can also search for this author in PubMed Google Scholar

    5. Dongjing Wang
      View author publications

      You can also search for this author in PubMed Google Scholar

    6. Xin Zhang
      View author publications

      You can also search for this author in PubMed Google Scholar

    Corresponding author

    Correspondence to Xin Zhang.

    Additional information

    Publisher’s note

    Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

    Jia Chen and Tao Chen contributed equally to this work.

    Rights and permissions

    Reprints and permissions

    About this article

    Check for updates. Verify currency and authenticity via CrossMark

    Cite this article

    Chen, J., Chen, T., Shen, M. et al. Gated three-tower transformer for text-driven stock market prediction. Multimed Tools Appl 81, 30093–30119 (2022). https://doi.org/10.1007/s11042-022-11908-1

    Download citation

    • Received:

    • Revised:

    • Accepted:

    • Published:

    • Issue Date:

    • DOI: https://doi.org/10.1007/s11042-022-11908-1

    Keywords

    Search

    Navigation