Skip to main content
SpringerLink
Log in

A synchronous deep reinforcement learning model for automated multi-stock trading

  • Regular Paper
  • Published:
Progress in Artificial Intelligence Aims and scope Submit manuscript

Abstract

Automated trading is one of the research areas that has benefited from the recent success of deep reinforcement learning (DRL) in solving complex decision-making problems. Despite the large number of researches done, casting the stock trading problem in a DRL framework still remains an open research area due to many reasons, including dynamic extraction of financial data features instead of handcrafted features, applying a scalable DRL technique that can benefit from the huge historical trading data available within a reasonable time frame and adopting an efficient trading strategy. In this paper, a novel multi-stock trading model is presented, based on free-model synchronous multi-agent deep reinforcement learning, which is able to interact with the trading market and to capture the financial market dynamics. The model can be executed on a standard personal computer with multiple core CPU or a GPU in a convenient time frame. The superiority of the proposed model is verified on datasets of different characteristics from the American stock market with huge historical trading data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. https://goldenjumper.wordpress.com/tag/ta-lib/.

References

  1. Hasbrouck., J.: 22 Modeling market microstructure time series, In: Handbook of Statistics, Vol. 14, pp. 647-692, ELSEVIER(1996). https://doi.org/10.1016/S0169-7161(96)14024-4

  2. Pate, J., Shah, S., Thakkar, P.: Predicting stock and stock price index movement using trend deterministic data preparation and machine learning techniques. Exp. Syst. Appl. 42(1), 259–268 (2015). https://doi.org/10.1016/j.eswa.2014.07.040. Elsevier

    Article  Google Scholar 

  3. Cavalcantea, R.C., Brasileirob, R.C., Souza, V.L., Nobrega, J.P., Oliveirab, A.L.I.: Computational intelligence and financial markets: A survey and future directions. Exp. Syst. Appl. 55, 194–211 (2016). https://doi.org/10.1016/j.eswa.2016.02.006

    Article  Google Scholar 

  4. Gupta, P., Mehlawat, M.K., Mittal, G.: Asset portfolio optimization using support vector machines and real-coded genetic algorithm. J. Global Optim. 53(2), 297–315 (2012)

    Article  MathSciNet  Google Scholar 

  5. Yang, B., Gong, Z.-J., Yang, W.: Stock market index prediction using deep neural network ensemble, In: 36th Chinese Control Conference (CCC), pp. 26-28, Dalian, China (2017). https://doi.org/10.23919/ChiCC.2017.8027964

  6. Zhang, J., Shicheng, C., Yan, X., Qianmu, L., Tao, L.: A novel data-driven stock price trend prediction system. Exp. Syst. Appl. 97, 60–69 (2018). https://doi.org/10.1016/j.eswa.2017.12.026

    Article  Google Scholar 

  7. Chonga, E., Han, C., Parka, F.C.: Deep learning networks for stock market analysis and prediction: Methodology, data representations, and case studies. Exp. Syst. Appl. 83, 187–205 (2017)

    Article  Google Scholar 

  8. Lee, J., Kang, J.: Effectively training neural networks for stock index prediction: Predicting the S&P 500 index without using its index data. PLoS ONE V15(4), e0230635 (2020). https://doi.org/10.1371/journal.pone.0230635

    Article  Google Scholar 

  9. Sezer, O., Ozbayoglu, M.: Algorithmic financial trading with deep convolutional neural networks: Time series to image conversion approach. Appl. Soft Comput. 70 (2018) https://doi.org/10.1016/j.asoc.2018.04.024

  10. Jiang, W.: Applications of deep learning in stock market prediction: recent progress, arXiv:2003.01859 (2020), Preprint submitted to Elsevier Journal

  11. Murat, A.M., Omer, M.U., Sezer, B.S.: Deep learning for financial applications : A survey. Appl. Soft Comput. 93, 106384 (2020). https://doi.org/10.1016/j.asoc.2020.106384

    Article  Google Scholar 

  12. Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., Driessche, G., Graepel, T., Hassabis, D.: Mastering the game of go without human knowledge. Int. J. Sci. Nat. 550, 354–359 (2017)

    Google Scholar 

  13. https://colah.github.io/posts/2015-08-Understanding-LSTMs/ Accessed (August 2020)

  14. Watkins, C.J., Dayan, P.: Q-learning, Machine Learning, vol. 8, pp. 279–292. Springer, Berlin (1992)

    MATH  Google Scholar 

  15. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  16. Schmid, Huber J.: Deep learning in neural networks: An overview. Neural Netw. V 61, 85–117 (2015)

    Article  Google Scholar 

  17. Hinton, G.E., Osindero, S., Teh, Y.W.: A fast learning algorithm for deep belief nets. Neural Comput. V18(7), 1527–1554 (2006)

    Article  MathSciNet  Google Scholar 

  18. Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. Adv. Neural Inf. Process. Syst. V(19), 153–160 (2006)

    Google Scholar 

  19. Hasselt, H.V.: Double Q-learning. Adv. Neural Inf. Process. Syst. 23, 2613–2621 (2010)

    Google Scholar 

  20. Wang, Z., Freitas, N., de., Lanctot, M.: Dueling network architectures for deep reinforcement learning, In the International Conference on Machine Learning (ICML), (2015). arXiv preprint arXiv:1511.06581

  21. Hessel, M., Modayil, J., van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., Silver, D.: Rainbow: Combining Improvements in Deep Reinforcement Learning, Thirty-Second AAAI Conference on Artificial Intelligence (2017). arXiv preprint arXiv:1710.02298

  22. Sutton, R.S., McAllester, D.A., Singh, S.P., Mansour, Y.: Policy gradient methods for reinforcement learning with function approximation, Advances in Neural Information Processing Systems, Vol. 12, pp. 1057–1063. (NIPS 1999) MIT Press, Cambridge, MA (2000)

  23. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D.: Continuous control with deep reinforcement learning, In: International Conference Learning Representations (2016). arXiv preprint arXiv:1509.02971

  24. Schulman, J., Levine, S., Abbeel, P., Jordan, M.I., Moritz, P.: Trust Region Policy Optimization, In: 32nd International Conference on Machine Learning, Vol. 37, pp. 1889–1897, PMLR. http://proceedings.mlr.press/v37/schulman15.html(2015)

  25. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.: Proximal policy optimization algorithms, Computing Research Repository (CoRR), 1707.06347 (2017). arXiv preprint arXiv:1707.06347

  26. Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., Freitas, N.: Sample efficient actor-critic with experience replay,ICLR (2016). arXiv preprint arXiv:1611.01224

  27. OpenAI, https://openai.com/, Accessed 1.7 April 2020

  28. OpenAI Baselines: ACKTR & A2C, https://openai.com/blog/baselines-acktr-a2c/, Accessed 17 April 2020

  29. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmil-ler, M.: Playing atari with deep reinforcement learning, In NIPS Deep Learning Work-shop (2013)

  30. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T.P., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning, In: 33rd International Conference on Machine Learning, Vol. 48, pp. 1928-1937, PMLR (2016)

  31. Moody, J., Saffell, M.: Learning to trade via direct reinforcement. IEEE Trans. Neural Netw. 12(4), 875–889 (2001). https://doi.org/10.1109/72.935097

    Article  Google Scholar 

  32. Deng, Y., Bao, F., Youyong, K., Zhiquan, R., Qionghai, D.: Deep direct reinforcement learning for financial signal representation and trading. IEEE Trans. Neural Netw. Learn. Syst. 28(3), 653–664 (2017)

    Article  Google Scholar 

  33. Almahdi, S., Yang, S.Y.: An adaptive portfolio trading system: A risk-return portfolio optimization using recurrent reinforcement learning with expected maximum drawdown. Exp. Syst. Appl. V(87), 267–279 (2017). https://doi.org/10.1016/j.eswa.2017.06.023

    Article  Google Scholar 

  34. Jiang, Z., Xu, D., Liang, J.: A deep reinforcement learning framework for the financial portfolio management problem, arXiv:1706.10059 (2017)

  35. https://github.com/OLPS/OLPS, last accessed October 2020

  36. Li, B., Sahoo, D., S. CH. Hoi.: Olps: A toolbox for online portfolio selection., J. Mach. Learn. Res. (JMLR), (2015)

  37. Li, B., Hoi, S.C.H.: Online portfolio selection: A survey. ACM Comput. Surv. (CSUR) V46(3), 35 (2014)

    MATH  Google Scholar 

  38. https://github.com/ZhengyaoJiang/PGPortfolio , last accessed October 2020

  39. Jiang, Z., Liang, J.: Cryptocurrency portfolio management with deep reinforcement learning., Intelligent Systems Conference., SAI Conferences,2017. Preprint: arXiv:1612.01277

  40. Liang, Z., Chen, H., Zhu, J., Jiang, K., Li, Y.: Adversarial Deep Reinforcement Learning in Portfolio Management, arXiv:1808.09940 (2018)

  41. Hegde, S., Kumar, V., Singh, A.: Risk aware portfolio construction using deep deterministic policy gradients, IEEE Symposium Series on Computational Intelligence (2018)

  42. Wang, J., Zhang, Y., Tang, K., Wu, J., Xiong, Z.: AlphaStock: A Buying Winners-and-Selling-Losers Investment Strategy using Interpretable Deep Reinforcement Attention Networks, 25th ACM SIGKDD, pp.1900-1908 (2019)

  43. Li, Y., Zheng, W., Zheng, Z.: Deep Robust Reinforcement Learning for Practical Algorithmic Trading, IEEE Access, pp.108014–108022 (2019)

  44. Soleymani, F., Elodie, P.: Financial portfolio optimization with online deep reinforcement learning and restricted stacked autoencoder - DeepBreath”. Exp. Syst. Appl. 156, 113456 (2020)

    Article  Google Scholar 

  45. Leem, J., Kim, H.Y.: Action specialized expert ensemble trading system with extended discrete action space using deep reinforcement learning. PLoS ONE 15(7), e0236178 (2020). https://doi.org/10.1371/journal.pone.0236178

    Article  Google Scholar 

  46. Mosavi, A., Ghamisi, P., Faghan, Y., Duan, P.: Shamshirband. Comprehensive Review of Deep Reinforcement Learning Methods and Applications in Economics. (2020). https://doi.org/10.20944/preprints202003.0309.v1

  47. Charpentier, A., Elie, R., Remlinger, C.: Reinforcement Learning in Economics and Finance (2020) arXiv:2003.10014

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, College of Computing and Information Technology, Arab Academy for Science Technology and Maritime transport, Alexandria, Egypt

    Rasha AbdelKawy

  2. Software Engineering Department, College of Computing and Information Technology, Arab Academy for Science Technology and Maritime Transport, Alexandria, Egypt

    Walid M. Abdelmoez

  3. Computer Science and Engineering Department, Egypt-Japan University of Science and Technology, Alexandria, 21934, Egypt

    Amin Shoukry

  4. Computer and Systems Engineering Department, Faculty of Engineering, Alexandria University, Alexandria, 21544, Egypt

    Amin Shoukry

Authors
  1. Rasha AbdelKawy
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Walid M. Abdelmoez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Amin Shoukry
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rasha AbdelKawy.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

AbdelKawy, R., Abdelmoez, W.M. & Shoukry, A. A synchronous deep reinforcement learning model for automated multi-stock trading. Prog Artif Intell 10, 83–97 (2021). https://doi.org/10.1007/s13748-020-00225-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s13748-020-00225-z

Keywords

Search

Navigation