Abstract
Deep reinforcement learning (DRL) is a recently concerned research field for stock portfolio optimization. The existing solutions face various challenge. In this paper, we propose a DRL framework to the stock portfolio optimization problem, which mainly includes the following three contributions: 1) We propose an Over-fitting Prevention Objective Function (OPOF) to avoid over-fitting in the training process. 2) An algorithm called Batch-Forward Recurrent Reinforcement Learning (BFRRL) is proposed to improve the stability of the training process. 3) A neural network called Multi Times Scale Transformer (MTS-Trans) is proposed to enhance stock series local feature extraction ability in multiple time scales. Compared with the current SOTA algorithm, our approach improves returns by 63% in the Chinese stock market and 138% in the U.S. stock market, while the strategy’s risk is also reduced.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271 (2018)
Bakshaev, A.: Market-making with reinforcement-learning (SAC). arXiv preprint arXiv:2008.12275 (2020)
Jiang, Z., Xu, D., Liang, J.: A deep reinforcement learning framework for the financial portfolio management problem. arXiv preprint arXiv:1706.10059 (2017)
Jin, O., El-Saawy, H.: Portfolio management using reinforcement learning. Stanford University (2016)
Kelly Jr., J.L.: A new interpretation of information rate. In: The Kelly Capital Growth Investment Criterion: Theory and Practice, pp. 25–34. World Scientific (2011)
Li, X., Li, Y., Zhan, Y., Liu, X.Y.: Optimistic bull or pessimistic bear: adaptive deep reinforcement learning for stock portfolio allocation. arXiv preprint arXiv:1907.01503 (2019)
Liang, Z., Chen, H., Zhu, J., Jiang, K., Li, Y.: Adversarial deep reinforcement learning in portfolio management. arXiv preprint arXiv:1808.09940 (2018)
Lim, B., Arık, S.Ö., Loeff, N., Pfister, T.: Temporal fusion transformers for interpretable multi-horizon time series forecasting. Int. J. Forecast. 37, 1748–1764 (2021)
Markowitz, H.: Portfolio selection. J. Financ. 7(1), 77–91 (1952)
Mnih, V., et al.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (2015)
Moody, J., Wu, L., Liao, Y., Saffell, M.: Performance functions and reinforcement learning for trading systems and portfolios. J. Forecast. 17(5–6), 441–470 (1998)
Mozer, M.C.: A focused back-propagation algorithm for temporal pattern recognition. Complex Syst. 3(4), 349–381 (1989)
Vaswani, A., et al.: Attention is all you need. arXiv preprint arXiv:1706.03762 (2017)
Wang, X., Wang, Y., Weng, B., Vinel, A.: Stock2Vec: a hybrid deep learning framework for stock market prediction with representation learning and temporal convolutional network. arXiv preprint arXiv:2010.01197 (2020)
Williams, R.J., Zipser, D.: A learning algorithm for continually running fully recurrent neural networks. Neural Comput. 1(2), 270–280 (1989)
Xiong, Z., Liu, X.Y., Zhong, S., Yang, H., Walid, A.: Practical deep reinforcement learning approach for stock trading. arXiv preprint arXiv:1811.07522 (2018)
Xu, K., Zhang, Y., Ye, D., Zhao, P., Tan, M.: Relation-aware transformer for portfolio policy learning. In: IJCAI (2020)
Acknowledgements
This work was supported in part by the National Nature Science Foundation of China under Grant U2013201 and in part by the Pearl River Talent Plan of Guangdong Province under Grant 2019ZT08X603.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Hu, S., Zheng, H., Chen, J. (2021). A Novel Deep Reinforcement Learning Framework for Stock Portfolio Optimization. In: Mantoro, T., Lee, M., Ayu, M.A., Wong, K.W., Hidayanto, A.N. (eds) Neural Information Processing. ICONIP 2021. Communications in Computer and Information Science, vol 1516. Springer, Cham. https://doi.org/10.1007/978-3-030-92307-5_24
Download citation
DOI: https://doi.org/10.1007/978-3-030-92307-5_24
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-92306-8
Online ISBN: 978-3-030-92307-5
eBook Packages: Computer ScienceComputer Science (R0)