skip to main content
10.1145/3627673.3680101acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Cross-contextual Sequential Optimization via Deep Reinforcement Learning for Algorithmic Trading

Published: 21 October 2024 Publication History

Abstract

High-frequency algorithmic trading has consistently attracted attention in both academic and industrial fields, which is formally modeled as a near real-time sequential decision problem. DRL methods are treated as a promising direction compared with the traditional approaches, as they have shown great potential in chasing maximum accumulative return. However, the financial data gathered from volatile market change rapidly, which makes it dramatically difficult to grasp crucial factors for effective decision-making. Existing works mainly focus on capturing temporal relations while ignoring deriving essential factors across features. Therefore, we propose a DRL-based cross-contextual sequential optimization (CCSO) method for algorithmic trading. In particular, we employ a convolution module in the first stage to derive latent factors via inter-sequence aggregation and apply a well-designed self-attention module in the second stage to capture market dynamics by aggregating temporal intra-sequence details. With the two-stage extractor as encoder and a RNN-based decision-maker as decoder, an Encoder-Decoder module is established as the policy network to conduct potent feature analysis and suggest action plans. Then, we design a dynamic programming based learning method to address the challenge of complex network updates in reinforcement learning, leading to considerable enhancement in learning stability and efficiency. To the best of our knowledge, this is the first work that solves the sequential optimization problem by joint representation of trading data across time and features in the DRL framework. Extensive experiments demonstrate the superior performance of our method compared to other state-of-the-art algorithmic trading approaches in various widely-used metrics.

References

[1]
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150 (2020).
[2]
, Lin Chen and Qiang Gao. 2019. Application of deep reinforcement learning on automated stock trading. In 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS). IEEE, 29--33.
[3]
Dawei Cheng, Fangzhou Yang, Xiaoyang Wang, Ying Zhang, and Liqing Zhang. 2020. Knowledge graph-based event embedding framework for financial quantitative investments. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 2221--2230.
[4]
Dawei Cheng, Fangzhou Yang, Sheng Xiang, and Jin Liu. 2022. Financial time series forecasting with multi-modality graph neural network. Pattern Recognition, Vol. 121 (2022), 108218.
[5]
Zhicheng Cui, Wenlin Chen, and Yixin Chen. 2016. Multi-scale convolutional neural networks for time series classification. arXiv preprint arXiv:1603.06995 (2016).
[6]
Zhongjie Duan, Cen Chen, Dawei Cheng, Yuqi Liang, and Weining Qian. 2022. Optimal Action Space Search: An Effective Deep Reinforcement Learning Method for Algorithmic Trading. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management. 406--415.
[7]
Fuli Feng, Huimin Chen, Xiangnan He, Jie Ding, Maosong Sun, and Tat-Seng Chua. 2019. Enhancing Stock Movement Prediction with Adversarial Training. In IJCAI, Vol. 19. 5843--5849.
[8]
Siyu Gao, Yunbo Wang, and Xiaokang Yang. 2023. StockFormer: Learning Hybrid Trading Machines with Predictive Coding. In IJCAI. 4766--4774.
[9]
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. 2018. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 (2018).
[10]
Li Han, Nan Ding, Guoxuan Wang, Dawei Cheng, and Yuqi Liang. 2023. Efficient Continuous Space Policy Optimization for High-frequency Trading. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 4112--4122.
[11]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, Vol. 9, 8 (1997), 1735--1780.
[12]
Yifan Hu, Peiyuan Liu, Peng Zhu, Dawei Cheng, and Tao Dai. 2024. Adaptive Multi-Scale Decomposition Framework for Time Series Forecasting. arXiv 2406.03751 (2024).
[13]
Chien Yi Huang. 2018. Financial trading as a game: A deep reinforcement learning approach. arXiv preprint arXiv:1807.02787 (2018).
[14]
Zhigang Jin, Yang Yang, and Yuhong Liu. 2020. Stock closing price prediction based on sentiment analysis and LSTM. Neural Computing and Applications, Vol. 32 (2020), 9713--9729.
[15]
Tzu-Ya Lai, Wen Jung Cheng, and Jun-En Ding. 2023. Sequential graph attention learning for predicting dynamic stock trends (student abstract). In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 16244--16245.
[16]
Chang Li, Dongjin Song, and Dacheng Tao. 2019. Multi-task recurrent neural networks and higher-order Markov random fields for stock price movement prediction: Multi-task RNN and higer-order MRFs for stock price classification. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining. 1141--1151.
[17]
Fan Li, Zhiyu Xu, Dawei Cheng, and Xiaoyang Wang. 2024. AdaRisk: Risk-adaptive Deep Reinforcement Learning for Vulnerable Nodes Detection. IEEE Transactions on Knowledge and Data Engineering (2024).
[18]
Hengxu Lin, Dong Zhou, Weiqing Liu, and Jiang Bian. 2021. Learning multiple stock trading patterns with temporal routing adaptor and optimal transport. In Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining. 1017--1026.
[19]
Xiao-Yang Liu, Hongyang Yang, Jiechao Gao, and Christina Dan Wang. 2021. FinRL: Deep reinforcement learning framework to automate trading in quantitative finance. ACM International Conference on AI in Finance (ICAIF) (2021).
[20]
Dongdong Lv, Dong Wang, Meizi Li, and Yang Xiang. 2020. DNN models based on dimensionality reduction for stock trading. Intelligent data analysis, Vol. 24, 1 (2020), 19--45.
[21]
Naseh Majidi, Mahdi Shamsi, and Farokh Marvasti. 2024. Algorithmic trading using continuous action space deep reinforcement learning. Expert Systems with Applications, Vol. 235 (2024), 121245.
[22]
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. 2020. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, Vol. 21, 181 (2020), 1--50.
[23]
Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022. A time series is worth 64 words: Long-term forecasting with transformers. arXiv preprint arXiv:2211.14730 (2022).
[24]
Hao Qian, Hongting Zhou, Qian Zhao, Hao Chen, Hongxiang Yao, Jingwei Wang, Ziqi Liu, Fei Yu, Zhiqiang Zhang, and Jun Zhou. 2024. MDGNN: Multi-Relational Dynamic Graph Neural Network for Comprehensive and Dynamic Stock Investment Prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 14642--14650.
[25]
Molei Qin, Shuo Sun, Wentao Zhang, Haochong Xia, Xinrun Wang, and Bo An. 2024. Earnhft: Efficient hierarchical reinforcement learning for high frequency trading. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 14669--14676.
[26]
Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison Cottrell. 2017. A dual-stage attention-based recurrent neural network for time series prediction. arXiv preprint arXiv:1704.02971 (2017).
[27]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017).
[28]
Omer Berat Sezer and Ahmet Murat Ozbayoglu. 2018. Algorithmic financial trading with deep convolutional neural networks: Time series to image conversion approach. Applied Soft Computing, Vol. 70 (2018), 525--538.
[29]
Alex Sherstinsky. 2020. Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network. Physica D: Nonlinear Phenomena, Vol. 404 (2020), 132306.
[30]
Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe. 2022. Curriculum learning: A survey. International Journal of Computer Vision, Vol. 130, 6 (2022), 1526--1565.
[31]
Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, Vol. 12 (1999).
[32]
Thibaut Théate and Damien Ernst. 2021. An application of deep reinforcement learning to algorithmic trading. Expert Systems with Applications, Vol. 173 (2021), 114632.
[33]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, Vol. 30 (2017).
[34]
Heyuan Wang, Shun Li, Tengjiao Wang, and Jiayi Zheng. 2021. Hierarchical Adaptive Temporal-Relational Modeling for Stock Trend Prediction. In IJCAI. 3691--3698.
[35]
Heyuan Wang, Tengjiao Wang, Shun Li, Jiayi Zheng, Shijie Guan, and Wei Chen. 2022. Adaptive Long-Short Pattern Transformer for Stock Investment Selection. In IJCAI. 3970--3977.
[36]
Xintong Wang, Gary Qiurui Ma, Alon Eden, Clara Li, Alexander Trott, Stephan Zheng, and David Parkes. 2023. Platform behavior under market shocks: A simulation framework and reinforcement-learning based study. In Proceedings of the ACM Web Conference 2023. 3592--3602.
[37]
Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. 2021. DeepTrader: a deep reinforcement learning approach for risk-return balanced portfolio management with market conditions Embedding. In Proceedings of the AAAI conference on artificial intelligence, Vol. 35. 643--650.
[38]
Lex Weaver and Nigel Tao. 2013. The optimal reward baseline for gradient-based reinforcement learning. arXiv preprint arXiv:1301.2315 (2013).
[39]
Hongjie Xia, Huijie Ao, Long Li, Yu Liu, Sen Liu, Guangnan Ye, and Hongfeng Chai. 2024. CI-STHPAN: Pre-trained Attention Network for Stock Selection with Channel-Independent Spatio-Temporal Hypergraph. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 9187--9195.
[40]
Linyi Yang, Jiazheng Li, Ruihai Dong, Yue Zhang, and Barry Smyth. 2022. Numhtml: Numeric-oriented hierarchical transformer model for multi-task financial forecasting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 36. 11604--11612.
[41]
Mengyuan Yang, Xiaolin Zheng, Qianqiao Liang, Bing Han, and Mengying Zhu. 2022. A Smart Trader for Portfolio Management based on Normalizing Flows. In IJCAI. 4014--4021.
[42]
Sungyeob Yoo, Hyunsung Kim, Jinseok Kim, Sunghyun Park, Joo-Young Kim, and Jinwook Oh. 2023. LightTrader: A standalone high-frequency trading system with deep learning inference accelerators and proactive scheduler. In 2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 1017--1030.
[43]
Zhen Zeng, Rachneet Kaur, Suchetha Siddagangappa, Saba Rahimi, Tucker Balch, and Manuela Veloso. 2023. Financial time series forecasting using cnn and transformer. arXiv preprint arXiv:2304.04912 (2023).
[44]
Wentao Zhang, Yilei Zhao, Shuo Sun, Jie Ying, Yonggang Xie, Zitao Song, Xinrun Wang, and Bo An. 2024. Reinforcement Learning with Maskable Stock Representation for Portfolio Management in Customizable Stock Pools. In Proceedings of the ACM on Web Conference 2024. 187--198.
[45]
Yunhao Zhang and Junchi Yan. 2022. Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting. In The Eleventh International Conference on Learning Representations.
[46]
Peng Zhu, Dawei Cheng, Siqiang Luo, Ruyao Xu, Yuqi Liang, and Yifeng Luo. 2022. Leveraging enterprise knowledge graph to infer web events? influences via self-supervised learning. Journal of Web Semantics, Vol. 74 (2022), 100722.

Index Terms

  1. Cross-contextual Sequential Optimization via Deep Reinforcement Learning for Algorithmic Trading

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      CIKM '24: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management
      October 2024
      5705 pages
      ISBN:9798400704369
      DOI:10.1145/3627673
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 21 October 2024

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. deep reinforcement learning
      2. high-frequency trading
      3. policy optimization

      Qualifiers

      • Research-article

      Funding Sources

      Conference

      CIKM '24
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

      Upcoming Conference

      CIKM '25

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • 0
        Total Citations
      • 112
        Total Downloads
      • Downloads (Last 12 months)112
      • Downloads (Last 6 weeks)13
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media