skip to main content
10.1145/3490354.3494372acmconferencesArticle/Chapter ViewAbstractPublication PagesicaifConference Proceedingsconference-collections
research-article

Towards a fully rl-based market simulator

Published: 04 May 2022 Publication History

Abstract

We present a new financial framework where two families of RL-based agents representing the Liquidity Providers and Liquidity Takers learn simultaneously to satisfy their objective. Thanks to a parametrized reward formulation and the use of Deep RL, each group learns a shared policy able to generalize and interpolate over a wide range of behaviors. This is a step towards a fully RL-based market simulator replicating complex market conditions particularly suited to study the dynamics of the financial market under various scenarios.

References

[1]
Yakov Amihud and Haim Mendelson. 1980. Dealership market: Market-making with inventory. Journal of Financial Economics 8, 1 (1980), 31--53.
[2]
Marco Avellaneda and Sasha Stoikov. 2008. High-frequency trading in a limit order book. Quantitative Finance 8, 3 (2008), 217--224. arXiv:https://doi.org/10.1080/14697680701381228
[3]
Daniel S. Bernstein, Shlomo Zilberstein, and Neil Immerman. 2000. The Complexity of Decentralized Control of Markov Decision Processes. In Proceedings of the Sixteenth Conference on Uncertainty in Artificial Intelligence (Stanford, California) (UAI'00). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA, 32--37.
[4]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540 (2016).
[5]
Nicholas Tung Chan and Christian Shelton. 2001. An electronic market-maker. (2001).
[6]
Tung Chan. 2001. Artificial markets and intelligent agents. Ph.D. Dissertation. Massachusetts Institute of Technology.
[7]
R Cont and MS Mueller. 2021. A stochastic PDE model for limit order book dynamics. SIAM Journal on Financial Mathematics (2021).
[8]
Wei Cui and Anthony Brabazon. 2012. An agent-based modeling approach to study price impact. In 2012 IEEE Conference on Computational Intelligence for Financial Engineering Economics (CIFEr). 1--8.
[9]
Sanmay Das. 2003. An agent-based model of dealership markets. In Proceedings of the International Workshop on Complex Agent-based Dynamic Networks, Oxford.
[10]
Sumitra Ganesh, Nelson Vadori, Mengda Xu, Hua Zheng, Prashant Reddy, and Manuela Veloso. 2019. Reinforcement learning for market making in a multi-agent dealer market. arXiv preprint arXiv:1911.05892 (2019).
[11]
Mark B. Garman. 1976. Market microstructure. Journal of Financial Economics 3, 3 (1976), 257--275.
[12]
Olivier Guéant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia. 2013. Dealing with the inventory risk: a solution to the market making problem. Mathematics and financial economics 7, 4 (2013), 477--507.
[13]
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. 2018. RLlib: Abstractions for Distributed Reinforcement Learning. In Proceedings of the 35th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 80), Jennifer Dy and Andreas Krause (Eds.). PMLR, 3053--3062. http://proceedings.mlr.press/v80/liang18b.html
[14]
Thomas Spooner and Rahul Savani. 2020. Robust Market Making via Adversarial Reinforcement Learning. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 4590--4596. Special Track on AI in FinTech.
[15]
Justin K. Terry, Nathaniel Grammel, Benjamin Black, Ananth Hari, Caroline Horsch, and Luis Santos. 2020. Agent Environment Cycle Games. CoRR abs/2009.13051 (2020). arXiv:2009.13051 https://arxiv.org/abs/2009.13051
[16]
Nelson Vadori, Sumitra Ganesh, Prashant Reddy, and Manuela Veloso. 2020. Calibration of Shared Equilibria in General Sum Partially Observable Markov Games. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 14118--14128. https://proceedings.neurips.cc/paper/2020/file/a2f04745390fd6897d09772b2cd1f581-Paper.pdf
[17]
Svitlana Vyetrenko, David Byrd, Nick Petosa, Mahmoud Mahfouz, Danial Dervovic, Manuela Veloso, and Tucker Hybinette Balch. 2019. Get Real: Realism Metrics for Robust Limit Order Book Market Simulations. arXiv:1912.04941 [q-fin.TR]
[18]
Elaine Wah and Michael P. Wellman. 2015. Welfare Effects of Market Making in Continuous Double Auctions. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems (Istanbul, Turkey) (AAMAS '15). International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 57--66.

Cited By

View all
  • (2024)A Financial Market Simulation Environment for Trading Agents Using Deep Reinforcement LearningProceedings of the 5th ACM International Conference on AI in Finance10.1145/3677052.3698639(117-125)Online publication date: 14-Nov-2024
  • (2024)Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650035(1-9)Online publication date: 30-Jun-2024
  • (2024)Dynamic datasets and market environments for financial reinforcement learningMachine Language10.1007/s10994-023-06511-w113:5(2795-2839)Online publication date: 26-Feb-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
ICAIF '21: Proceedings of the Second ACM International Conference on AI in Finance
November 2021
450 pages
ISBN:9781450391481
DOI:10.1145/3490354
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 May 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. market making
  2. multi-agent
  3. reinforcement learning

Qualifiers

  • Research-article

Conference

ICAIF'21
Sponsor:

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)32
  • Downloads (Last 6 weeks)4
Reflects downloads up to 01 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Financial Market Simulation Environment for Trading Agents Using Deep Reinforcement LearningProceedings of the 5th ACM International Conference on AI in Finance10.1145/3677052.3698639(117-125)Online publication date: 14-Nov-2024
  • (2024)Reinforcement Learning in Agent-Based Market Simulation: Unveiling Realistic Stylized Facts and Behavior2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10650035(1-9)Online publication date: 30-Jun-2024
  • (2024)Dynamic datasets and market environments for financial reinforcement learningMachine Language10.1007/s10994-023-06511-w113:5(2795-2839)Online publication date: 26-Feb-2024
  • (2024)Learning Reward Machines in Cooperative Multi-agent TasksAutonomous Agents and Multiagent Systems. Best and Visionary Papers10.1007/978-3-031-56255-6_3(43-59)Online publication date: 30-Mar-2024
  • (2023)Towards multi‐agent reinforcement learning‐driven over‐the‐counter market simulationsMathematical Finance10.1111/mafi.1241634:2(262-347)Online publication date: 20-Sep-2023
  • (2023)Dynamics of market making algorithms in dealer markets: Learning and tacit collusionMathematical Finance10.1111/mafi.1240134:2(467-521)Online publication date: 30-May-2023
  • (2023)CNN- DRL with Shuffled Features in Finance2023 International Conference on Computational Science and Computational Intelligence (CSCI)10.1109/CSCI62032.2023.00055(312-317)Online publication date: 13-Dec-2023
  • (2022)FinRL-metaProceedings of the 36th International Conference on Neural Information Processing Systems10.5555/3600270.3600404(1835-1849)Online publication date: 28-Nov-2022

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media