Skip to main content
SpringerLink
Log in

Regime-Switching Recurrent Reinforcement Learning in Automated Trading

  • Chapter
Natural Computing in Computational Finance

Part of the book series: Studies in Computational Intelligence ((SCI,volume 380))

Summary

The regime-switching recurrent reinforcement learning (RSRRL) model was first presented in [19], in the form of a GARCH-based threshold version that extended the standard RRL algorithm developed by [22]. In this study, the main aim is to investigate the influence of different transition variables, in multiple RSRRL settings and for various datasets, and compare and contrast the performance levels of the RRL and RSRRL systems in algorithmic trading experiments. The transition variables considered are GARCH-based volatility, detrended volume, and the rate of information arrival, the latter being modelled on the Mixture Distribution Hypothesis (MDH). A frictionless setting was assumed for all the experiments. The results showed that the RSRRL models yield higher Sharpe ratios than the standard RRL in-sample, but struggle to reproduce the same performance levels out-of-sample. We argue that the lack of in- and out-of-sample correlation is due to a drastic change in market conditions, and find that the RSRRL can consistently outperform the RRL only when certain conditions are present. We also find that trading volume presents a lot of promise as an indicator, and could be the way forward for the design of more sophisticated RSRRL systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Andersen, T.G.: Return volatility and trading volume: An information flow interpretation of stochastic volatility. Journal of Finance 51(1), 169–204 (1996)

    Article  Google Scholar 

  2. Bertoluzzo, F., Corazza, M.: Making financial trading by recurrent reinforcement learning. In: Apolloni, B., Howlett, R.J., Jain, L. (eds.) KES 2007, Part II. LNCS (LNAI), vol. 4693, pp. 619–626. Springer, Heidelberg (2007), http://dx.doi.org/10.1007/978-3-540-74827-4_78

    Chapter  Google Scholar 

  3. Danielsson, J., Richard, J.F.: Accelerated gaussian importance sampler with application to dynamic latent variable models. Journal of Applied Econometrics 8, 153–173 (1993)

    Article  Google Scholar 

  4. DeJong, D.N., Dharmarajan, H., Liesenfeld, R., Richard, J.F.: An efficient filtering approach to likelihood approximation for state-space representations. Economics Working Papers, 25. Christian-Albrechts-University of Kiel, Department of Economics (2007), http://ideas.repec.org/p/zbw/cauewp/6339.html

  5. Dempster, M.A.H., Leemans, V.: An automated fx trading system using adaptive reinforcement learning. Expert Systems with Applications 30(3), 543–552 (2006); Intelligent Information Systems for Financial Engineering

    Article  Google Scholar 

  6. Doucet, A., Johansen, A.M.: A tutorial on particle filtering and smoothing: Fifteen years later. The Oxford Handbook of Nonlinear Filtering (2009)

    Google Scholar 

  7. Epps, T.W.: Security price changes and transaction volumes: Theory and evidence. The American Economic Review 65(4), 586–597 (1975)

    Google Scholar 

  8. Epps, T.W.: Security price changes and transaction volumes: Some additional evidence. Journal of Financial and Quantitative Analysis 12(1), 141–146 (1977)

    Article  Google Scholar 

  9. Fleming, J., Kirby, C., Ostdiek, B.: Stochastic volatility, trading volume, and the daily flow of information. Journal of Business 79(3), 1551–1590 (2006)

    Article  MathSciNet  Google Scholar 

  10. Franses, P.H., van Dijk, D.: Nonlinear time series models in empirical finance. Cambridge University Press, Cambridge (2000)

    Book  Google Scholar 

  11. Gold, C.: FX trading via recurrent reinforcement learning. In: Proceedings of IEEE International Conference on Computational Intelligence for Financial Engineering, pp. 363–370 (2003), doi:10.1109/CIFER.2003.1196283

    Google Scholar 

  12. Hamilton, J.D.: A new approach to the economic analysis of nonstationary time series and the business cycle. Econometrica 57(2), 357–384 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  13. Kaelbling, L.P., Littman, M.L., Moore, A.W.: Reinforcement learning: A survey. Journal of Artificial Intelligence 4(1), 237–285 (1996)

    Google Scholar 

  14. Karpoff, J.M.: The relation between price changes and trading volume: A survey. The Journal of Financial and Quantitative Analysis 22(1), 109–126 (1987)

    Article  Google Scholar 

  15. Koutmos, G.: Feedback trading and the autocorrelation pattern of stock returns: Further empirical evidence. Journal of International Money and Finance 16(4), 625–636 (1997), doi:10.1016/S0261-5606(97)00021-1

    Article  Google Scholar 

  16. Lamoureux, C.G., Lastrapes, W.D.: Endogenous trading volume and momentum in stock-return volatility. Journal of Business and Economic Statistics 12(2), 253–260 (1994)

    Article  Google Scholar 

  17. LeBaron, B.: Some relations between volatility and serial correlations in stock market returns. Journal of Business 65(2), 199–219 (1992)

    Article  Google Scholar 

  18. Liesenfeld, R.: A generalized bivariate mixture model for stock price volatility and trading volume. Journal of Econometrics 104(1), 141–178 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  19. Maringer, D. G., Ramtohul, T.: Threshold recurrent reinforcement learning model for automated trading. In: Di Chio, C., Brabazon, A., Di Caro, G.A., Ebner, M., Farooq, M., Fink, A., Grahl, J., Greenfield, G., Machado, P., O’Neill, M., Tarantino, E., Urquhart, N. (eds.) EvoApplications 2010. LNCS, vol. 6025, pp. 212–221. Springer, Heidelberg (2010)

    Chapter  Google Scholar 

  20. McKenzie, M.D., Faff, R.W.: The determinants of conditional autocorrelation in stock returns. Journal of Financial Research 26(2), 259–274 (2003)

    Article  Google Scholar 

  21. Moody, J., Saffell, M.: Learning to trade via direct reinforcement. IEEE Transactions on Neural Networks 12(4), 875–889 (2001), doi:10.1109/72.935097

    Article  Google Scholar 

  22. Moody, J., Wu, L.: Optimization of trading systems and portfolios. In: Proceedings of the IEEE/IAFE 1997, Computational Intelligence for Financial Engineering (CIFEr), pp. 300–307 (1997), doi:10.1109/CIFER.1997.618952

    Google Scholar 

  23. Moody, J., Wu, L., Liao, Y., Saffell, M.: Performance functions and reinforcement learning for trading systems and portfolios. Journal of Forecasting 17(56), 441–470 (1998)

    Article  Google Scholar 

  24. Richard, J.F.: Effcient high-dimensional monte carlo importance sampling. Tech. rep., University of Pittsburgh (1998)

    Google Scholar 

  25. Richard, J.F., Zhang, W.: Efficient high-dimensional importance sampling. Working Papers 321. University of Pittsburgh, Department of Economics (2007), http://ideas.repec.org/p/pit/wpaper/321.html

  26. Sentana, E., Wadhwani, S.B.: Feedback traders and stock return autocorrelations: Evidence from a century of daily data. Economic Journal 102(411), 415–425 (1992)

    Article  Google Scholar 

  27. Sharpe, W.F.: Mutual fund performance. Journal of Business 39, 119–138 (1966)

    Article  Google Scholar 

  28. Sun, W.: Relationship between trading volume and security prices and returns. Tech. rep., Technical Report (2003)

    Google Scholar 

  29. Sutton, R.S., Barto, A.G.: Introduction to Reinforcement Learning. MIT Press, Cambridge (1998)

    Google Scholar 

  30. Tauchen, G.E., Pitts, M.: The price variability-volume relationship on speculative markets. Econometrica 51(2), 485–505 (1983)

    Article  MATH  Google Scholar 

  31. Teräsvirta, T.: Specification, estimation, and evaluation of smooth transition autoregressive models. Journal of the American Statistical Association 89(425), 208–218 (1994)

    Article  Google Scholar 

  32. Tong, H.: On a threshold model. In: Chen, C. (ed.) Pattern Recognition and Signal Processing, pp. 101–141. Sijthoff & Noordhoff (1978)

    Google Scholar 

  33. Werbos, P.J.: Backpropagation through time: What it does and how to do it. Proceedings of the IEEE 78(10), 1550–1560 (1990)

    Article  Google Scholar 

  34. White, H.: Some asymptotic results for learning in single hidden-layer feedforward network models. Journal of the American Statistical Association 84(408), 1003–1013 (1989)

    Article  MathSciNet  MATH  Google Scholar 

  35. Wood, R.A., McInish, T.H., Ord, J.K.: An investigation of transactions data for NYSE stocks. The Journal of Finance 40(3), 723–739 (1985)

    Article  Google Scholar 

  36. Ying, C.C.: Stock market prices and volumes of sales. Econometrica 34(3), 676–685 (1966)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Universität Basel, CH-4002, Basel, Switzerland

    Dietmar Maringer & Tikesh Ramtohul

Authors
  1. Dietmar Maringer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tikesh Ramtohul
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Quinn School of Business, University College Dublin, Belfield, 4, Dublin, Ireland

    Anthony Brabazon

  2. UCD Complex Adaptive Systems Laboratory, University College Dublin, Belfield, 4, Dublin, Ireland

    Michael O’Neill

  3. Büro 5.56 WirtschaftswissenschaftlichesZentrum (WWZ) Abteilung Quantitative Methoden, University of Basel, Peter Merian-Weg 6, 4002, Basel, Switzerland

    Dietmar Maringer

Rights and permissions

Reprints and permissions

Copyright information

© 2011 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Maringer, D., Ramtohul, T. (2011). Regime-Switching Recurrent Reinforcement Learning in Automated Trading. In: Brabazon, A., O’Neill, M., Maringer, D. (eds) Natural Computing in Computational Finance. Studies in Computational Intelligence, vol 380. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-23336-4_6

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-23336-4_6

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-23335-7

  • Online ISBN: 978-3-642-23336-4

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation