Abstract
In the Portfolio Management problem the agent has to decide how to allocate the resources among a set of stocks in order to maximize his gains. This decision-making problem is modeled by some researchers through Markov decision processes (MDPs) and the most widely used criterion in MDPs is maximizing the expected total reward. However, this criterion does not take risk into account. To deal with risky issues, risk sensitive Markov decision processes (RSMDPs) are used. To the best of our knowledge, RSMDPs and more specifically RSMDPs with exponential utility function have never been applied to handle this problem. In this paper we introduce a strategy to model the Portfolio Management problem focused on day trade operations in order to enable the use of dynamic programming. We also introduce a measure based on Conditional Value-at-Risk (CVaR) to evaluate the risk attitude. The experiments show that, with our model and with the use of RSMDPs with exponential utility function, it is possible to change and interpret the agent risk attitude in a very understandable way.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bookstaber, D.: Using Markov decision processes to solve a portfolio allocation problem. Undergraduate Thesis, Brown University (2005)
Brown, D.B., Smith, J.E.: Dynamic portfolio optimization with transaction costs: heuristics and dual bounds. Manage. Sci. 57(10), 1752–1770 (2011)
Chung, K.J., Sobel, M.J.: Discounted MDP’s: distribution functions and exponential utility maximization. SIAM J. Control Optim. 25(1), 49–62 (1987)
Filos, A.: Reinforcement learning for portfolio management. arXiv preprint arXiv:1909.09571 (2019)
Gupta, A., Dhingra, B.: Stock market prediction using hidden Markov models. In: 2012 Students Conference on Engineering and Systems, pp. 1–4. IEEE (2012)
Heger, M.: Consideration of risk in reinforcement learning. In: Machine Learning Proceedings 1994, pp. 105–111. Elsevier (1994)
Howard, R.A., Matheson, J.E.: Risk-sensitive Markov decision processes. Manage. Sci. 18(7), 356–369 (1972)
Littman, M.L., Szepesvári, C.: A generalized reinforcement-learning model: convergence and applications. In: ICML, vol. 96, pp. 310–318 (1996)
Lynch, A.W.: Portfolio choice and equity characteristics: characterizing the hedging demands induced by return predictability. J. Financ. Econ. 62(1), 67–130 (2001)
Mihatsch, O., Neuneier, R.: Risk-sensitive reinforcement learning. Mach. Learn. 49(2–3), 267–290 (2002)
Patek, S.D.: On terminating Markov decision processes with a risk-averse objective function. Automatica 37(9), 1379–1386 (2001)
Petrik, M., Subramanian, D.: An approximate solution method for large risk-averse Markov decision processes. In: Proceedings of the Twenty-Eighth Conference on Uncertainty in Artificial Intelligence, Arlington, Virginia, USA, pp. 805–814 (2012)
Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st edn. Wiley, New York (1994)
Uryasev, S., Rockafellar, R.T.: Conditional value-at-risk: optimization approach. In: Uryasev, S., Pardalos, P.M. (eds.) Stochastic Optimization: Algorithms and Applications. Applied Optimization, vol. 54, pp. 411–435. Springer, Boston (2001). https://doi.org/10.1007/978-1-4757-6594-6_17
Acknowledgment
Supported by grant #2018/11236-9, São Paulo Research Foundation (FAPESP).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Neto, E.L.P., Freire, V., Delgado, K.V. (2020). Risk Sensitive Markov Decision Process for Portfolio Management. In: Martínez-Villaseñor, L., Herrera-Alcántara, O., Ponce, H., Castro-Espinoza, F.A. (eds) Advances in Soft Computing. MICAI 2020. Lecture Notes in Computer Science(), vol 12468. Springer, Cham. https://doi.org/10.1007/978-3-030-60884-2_27
Download citation
DOI: https://doi.org/10.1007/978-3-030-60884-2_27
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60883-5
Online ISBN: 978-3-030-60884-2
eBook Packages: Computer ScienceComputer Science (R0)