From deterministic to stochastic: an interpretable stochastic model-free reinforcement learning framework for portfolio optimization

Abstract

As a fundamental problem in algorithmic trading, portfolio optimization aims to maximize the cumulative return by continuously investing in various financial derivatives within a given time period. Recent years have witnessed the transformation from traditional machine learning trading algorithms to reinforcement learning algorithms due to their superior nature of sequential decision making. However, the exponential growth of the imperfect and noisy financial data that is supposedly leveraged by the deterministic strategy in reinforcement learning, makes it increasingly challenging for one to continuously obtain a profitable portfolio. Thus, in this work, we first reconstruct several deterministic and stochastic reinforcement algorithms as benchmarks. On this basis, we introduce a risk-aware reward function to balance the risk and return. Importantly, we propose a novel interpretable stochastic reinforcement learning framework which tailors a stochastic policy parameterized by Gaussian Mixtures and a distributional critic realized by quantiles for the problem of portfolio optimization. In our experiment, the proposed algorithm demonstrates its superior performance on U.S. market stocks with a 63.1% annual rate of return while at the same time reducing the market value max drawdown by 10% when back-testing during the stock market crash around March 2020.

Appendices

Appendix A:: Mathematical details

1.1 A.1 Computing the determinant of the jacobin matrix

For $f: \mathbb {R}^{h} \to \mathbb {R}^{h}$ from the main manuscript (3), where h is the dimension of actions, we let a = f(x), and the Jacobin of this function is:

$$ J_{f}(x,\tau) = \frac{1}{\tau} \begin{pmatrix} a_{1}-{a_{1}^{2}} & -a_{1}a_{2} &{\cdots} & -a_{1}a_{h} \\ -a_{2}a_{1} & a_{2}-{a_{2}^{2}} &{\cdots} & -a_{2}a_{h} \\ {\vdots} & {\vdots} &{\ddots} & {\vdots} \\ -a_{h}a_{1} & -a_{h}z_{2} & {\cdots} & a_{h}-{a_{h}^{2}} \end{pmatrix}, $$

if we define v = (a₁,a₂,⋯ ,a_h)^T and D = diag(a), then we have:

$$ \begin{array}{@{}rcl@{}} \det(J_{f}(x,\tau)) &=& \det (\frac{1}{\tau}(D-vv^{T})) \\ &=&(\frac{1}{\tau})^{h} \cdot (1-v^{T}D^{-1}v) \cdot \det D \quad\quad \text{by the Matrix Determinant Lemma } \\ &=&(\frac{1}{\tau})^{h} \cdot (1-\sum\limits_{i=1}^{h}a_{i}) \cdot \prod\limits_{i=1}^{h}a_{i} \quad\quad\quad\quad \text{by the property of matrix } D \\ &=&(\frac{1}{\tau})^{h} \cdot \delta \cdot \prod\limits_{i=1}^{h}a_{i}\quad\quad\quad\quad\quad \qquad \text{by} \sum\limits_{i=1}^{h}a_{i} \approx 1 \end{array} $$

1.2 A.2 Computing the lower bound of the log probability

According to the formula of the transformation of random variables, we have $p_{\mathcal {A}}(a) = p_{\mathcal {A}^{\prime }}(a^{\prime })\vert \det J_{f}(a^{\prime },\tau )\vert ^{-1}$, if we let $p_{\mathcal {A}}(a):=\pi _{\theta }(a_{t}\vert s_{t})$, then its log-likelihood can be written as:

$$ \begin{array}{@{}rcl@{}} \log\pi_{\theta}(a_{t}\vert s_{t}) &=& \log p_{\mathcal{A}^{\prime}}(a_{t}^{\prime})-\log \vert \det J_{f}(a_{t}^{\prime},\tau)\vert \\ &=& \log p_{\mathcal{A}^{\prime}}(a_{t}^{\prime})+h\log(\tau)-\log(1-{\sum\limits_{i}^{h}}{a_{t}^{i}})-{\sum\limits_{i}^{h}}\log({a_{t}^{i}}) \\ &=& \log p_{\mathcal{A}^{\prime}}(a_{t}^{\prime})+h\log(\tau)-\log(\delta)-{\sum\limits_{i}^{h}}\log({a_{t}^{i}}) \\ &\gg& \log p_{\mathcal{A}^{\prime}}(a_{t}^{\prime})+h\log(\tau)-{\sum\limits_{i}^{h}}\log({a_{t}^{i}}) \end{array} $$

Therefore, the lower bound of the transformed log likelihood on a simplex region is $\log p_{\mathcal {A}^{\prime }}(a_{t}^{\prime })+h\log (\tau )-{{\sum }_{i}^{h}}\log ({a_{t}^{i}})$.

Appendix B: Supplementary tables

Table 4 Abbreviations and Full names of the used 22 stocks in the U.S. market

Table 5 Abbreviations and explanations of the used nine features