Student-t policy in reinforcement learning to acquire global optimum of robot control

Abstract

This paper proposes an actor-critic algorithm with a policy parameterized by student-t distribution, named student-t policy, to enhance learning performance, mainly in terms of reachability on global optimum for tasks to be learned. The actor-critic algorithm is one of the policy-gradient methods in reinforcement learning, and is proved to learn the policy converging on one of the local optima. To avoid the local optima, an exploration ability to escape it and a conservative learning not to be trapped in it are deemed to be empirically effective. The conventional policy parameterized by a normal distribution, however, fundamentally lacks these abilities. The state-of-the-art methods can somewhat but not perfectly compensate for them. Conversely, heavy-tailed distribution, including student-t distribution, possesses an excellent exploration ability, which is called Lévy flight for modeling efficient feed detection of animals. Another property of the heavy tail is its robustness to outliers. Namely, conservative learning is performed to not be trapped in the local optima even when it takes extreme actions. These desired properties of the student-t policy enhance the possibility of the agents reaching the global optimum. Indeed, the student-t policy outperforms the conventional policy in four types of simulations, two of which are difficult to learn faster without sufficient exploration and the others have the local optima.

Appendices

Appendices

A Details of the update rule

2.1 A.1 The gradient of PPO

The state-of-the-art method PPO [31] aims to maximize the following expected value by optimizing the policy parameter w_A.

$$ \begin{array}{@{}rcl@{}} &&\max_{\boldsymbol{w}_{A}} \mathbb{E}_{t} \left[ \hat r_{t}\hat A_{t} - \beta_{1} D_{KL}(\pi(\cdot \mid s_{t}, \boldsymbol{w}_{A,t-1}) \mid \pi(\cdot \mid s_{t}, \boldsymbol{w}_{A,t}))\right.\\ &&+ \left. \beta_{2} H(\pi(\cdot \mid s_{t}, \boldsymbol{w}_{A,t})) \right] \end{array} $$

(32)

where $\hat r_{t} = \frac {\pi (a_{t} \mid s_{t}, \boldsymbol {w}_{A,t})}{\pi (a_{t} \mid s_{t}, \boldsymbol {w}_{A,t-1})}$ represents the importance sampling and $\hat A_{t}$ is the advantage function approximated by GAE [30]. The gradient of this term corresponds to the term excluding PPO(⋅) from (7).

D_KL(⋅) and H(⋅) indicate the KL divergence and differential entropy with β_1,2 coefficients, respectively. Therefore, in (7), PPO(⋅) corresponds to the gradients of D_KL(⋅) and H(⋅). While the policy update is suppressed by the KL penalty, the entropy bonus encourages exploration. In addition, β₁ is adjusted by a simple heuristic approach in the original paper. The approach of adjusting β₁ is modified as follows:

$$ \beta_{1} \leftarrow \beta_{1} \exp\left( \frac{\bar D_{KL}^{\text{new}} - \bar D_{KL}^{\text{old}}}{\bar D_{KL}^{\text{new}} + \bar D_{KL}^{\text{old}}} \right) $$

(33)

where $\bar D_{KL}^{\mathrm {old,new}}$ are the moving averages before and after the update by the current D_KL(⋅).

Note that all D_KL(⋅) and H(⋅), except D_KL(⋅) for the student-t policy, can be analytically derived. In addition, their gradients are also analytically calculated. D_KL(⋅) for the student-t policy, however, needs to be approximated as a closed-form solution depending on ref. [11] to derive its gradient analytically.

Given $p_{1} \sim \mathcal {T}(\boldsymbol {\mu }_{1} , {\Sigma }_{1} , \nu _{1})$ and $p_{2} \sim \mathcal {T}(\boldsymbol {\mu }_{2} , {\Sigma }_{2} , \nu _{2})$, the KL divergence between them is approximated as follows:

$$ \begin{array}{@{}rcl@{}} D_{KL}(p_{1} \mid p_{2}) &\simeq& \frac{1}{2}\ln\frac{|{\Sigma}_{2}|}{|{\Sigma}_{1}|} + \frac{1}{2}\frac{\nu_{2}+d}{\nu_{2}}\frac{\nu_{1}}{\nu_{1}-2}\text{tr}({\Sigma}_{2}^{-1}{\Sigma}_{1}) \\ &&+ \frac{1}{2}\frac{\nu_{2}+d}{\nu_{2}}(\boldsymbol{\mu}_{1} - \boldsymbol{\mu}_{2})^{\top}{\Sigma}_{2}^{-1}(\boldsymbol{\mu}_{1} - \boldsymbol{\mu}_{2})\\ &&- \frac{\nu_{1}+d}{2}\left\{ \psi\left( \frac{\nu_{1}+d}{2}\right) - \psi\left( \frac{\nu_{1}}{2}\right) \right\} \end{array} $$

(34)

Note that ν₁ should be larger than 2, although the student-t policy can have ν smaller than 2. In this paper, its gradient is calculated by adding 2 − ν₀ as an offset to ν.

2.2 A.2 True online GAE

The original TD(λ) [34] recursively updates the weights for the value function w_C by using the eligibility trace e.

$$ \begin{array}{@{}rcl@{}} \boldsymbol{e}_{C,t} &=& \gamma \lambda \boldsymbol{e}_{C,t-1} + \boldsymbol{x}(s_{t}) \end{array} $$

(35)

$$ \begin{array}{@{}rcl@{}} \boldsymbol{g}_{C,t} &=& \delta_{t} \boldsymbol{e}_{C,t} \end{array} $$

(36)

$$ \begin{array}{@{}rcl@{}} \boldsymbol{w}_{C,t+1} &=& \boldsymbol{w}_{C,t} + \alpha \boldsymbol{g}_{C,t} \end{array} $$

(37)

This TD(λ) includes an approximation, and therefore, the true online TD(λ) [38] has derived the exact recursive update rule (see (2)–(4)).

GAE approximates the advantage function as $\hat A$ by accumulating the TD error δ.

$$ \hat A_{t} = \sum\limits_{k=0}^{\infty} (\gamma \lambda)^{k} \delta_{t+k} $$

(38)

By using this GAE [30] and REINFORCE [39], the offline policy gradient g_A is given as follows:

$$ \begin{array}{@{}rcl@{}} \boldsymbol{g}_{A} &=& \sum\limits_{t=0}^{\infty} \hat A_{t} \nabla_{\boldsymbol{w}_{A}}\ln\pi(a_{t} \mid s_{t}, \boldsymbol{w}_{A}) \end{array} $$

(39)

$$ \begin{array}{@{}rcl@{}} &=& \sum\limits_{t=0}^{\infty} \boldsymbol{\hat x}(s_{t},a_{t}) \sum\limits_{k=0}^{\infty} (\gamma \lambda)^{k} \delta_{t+k} \end{array} $$

(40)

$$ \begin{array}{@{}rcl@{}} &=& \sum\limits_{t=0}^{\infty} \delta_{t} \sum\limits_{k=0}^{t} (\gamma \lambda)^{k} \boldsymbol{\hat x}(s_{t_{k}},a_{t-k}) \end{array} $$

(41)

where $\boldsymbol {\hat x}$ is defined in (5). Here, $\boldsymbol {e}_{A,t} = {\sum }_{k=0}^{t}(\gamma$$\lambda )^{k} \boldsymbol {\hat x}(s_{t_{k}},a_{t-k})$ is defined, and consequently, this policy gradient can be recursively approximated by the following equations.

$$ \begin{array}{@{}rcl@{}} \boldsymbol{e}_{A,t} &=& \gamma \lambda \boldsymbol{e}_{A,t-1} + \boldsymbol{\hat x}(s_{t}, a_{t}) \end{array} $$

(42)

$$ \begin{array}{@{}rcl@{}} \boldsymbol{g}_{A,t} &=& \delta_{t} \boldsymbol{e}{A,t} \end{array} $$

(43)

$$ \begin{array}{@{}rcl@{}} \boldsymbol{w}_{A,t+1} &=& \boldsymbol{w}_{A,t} + \alpha \boldsymbol{g}_{A,t} \end{array} $$

(44)

As can be seen from the above equations, the update rule of the actor by the recursive GAE is equivalent to the update rule of the critic by the original TD(λ). Hence, the true online version of GAE can be derived as shown in (5)–(8).

B Task specifications

All the simulation environments have been created in V-REP [29] and uploaded in Github: https://github.com/kbys-t/gym_vrepto reproduce the results. Here, let us introduce their designs of state, action, reward, and end conditions.

3.1 B.1 (a) The rolling balance task

An agent, a half-sized humanoid robot NAO, tries to balance on a board on a cylinder. The agent stands at the center of the board horizontal to the ground at the start of an episode. State space is defined by three states: roll angle of the agent; roll angle of the board, and angular velocity of the roll angle of the board. Action space is defined by one action: an angular velocity of the roll angle of the agent. If the board comes in contact with the ground, the episode is terminated as a failure. In the case of the failure, the agent receives a big penalty of − 100. At every step of the episode, the agent gets a reward as the board and ground are almost parallel (the maximum is 1). In this task, local exploration is enough to reach the global optimum, although a highly efficient exploration would be able to maintain the balance faster.

3.2 B.2 (b) The task to navigate ballbot

An agent, an omnidirectional mobile robot on a ball, tries to reach a goal on (1.0, 1.5) while avoiding a set of dining table and chairs. The agent is stopped on (− 1.5, 1.5) at the start of an episode. State space is defined by four states: two-dimensional (2D) positions of the agent and 2D differences between the agent and a reference point. Action space is defined by two actions: 2D velocities of the reference point. If the agent falls over, the episode will be terminated as a failure with penalty − 1. At every step of the episode, the agent gets a reward according to the distance between the agent and the goal (the maximum is 1). The agent cannot recognize the set of dining table and chairs, which hinders the shortest path to the goal; hence, it is required to find a detour route to the goal only by sufficient exploration.

3.3 B.3 (c) The task to open door

An agent, a manipulator with seven degrees of freedom in front of a door, tries to open the door after turning its knob. The agent’s hand is on the knob at the start of an episode. State space is defined by five states: 3D positions of the agent’s hand, the angle of the knob, and the angle of the door. Action space is defined by three actions: 3D velocities of the agent’s hand. If the agent perfectly opens the door, the episode will be terminated as success without bonus. At every step of the episode, the agent gets a small reward according to the angle of the knob (the maximum is 0.05) and a large reward according to the angle of the door (the maximum is 1). Since the agent simultaneously gets two rewards, this task is prone to the local optima. For example, the agent turns the knob but does not push the door because when pushing the door, the agent’s hand is likely to get away from the knob.

3.4 B.4 (d) The task to reach targets

An agent, two planar manipulators, one of which is installed on the first link of another, tries to make their tips reach the respective targets, i.e., (− 0.5, 0) and (0.5, 0), simultaneously. State space is defined by four states: the angles of respective joints. Action space is defined by four actions: the angular velocities of the respective joints. If both tips reach within a radius of 0.1 m of the respective targets simultaneously, the agent gets a large bonus of 100 and the episode is terminated as a success. At every step of the episode, the agent cannot get any reward; namely, the reward is obtained only when the task succeeds. Such sparse reward makes it difficult to resolve the task by reinforcement learning with poor exploration ability.