D3PG: Decomposed Deep Deterministic Policy Gradient for Continuous Control

Abstract

In this paper, we study how structural decomposition and multiagent interactions can be utilized by deep reinforcement learning in order to address high dimensional robotic control problems. In this regard, we propose the D3PG approach, which is a multiagent extension of DDPG by decomposing the global critic into a weighted sum of local critics. Each of these critics is modeled as an individual learning agent that governs the decision making of a particular joint of a robot. We then propose a method to learn the weights during learning in order to capture different levels of dependencies among the agents. The experimental evaluation demonstrates that D3PG can achieve competitive or significantly improved performance compared to some widely used deep reinforcement learning algorithms. Another advantage of D3PG is that it is able to provide explicit interpretations of the final learned policy as well as the underlying dependencies among the joints of a learning robot.

Supported by National Natural Science Foundation of China under Grant No. 62076259.

A Appendix

1.1 A.1 Appendix

1.2 A.2 MuJoCo Platform

Below we provide some specifications for the states, actions and rewards of the four robot environments in MuJoCo.

Swimmer. The swimmer is a planar robot with 3 links and 2 actuated joints. Fluid is simulated through viscosity forces, which apply drag on each link, allowing the swimmer to move forward. The 8-dim observation includes the joint angles and velocities, and the coordinates of the center of mass. The reward is given by $r(s, a) = v_x-0.005\cdot$ $\parallel$ $a$ $\parallel$ $_2^2$, where $v_x$ is the forward velocity. No termination condition is applied.

Hopper. The hopper is a planar robot with 4 rigid links, corresponding to the torso, upper leg, lower leg, and foot, along with 3 actuated joints. The 11-dim observation includes joint angles, joint velocities, the coordinates of the center of mass, and the constraint forces. The reward is given by $r(s, a) = v_x-0.005\cdot$ $\parallel$ $a$ $\parallel$ $_2^2 + 1$, where the last term is a bonus for being “alive”. The episode is terminated when $z_{body}<$ 0.7, where $z_{body}$ is the z-coordinate of the body, or when $|\theta _y|<$ 0.2, where $\theta _y$ is the forward pitch of the body.

Walker. The walker is a planar biped robot consisting of 7 links, corresponding to two legs and a torso, along with 6 actuated joints. The 17-dim observation includes joint angles, joint velocities, and the coordinates of center of mass. The reward is given by $r(s, a) = v_x-0.005\cdot$ $\parallel$ $a$ $\parallel$ $_2^2$. The episode is terminated when $z_{body}<$ 0.8, or $z_{body}>$ 2.0, or $|\theta _y|>$ 1.0.

Half-Cheetah. The half-cheetah is a planar biped robot with 9 rigid links, including two legs and a torso, along with 6 actuated joints. The 17-dim state includes joint angles, joint velocities, and the coordinates of the center of mass. The reward $r(s, a) = v_x-0.005\cdot$ $\parallel$ $a$ $\parallel$ $_2^2$. No termination condition is applied.