SCC-rFMQ: a multiagent reinforcement learning method in cooperative Markov games with continuous actions

Abstract

Although many multiagent reinforcement learning (MARL) methods have been proposed for learning the optimal solutions in continuous-action domains, multiagent cooperation domains with independent learners (ILs) have received relatively few investigations, especially in traditional RL domain. In this paper, we propose an sample based independent learning method, named Sample Continuous Coordination with recursive Frequency Maximum Q-Value (SCC-rFMQ), which divides the multiagent cooperative problem with continuous actions into two layers. The first layer samples a finite set of actions from the continuous action spaces by a re-sampling mechanism with variable exploratory rates, and the second layer evaluates the actions in the sampled action set and updates the policy using a reinforcement learning cooperative method. By constructing cooperative mechanisms at both levels, SCC-rFMQ can handle cooperative problems in continuous action cooperative Markov games effectively. The effectiveness of SCC-rFMQ is experimentally demonstrated on two well-designed games, i.e., a continuous version of the climbing game and a cooperative version of the boat problem. Experimental results show that SCC-rFMQ outperforms other reinforcement learning algorithms.

Appendices

Bilinear interpolation

Using the bilinear interpolation techniques [31], we construct continuous game models extended by the CG (PSCG) game. Bilinear interpolation is an extension of linear interpolation for interpolating functions of two variables on a rectilinear 2D grid. The key idea is to perform linear interpolation first in one direction, and then again in the other direction (see Fig. 10).

Fig. 10

The four red dots show the data points and the green dot is the point at which we want to interpolate

Suppose that we want to find the value of the unknown function f at the point (x, y). It is assumed that we know the value of f at the four points $Q_{11}=(x_1, y_1)$, $Q_{12}=(x_1,y_2)$, $Q_{21}=(x_2,y_1)$, and $Q_{22}=(x_2,y_2)$. We first do linear interpolation in the x-direction:

$$\begin{aligned}\begin{array}{l} f\left( x,{y_1}\right) = \frac{{{x_2} - x}}{{{x_2} - {x_1}}}f\left( {Q_{11}}\right) + \frac{{x - {x_1}}}{{{x_2} - {x_1}}}f\left( {Q_{21}}\right) \\ f\left( x,{y_2}\right) = \frac{{{x_2} - x}}{{{x_2} - {x_1}}}f\left( {Q_{12}}\right) + \frac{{x - {x_1}}}{{{x_2} - {x_1}}}f\left( {Q_{22}}\right) \end{array}\end{aligned}$$

then proceed by interpolating in the y-direction to obtain the desired estimate:

$$\begin{aligned}f(x,y) = \frac{{{y_2} - y}}{{{y_2} - {y_1}}}f\left( x,{y_1}\right) + \frac{{y - {y_1}}}{{{y_2} - {y_1}}}f\left( x,{y_2}\right) \end{aligned}$$

Parameter setting

Table 1 where parameters used for the results presented in this section. With the detailed algorithmic descriptions in SCC-rFMQ and these parameters detail, all of the presented results are reproducible. For MAPPO, we used the official code and parameters directly (in GitHub²). For others, we select the values with the best performance after extensive simulations. In Table  1, $\epsilon$ and $\epsilon _i^{re}(s)$ are strategy valuables for SCC-rFMQ and rFMQ, which are defined as maps that decrease with increasing learning trials t. $A(0)_n$ defined in SCC-rFMQ, SMC and rFMQ are initial distributed action sets evenly sampled from action space [0, 1], where n is the sampling set size. For SMC and rFMQ, the common parameters (e.g., $\alpha _Q$ and $\gamma$) are set to be the same as SCC-rFMQ for a fair comparison. Parameter definition of $\sigma$ and $\tau$ are the same as [12], and $\lambda$ and $\sigma _L$ are the same as [8]. Parameter definitions of MADDPG and DDPG are the same as [18], where policies and critics are parameterized by a two-layer ReLU MLP followed by a fully connected layer (activated by tanh function for policy nets). Parameter definitions of L-DDQN and H-DDQN are the same as [25]. It should be noted that parameter changes do not significantly affect the conclusion of the experiment in our experiments. For SMC+rFMQ, we use rFMQ with $\epsilon$-greedy strategy to learn Q value, and use the resampling strategy of SMC to update action set every $c=200$ episodes. Weight value $w_{i}^{t+1}(s,a)$ used in the SMC resampling strategy is calculated by the Boltzmann exploration strategy, where $\varDelta {\mathrm{Q}}_i^{t + 1}(s,a) = {\mathrm{Q}}_i^{t + 1}(s,a) - {\mathrm{Q}}_i^t(s,a)$.

Table 1 Parameter setting