Normalizing Flow Policies for Multi-agent Systems

Abstract

Stochastic policy gradient methods using neural representations have had considerable success in single-agent domains with continuous action spaces. These methods typically use networks that output the parameters of a diagonal Gaussian distribution from which the resulting action is sampled. In multi-agent contexts, however, better policies may require complex multimodal action distributions. Based on recent progress in density modeling, we propose an alternative for policy representation in the form of conditional normalizing flows. This approach allows for greater flexibility in action distribution representation beyond mixture models. We demonstrate their advantage over standard methods on a set of tasks including human behavior modeling and reinforcement learning in multi-agent settings.

6 Appendix

Policy Implementation. Our implementation is based on the Garage [10] reinforcement learning library. We use a multi-layer perceptron (MLP) consisting of 3 hidden layers with 64, 64, and 32 hidden units for the Gaussian, Cholesky Gaussian, GMM and MCG policies. The mean and covariance (and weights for mixture models) use the same MLP except the last layer for better knowledge sharing. The NFP1 policy uses the standard RealNVP structure with 5 coupling layers. An additional state conditioning layer is added after the first coupling layer. The state conditioning layer uses an MLP of 2 hidden layers with 64 hidden units [47]. For the proposed conditional flow policy, we use 5 coupling layers and each coupling layer has an MLP of 2 hidden layers with 32 hidden units. The MLP takes the concatenation of the observation and half of the latent variables ($x_{1:d},d=\lfloor D/2 \rfloor$) as the input, and output the scale and translation factors $\alpha$ and t as introduced in Sect. 3.1. The output $\alpha$ is then clipped between $[-5,5]$ for better numerical stability. We additionally add a tanh on the final outputs for all policies similar to Haarnoja et al. [14], which helps limit the policy output space as well as bound the entropy term in loss. We make sure that the total number of parameters for different models stay close to $10^4$ for a fair comparison. All the hidden layers use ReLU activations. For the farm security game, since the inputs are images, we additionally add a convolutional neural network (CNN) of 2 convolution layers as the feature extractor to all models. The convolution layers have 32 and 16 channels. The filter sizes are $16\times 16$ and $4 \times 4$, and the strides are $8 \times 8$ and $2 \times 2$. This CNN structure is suggested by Kamra et al. [20].

Agent Modeling. We use behavior cloning as our training algorithm in Sect. 4.1 which maximizes the likelihood of actions in the training data [36]. We use a batch size of 1024. The learning rate starts from 0.01 and decays at a rate of 0.8 every 1000 iterations. We train each policy with $5 \times 10^{3}$ and $2 \times 10^{4}$ iterations on the synthetic and real world datasets.

Multi-agent RL. We use proximal policy optimization (PPO) [41] as our policy optimization algorithm in Sect. 4.2. We add an extra entropy term to the loss function for better exploration. The entropy of the policies are estimated using the negative log-likelihood of one sampled action for each state. The weight of the entropy loss starts with 1.0 and decays at a rate of 0.999 per iteration. We train all players independently at the same time. We use the Adam optimizer [24] with a fixed learning rate of $10^{-4}$. The training lasts $10^4$ epochs with a batch size of 512 for the repeated iterated games and 2048 for the farm security game.