Landscape Analysis of Stochastic Policy Gradient Methods

Abstract

Policy gradient methods are among the most important techniques in reinforcement learning. Despite the inherent non-concave nature of policy optimization, these methods demonstrate good behavior, both in practice and in theory. Hence, it is important to study the non-concave optimization landscape. This paper aims to provide a comprehensive landscape analysis of the objective function optimized by stochastic policy gradient methods. Using tools borrowed from statistics and topology, we prove a uniform convergence result for the empirical objective function, (and its gradient, Hessian and stationary points) to the corresponding population counterparts. Specifically, we derive $\tilde{O}(\sqrt{|\mathcal {S}||\mathcal {A}|}/(1-\gamma )\sqrt{n})$ rates of convergence, with the sample size n, the state space $\mathcal {S}$, the action space $\mathcal {A}$, and the discount factor $\gamma$. Furthermore, we prove the one-to-one correspondence of the non-degenerate stationary points between the population and the empirical objective. In particular, our findings are agnostic to the choice of the algorithm and hold for a wide range of gradient-based methods. Consequently, we are able to recover and improve numerous existing results through the vanilla policy gradient. To the best of our knowledge, this is the first work theoretically characterizing optimization landscapes of stochastic policy gradient methods.