An Empirical Analysis of Gumbel MuZero on Stochastic and Deterministic Einstein Würfelt Nicht!

Abstract

MuZero and its successors, Gumbel MuZero and Stochastic MuZero, have achieved superhuman performance in many domains. MuZero combines Monte Carlo tree search and model-based reinforcement learning, which allows it to be utilized in complex environments without prior knowledge of actual dynamics. Gumbel MuZero enhances the training quality of MuZero by guaranteeing policy improvement, which allows it to learn with a limited number of simulations for tree search. Stochastic MuZero broadens the applicable domains using a redesigned model, which allows it to cope with stochastic environments. Recently, an approach combining Gumbel MuZero and Stochastic MuZero was applied to a stochastic game called 2048, discovering a counterintuitive phenomenon: agents trained with only 3 simulations performed better than agents trained with 16 or 50 simulations. However, this phenomenon has only been observed in 2048 and awaits further investigations. This paper aims to examine two questions, namely Question 1: whether this phenomenon also happens in another well-known stochastic game, EinStein würfelt nicht! (EWN), and Question 2: whether the stochasticity of the environment is the main reason for the phenomenon. To investigate these questions, this paper analyzes the training results using stochastic EWN and four deterministic EWN variants. The experiments confirm that the phenomenon also happens in the stochastic EWN, while not in the deterministic variants, suggesting that stochasticity leads to better performance of agents trained with lower simulations.

C.-L. Kuo and P.-T. Chen—Equal contribution.