Modeling the Opponent’s Action Using Control-Based Reinforcement Learning

Abstract

In this paper, we propose an alternative to model-free reinforcement learning approaches that recently have demonstrated Theory-of-Mind like behaviors. We propose a game theoretic approach to the problem in which pure RL has demonstrated to perform below the standards of human-human interaction. In this context, we propose alternative learning architectures that complement basic RL models with the ability to predict the other’s actions. This architecture is tested in different scenarios where agents equipped with similar or varying capabilities compete in a social game. Our different interaction scenarios suggest that our model-based approaches are especially effective when competing against models of equivalent complexity, in contrast to our previous results with more basic predictive architectures. We conclude that the evolution of mechanisms that allow for the control of other agents provide different kinds of advantages that can become significant when interacting with different kinds of agents. We argue that no single proposed addition to the learning architecture is sufficient to optimize performance in these scenarios, but a combination of the different mechanisms suggested is required to achieve near-optimal performance in any case.

ITF and JP have contributed equally to this work.