Finding intrinsic rewards by embodied evolution and constrained reinforcement learning

Abstract

Understanding the design principle of reward functions is a substantial challenge both in artificial intelligence and neuroscience. Successful acquisition of a task usually requires not only rewards for goals, but also for intermediate states to promote effective exploration. This paper proposes a method for designing ‘intrinsic’ rewards of autonomous agents by combining constrained policy gradient reinforcement learning and embodied evolution. To validate the method, we use Cyber Rodent robots, in which collision avoidance, recharging from battery packs, and ‘mating’ by software reproduction are three major ‘extrinsic’ rewards. We show in hardware experiments that the robots can find appropriate ‘intrinsic’ rewards for the vision of battery packs and other robots to promote approach behaviors.

Introduction

In application of reinforcement learning algorithms to real world problems, the design of the reward function is critical for successful achievement of a task. Although it appears straightforward to assign positive rewards to desired goal states and negative rewards to states to be avoided, finding a good balance between multiple rewards often needs careful tuning (Kamioka, Uchibe, & Doya, 2007). Furthermore, if rewards are given only at isolated goal states, blind exploration of the state space takes an extremely long time except in toy problems. Rewards at intermediate sub-goals or even along the trajectories leading to goal promote focused exploration, but appropriate design of such additional rewards usually requires prior knowledge of the task or trial and error by the experimenter.

In this paper, we consider a reinforcement learning framework with two types of reward functions: the extrinsic rewards that are directly linked with the achievement of a task or the fitness of an agent and the intrinsic rewards that implicitly help success of the task or fitness of the agent. We propose a method for autonomous agents to find appropriate intrinsic rewards by combining constrained reinforcement learning (Uchibe & Doya, 2007a) and embodied evolution (Elfwing, 2007, Elfwing et al., in press).

A popular way for promoting exploratory behavior is the use of exploratory bonuses (Främling, 2007), or equivalently, to set optimistic initial value functions. While exploration bonuses just promote uniform scanning of the state space, recent studies called ‘Intrinsically Motivated Reinforcement Learning’ (IMRL) aimed at designing intrinsic rewards to guide robots to ‘interesting’ parts of the state space. The criteria for such intrinsic rewards include the prediction errors of the robot’s internal model (Barto et al., 2004, Meeden et al., 2004, Singh et al., 2005, Stout et al., 2005) and the reduction in the prediction errors (Oudeyer and Kaplan, 2004, Oudeyer et al., 2007). In this study, instead of assuming particular forms of intrinsic rewards, we let distributed autonomous robots, Cyber Rodents (Doya & Uchibe, 2005), find appropriate intrinsic reward functions through evolution. While a fixed set of extrinsic rewards specify the agents’ constraints of survival by capturing battery packs and reproduction by exchanging their ‘genes’ by infrared communication, intrinsic reward functions that facilitate goal-directed exploration are found by evolution in a colony of the robots.

We first outline the general method of reinforcement learning by intrinsic reward functions under the task constraints imposed as extrinsic reward functions (Uchibe & Doya, 2007a) and finding appropriate intrinsic reward functions by embodied evolution. We then introduce the Cyber Rodent platform we use for our experiments and describe the implementation and result of the experiments. We presented our preliminary results at ICONIP2007 (Uchibe & Doya, 2007b). Here we present the results of more systematic experiments on the robustness and the effectiveness of our proposed approach.