Toward a Mechanistic Account for Imitation Learning: An Analysis of Pendulum Swing-Up

Abstract

Learning an action from others require to infer their underlying goals, and recent psychological studies have reported behavioral evidences that young children do infer others’ underlying goals by observing their actions. The goal of the present study is to propose a mechanistic account for how this goal inference is possible by observing others’ actions. For this purpose, we performed a series of simulations in which two agents control pendulums toward different goals, and analyzed with which types of features it is possible to infer their different latent goals and control schemes. Our analysis showed that pointwise dimension, a type of fractal dimension, of the pendulum movements is sufficiently informative to classify the types of agents. With respect to its invariant nature, this result suggests that the fine-grained movement patterns such as the fractal dimension reflect the structure of the underlying control schemes and goals.

A Reinforcement learning

Reinforcement learning [12] is a framework rooted in behavioral psychology and control theory. In the task environment in state s, the learner takes an action a and receives a reward r from the environment in response to the action. Next, the learner faces with the environment in a new state $s' = Q(s'|s)$, where Q is a transition function. The goal of learning is to acquire a control scheme g(a|s) that maximizes the cumulative reward.

The pendulum swing-up task is a classic control problem with continuous space and time [5, 11]. There are many researches to solve this task (e.g., [6] for recent updates). The simple and basic algorithm for this task is so-called actor-critic architecture [6, 12]. It is composed of two, the actor and critic components. The actor represents the control scheme g(a|s). On the other hand, the critic represents the value function V(s), that tells the learner the discounted expected reward of state s.

Since the task is in continuous space and time, it involves several engineering problems. The typical approach is discretization of the continuous space and time. For continuous time, we used discretized time steps for Eular integration (step size $\text {d}t = 0.01$) and we sampled per 3 time steps. For continuous state space, we adopted a discretized representation (tile coding [12]) in which the continuous state space $(\theta , \dot{\theta }) \in [-\pi ,\pi ] \times [-2\pi ,2\pi ]$ is equally divided into the grid of size 40 $\times$ 40 (one of them is called here state s).

Learning proceeds with estimation of state values, and that shapes g(a|s) to navigate to more rewarding states. Suppose the pendulum is now in state s (one of the grid) at time t. The learner supplies a control input u sampled from $g(u|s) = N(\mu _s,\sigma _s)$, a normal distribution of mean $\mu _s$ and variance $\sigma _s^2$. In response, the learner observes the pendulum in a new state $s'$ and a reward r. Then the learner increments his value function V(s) for every s by

$$\begin{aligned} \varDelta V(s) = \alpha _c\,[r + \gamma _c\,V(s') - V(s)]\,E_c(s) \end{aligned}$$

(3)

where $\alpha _c = 0.1$ is a learning rate, $\gamma _c = 0.97$ is a discount rate, and $E_c$ is an eligibility trace with exponential decay (given by $\lambda _c = 0.65$) that is a device for continuous tasks that assigns higher weights for recently visited states.

For continuous control inputs, the control scheme $g(\cdot )$ is expressed by a collection of normal distributions for each state s. So the learner has to determine the mean $\mu$ and variance $\sigma ^2$ for each state s. Formally, the learner modifies his control scheme $\mu _s$ and $\sigma _s$ for every s by

$$\begin{aligned}&\varDelta \mu _s = \alpha _a\,[r + \gamma _a\,V(s') - V(s)]\,\frac{\partial N(\mu _s,\sigma _s)}{\partial \mu _s}(u)\,E_a(s) \end{aligned}$$

(4)

$$\begin{aligned}&\varDelta \sigma _s = \alpha _a\,[r + \gamma _a\,V(s') - V(s)]\,\frac{\partial N(\mu _s,\sigma _s)}{\partial \sigma _s}(u)\,E_a(s) \end{aligned}$$

(5)

where $\alpha _a = 0.001$ is a learning rate, $\gamma _a = 0.65$ is a discount rate, and $E_a$ is an eligibility trace with decay (given by $\lambda _a = 0.0$).

The reward function $r = f(s, a)$ must be designed carefully. From [5], we set the reward function $r = \cos (\theta )$ for this task. It only depends on angle $\theta$. Remark that $\cos (\theta )$ characterizes the goal of this task, because the inverted position $\theta = 0$ gives the highest reward $\cos (0) = 1$, and the hanging-down position $\theta = \pm \pi$ gives the lowest reward $\cos (\pi ) = -1$.