Autonomous exploration of motor skills by skill babbling

Abstract

Autonomous exploration of motor skills is a key capability of learning robotic systems. Learning motor skills can be formulated as inverse modeling problem, which targets at finding an inverse model that maps desired outcomes in some task space, e.g., via points of a motion, to appropriate actions, e.g., motion control policy parameters. In this paper, autonomous exploration of motor skills is achieved by incrementally learning inverse models starting from an initial demonstration. The algorithm is referred to as skill babbling, features sample-efficient learning, and scales to high-dimensional action spaces. Skill babbling extends ideas of goal-directed exploration, which organizes exploration in the space of goals. The proposed approach provides a modular framework for autonomous skill exploration by separating the learning of the inverse model from the exploration mechanism and a model of achievable targets, i.e. the workspace. The effectiveness of skill babbling is demonstrated for a range of motor tasks comprising the autonomous bootstrapping of inverse kinematics and parameterized motion primitives.

Appendix: Learning algorithms

For skill babbling, any learning algorithm is applicable which solves the weighted regression problem (6) online. In this paper, the two following learning algorithms are applied.

1.1 Locally linear model

The first learner is a Locally Linear Model (LLM, Ritter 1991) as it is implemented by Rolf et al. (2011). It comprises $l=1,\ldots ,L$ linear models

$$\begin{aligned} g_{l}(\mathbf {x}) = {\mathbf {W}}_l \mathbf {x}+ {\mathbf {b}}_l . \end{aligned}$$

The linear models $g_l(\mathbf {x})$ are combined according to

$$\begin{aligned} \mathbf {y}(\mathbf {x}) = \frac{1}{n(\mathbf {x})} \sum _{l=1}^L b\left( \frac{\mathbf {x}- \mathbf {p}_l}{d}\right) g_l\left( \frac{\mathbf {x}- \mathbf {p}_l}{d}\right) \end{aligned}$$

with Gaussian responsibilities

$$\begin{aligned} b(\mathbf {x}) = exp(-||\mathbf {x}||^2) \end{aligned}$$

and the normalization

$$\begin{aligned} n(\mathbf {x}) = \sum _{l=1}^L b\left( \frac{\mathbf {x}- \mathbf {p}_l}{d}\right) , \end{aligned}$$

where the radius $d$ determines the area of responsibility for each linear model around prototypical centers $\mathbf {p}_l$.

For supervised training, appropriate centers $\mathbf {p}_l$ for the linear models have to be found and the parameters ${\mathbf {W}}_l$ and ${\mathbf {b}}_l$ of the linear models have to be learned. Following the implementation by Rolf et al. (2011), centers are incrementally added to the learner according to a vector quantization algorithm. For the initial training sample $(\mathbf {x}_1, \mathbf {y}_1)$, the first linear model with center $\mathbf {p}_1 = \mathbf {x}_1$, linear part ${\mathbf {W}}_1 = {\mathbf {0}}$, and bias ${\mathbf {b}}_1 = \mathbf {y}_1$ is created. For new training samples $(\mathbf {x}_n, \mathbf {y}_n)$ together with the weight $w_n$, the weighted square error $w_n ||\mathbf {y}_n - \mathbf {y}(\mathbf {x}_n)||^2$ is minimized by online gradient descend with learning rate $\eta$.

1.2 Extreme learning machine with weighted recursive least squares learning

The second learner is a variant of an Extreme Learning Machine (ELM, Huang et al. 2004) with weighted recursive least squares learning. ELMs are feedforward neural networks with a single hidden layer. The output is computed according to

$$\begin{aligned} \mathbf {y}(\mathbf {x}) = {\mathbf {W}}^{\text {out}}{^T} \varvec{\sigma }({\mathbf {W}}^{\text {inp}}\mathbf {x}+ {\mathbf {b}}), \end{aligned}$$

where $\sigma (a) = 1/(1 + exp(-a))$ is a sigmoid activation function applied component-wise to the synaptic summations ${\mathbf {a}} = {\mathbf {W}}^{\text {inp}}\mathbf {x}+ {\mathbf {b}}$. The special property of ELMs is that learning is restricted to the read-out weights ${\mathbf {W}}^{\text {out}}$, which makes backpropagation of errors dispensable. Learning then boils down to a simple linear regression problem for ${\mathbf {W}}^{\text {out}}$. The input weights ${\mathbf {W}}^{\text {inp}}\in {\mathbb {R}}^{H\times dim(\mathbf {x})}$ and biases ${\mathbf {b}}\in {\mathbb {R}}^{H}$ are initialized randomly and remain fixed. Typically, the number of hidden neurons $H$ is chosen large in comparison to the number of inputs. In this paper, the values of ${\mathbf {W}}^{\text {inp}}$ and ${\mathbf {b}}$ are drawn from uniform distributions in range $[-2, 2]$ and $[-1, 1]$ if not stated otherwise. Note that the idea of using feedforward neural networks with a random hidden layer has been proposed earlier, e.g., by Schmidt et al. (1992).

While the variant of ELMs proposed by Liang et al. (2006) does feature sequential learning, it does not incorporate a weighted error criterion and does not make use of a forgetting factor. In this paper, a weighted recursive least squares algorithm similar to Haykin (1991) is applied in order to update the read-out weights ${\mathbf {W}}^{\text {out}}$ sequentially. For the first training sample $(\mathbf {x}_1, \mathbf {y}_1)$, the read-out weights are initialized according to

$$\begin{aligned} {\mathbf {W}}^{\text {out}}_1 = {\mathbf {P}}_1 \mathbf {h}(\mathbf {x}_1) \mathbf {y}_1^T \, \text { with } \, {\mathbf {P}}_1 = (\mathbf {h}(\mathbf {x}_1) \mathbf {h}(\mathbf {x}_1)^T + \varepsilon \mathbbm {1}_{H})^{-1}, \end{aligned}$$

where

$$\begin{aligned} \mathbf {h}(\mathbf {x}) = \sigma ({\mathbf {W}}^{\text {inp}}\mathbf {x}+ {\mathbf {b}}) \, \in {\mathbb {R}}^{H} \end{aligned}$$

are the hidden neuron activations for input $\mathbf {x}$ and $\varepsilon > 0$ is a regularization parameter. For new training samples $(\mathbf {x}_n, \mathbf {y}_n)$ together with the weight $w_n$, the read-out weights are updated sequentially according to the weighted recursive least squares rule

$$\begin{aligned} \varvec{\kappa }_n&= \frac{1}{\lambda + \mathbf {h}(\mathbf {x}_n)^T {\mathbf {P}}_{n-1} \mathbf {h}(\mathbf {x}_n)} {\mathbf {P}}_{n-1} \mathbf {h}(\mathbf {x}_n) \\ {\mathbf {W}}^{\text {out}}_n&= {\mathbf {W}}^{\text {out}}_{n-1} + w_n \varvec{\kappa }_n \left( \mathbf {y}_n - {\mathbf {W}}^{\text {out}}_{n-1}{^T} \mathbf {h}(\mathbf {x}_n)\right) \\ {\mathbf {P}}_n&= \frac{1}{\lambda } \left( {\mathbf {P}}_{n-1} - \varvec{\kappa }_n^T \mathbf {h}(\mathbf {x}_n) {\mathbf {P}}_{n-1}\right) , \end{aligned}$$

where $0 \ll \lambda \le 1$ is a forgetting factor. That is, $\lambda = 1$ corresponds to no forgetting.