Discovering relevant task spaces using inverse feedback control

Abstract

Learning complex skills by repeating and generalizing expert behavior is a fundamental problem in robotics. However, the usual approaches do not answer the question of what are appropriate representations to generate motion for a specific task. Since it is time-consuming for a human expert to manually design the motion control representation for a task, we propose to uncover such structure from data-observed motion trajectories. Inspired by Inverse Optimal Control, we present a novel method to learn a latent value function, imitate and generalize demonstrated behavior, and discover a task relevant motion representation. We test our method, called Task Space Retrieval Using Inverse Feedback Control (TRIC), on several challenging high-dimensional tasks. TRIC learns the important control dimensions for the tasks from a few example movements and is able to robustly generalize to new situations.

Appendix

1.1 Proof of proposition 1 for the direction of IK generated motion steps

Proposition 1 If $\varrho \rightarrow \infty$ then the IK solution $q_{t+1}$ minimizing Eq. (9) has the property that the next step $q_{t+1} - q_{t}$ is approximately proportional to the value function gradient $\mathcal {J}$ in a small region around $q_t$.

Proof

If $\varrho \rightarrow \infty$ then the term $||f \circ \phi (q) - f \circ \phi (q_t) + \delta ||^2$ of Eq. (9) is weighted so high that $C_{prior}$ and any other cost terms we might add are neglected. Let $\mathcal {J}$ be the gradient of the value function $f \circ \phi (q)$ evaluated at $q = q_t$. Using the linearization $f \circ \phi (q_{t+1}) = f \circ \phi (q_t) + \mathcal {J}(q_{t+1} - q_t)$, we can apply the IK Equation (Toussaint 2011):

$$\begin{aligned} q_{t+1}&= q_t - \delta \mathcal {J^{\sharp }}\\ \mathcal {J^{\sharp }}&= {\left( \varrho \mathcal {J}^T\mathcal {J} + \mathbb {I} \right) }^{-1}\mathcal {J}^T\varrho \\&= \mathcal {J}^T {\left( \mathcal {J}\mathcal {J}^T + {\varrho }^{-1}\right) }^{-1} = \frac{1}{||\mathcal {J}||^2}\mathcal {J}^T \end{aligned}$$

We have used the Woodbury identity and the fact that $\mathcal {J}\mathcal {J}^T = ||\mathcal {J}||^2$ in the case where we have a 1-dimensional task variable $y$ (in that case the Jacobian is a row vector gradient). $\mathcal {J^{\sharp }}$ is called the pseudoinverse of $\mathcal {J}$. Thus, the steps generated by our motion model are proportional to $\mathcal {J}$ times a negative scalar number.$\square$

1.2 Proof of proposition 2 for Lyapunov attractor properties of TRIC

Proposition 2 Suppose we have trained TRIC on a single trajectory $\{q_t, y_t\}_{t=1}^T$ and that $f \circ \phi (q_T)$ is a minimum of the value function. Additionally, we generate motion with $\varrho \rightarrow \infty$, i.e. very high weighting of the value function. Then the motion generated by the model in Eq. (8) fulfills the conditions of Theorem 1 and is thus asymptotically stable at the attractor subspace $Q' = \{q': \phi (q') = \phi (q_T) = y_T \}$.

Proof

Because of $\varrho \rightarrow \infty$ the term decreasing the value function $f$ will dominate the motion equation and we can ignore the effect of the other terms. Let’s construct $V(q) = f\circ \phi (q) - c_T$, where $c_T = f\circ \phi (q_T)$. Then the Lyapunov stability conditions hold:

• (a) holds directly because of the assumption that $c_T= f\circ \phi (q_T)$ is minimum and any other joint state $q$ s.t. $\phi (q) \ne \phi (q_T)$ will have higher value $f\circ \phi (q)$ than it.

• (b) holds by the construction of $V(q)$ directly implying that

  $$\begin{aligned} V(q_T) = f\circ \phi (q_T) - c_T = 0 \end{aligned}$$

• Proposition 1 holds because we assumed that $\varrho \rightarrow \infty$. This implies that the steps of the motion model are proportional to the gradient $\mathcal {J}$. Thus the motion model of Eq. (9) will make steps decreasing the value $f\circ \phi (q_t)$ constantly and (c) holds.

• (d) holds because $c_T= f\circ \phi (q_T)$ is local minimum and after we reach a joint state $q'$ s.t. $\phi (q') = \phi (q_T)$ the gradient of the value function will be 0 and no further decrease will be possible.$\square$