Robot Trajectory Optimization with Reinforcement Learning Based on Local Dynamic Fitting

Abstract

With the development of artificial intelligence, reinforcement learning plays an increasingly important role in the robot operation filed. In this paper, a trajectory optimization method based on local dynamic model fitting is proposed to improve sample utilization and reduce the difficulty of dynamic model learning. Firstly, the Gaussian mixture model of the robot was constructed, and based on this, the accurate local dynamics model was obtained through the Normal-inverse-wishart distribution. Secondly, LQR optimization algorithm was used to optimize the robot trajectory, and the optimal control strategy was obtained during the grasping process of the robot. Finally, the effectiveness of the proposed algorithm is verified on the dynamic simulation platform. The experimental results show that the method proposed in this paper can significantly improve sample utilization and learning efficiency.

Appendix

In this appendix, we present the derivation of the dynamic model coefficients. Firstly, we need to define some temporary variables.

The random variable $\left[ {s_t ,a_t } \right]$ and $s_{t + 1}$ are represented by $Y$ and $X$ respectively. $\mu_X$ and $\Sigma_X$ represent the mean value and covariance of $X$, and $\Sigma_X = \delta_X \delta_X^T$. $\mu_Y$ and $\Sigma_Y$ represent the mean value and covariance of $Y$, and $\Sigma_Y = \delta_Y \delta_Y^T$.

Secondly, we prove $F_{sat}^T = \Sigma_Y^{ - 1} \Sigma_{YX}$.

$\begin{gathered} \Sigma_Y F_{sat}^T - \Sigma_{YX} \hfill \\ = \delta_Y \delta_Y^T F_{sat}^T - \delta_Y \delta_X^T \hfill \\ = \iint {p(x,y)\left[ {(y - E(y))(y - E(y))^T F_{sat}^T - (y - E(y))(x - E(x))^T } \right]}dxdy \hfill \\ = \int_y {(y - E(y))dy\int_x {p(x,y)\left[ {F_{sat} y - F_{sat} E(y) - x + E(x)} \right]^T dx} } \hfill \\ \end{gathered}$ Where $p(x,y) = p(s_t ,a_t ,s_{t + 1} )$.

As $E(X) = E(F_{sat} y + f_t ) = F_{sat} E(y) + f_t$, substitute this into equation above we can get:

$$ \begin{gathered} \Sigma_Y F_{sat}^T - \Sigma_{YX} \hfill \\ = \int_y {(y - E(y))dy\int_x {p(x,y)\left[ {F_{sat} y - F_{sat} E(y) - x + F_{sat} E(y) + f_t } \right]^T dx} } \hfill \\ = \int_y {(y - E(y))dy\int_x {p(x,y)\left[ {F_{sat} y - x + f_t } \right]^T dx} } \hfill \\ = \int_y {(y - E(y))p_y (y)dy\left[ {(F_{sat} y + f_t - E(x|y))} \right]} \hfill \\ = \int_y {(y - E(y))p_y (y)dy*} 0 \hfill \\ = 0 \hfill \\ \end{gathered} $$

Thirdly, we prove $f_t = \mu_X - F_{sat} \mu_Y$.

As $E(X) = E(s_{t + 1} ) = \mu_X$,$E(Y) = E(s_t ,a_t ) = \mu_Y$, we can obtain:

$$ f_t = \mu_X - F_{sat} \mu_Y $$

Finally, we prove $\delta_{st} = \Sigma_X - F_{sat} \Sigma_Y F_{sat}^T$.

As we know that $E(X|Y = y) = F_{sat} y + f_t$, we can derive that:

$$ D(E(X|Y)) = D(F_{sat} y + f_t ) = F_{sat} D(Y)F_{sat}^T = F_{sat} \Sigma_Y F_{sat}^T $$

And because of $D(X|Y) = \delta_{st}$, we can derive that:

$$ E(D(X|Y)) = \delta_{st} $$

As $D(X) = D(E(X|Y)) + E(D(X|Y))$, we can get:

$$ \Sigma_X = F_{sat} \Sigma_Y F_{sat}^T + \delta_{st} $$