Iterative residual tuning for system identification and sim-to-real robot learning

Abstract

Robots are increasingly learning complex skills in simulation, increasing the need for realistic simulation environments. Existing techniques for approximating real-world physics with a simulation require extensive observation data and/or thousands of simulation samples. This paper presents iterative residual tuning (IRT), a deep learning system identification technique that modifies a simulator’s parameters to better match reality using minimal real-world observations. IRT learns to estimate the parameter difference between two parameterized models, allowing repeated iterations to converge on the true parameters similarly to gradient descent. In this paper, we develop and analyze IRT in depth, including its similarities and differences with gradient descent. Our IRT implementation, TuneNet, is pre-trained via supervised learning over an auto-generated simulated dataset. We show that TuneNet can perform rapid, efficient system identification even when the true parameter values lie well outside those in the network’s training data, and can also learn real-world parameter values from visual data. We apply TuneNet to a sim-to-real task transfer experiment, allowing a robot to perform a dynamic manipulation task with a new object after a single observation.

Appendix

1.1 TuneNet architecture implementation details

We implement all networks using PyTorch (Paszke et al. 2017), and train them using stochastic gradient descent. Our TuneNet architecture for all experiments consists of fully-connected (FC) layers with ReLU activation functions for each of the feature extractors, with each feature extractor output size set to 128. The estimator therefore has an input size of 256. We model the estimator with two fully connected layers with a hidden size of 128, ReLU activation for the first layer, and a tanh activation for the final output.

1.2 Details for Experiment 1: End-effector mass identification

The object’s mass is randomly chosen, $m \sim U({0}\,\hbox {kg}, {1}\,\hbox {kg})$, for each training point. In each simulated episode, which runs for 5 s, the robot moves its end effector in a circle defined by the coordinates $[-\,0.4, 0.2\text {cos}(\tau ), 0.2\text {sin}(\tau )], \tau = 1.2t$. We collect 1000, 300, and 300 episodes for the training, validation, and test sets, respectively. We trained the model for 1000 epochs using a batch size of 50, a learning rate of 0.01, L2 regularization $\lambda = 1e{-}3$, and 1% learning rate decay every 50 epochs. During training, the input torques were normalized to the range [0, 1] but the outputs were not transformed.

1.3 Details for Experiment 2: Bouncing ball COR tuning

For both datasets, we trained TuneNet for 200 epochs, using a batch size of 50 and a learning rate of 1e${-}$2, with learning rate decay of 1% per 5 epochs. The L2 normalization constant $\lambda$ is set to 0.01. During training, the input and output channels were normalized to the range [0, 1].

For the Obs case, the visual OpenCV tracker uses hue, saturation, and value windows to threshold a camera image and isolate an object. It then calculates the y pixel position of the ball in each frame and normalizes the image pixel positions to the range [0, 1]. For each randomized simulation render, the PyBullet camera was placed at random polar coordinates $r \sim \text {U}$ (5 m, 10 m), $\theta \sim \text {U}(0, 2\pi )$, $z \sim \text {U}$ (5 m, 8 m), and aimed at the point (0, 0, 2 m).

1.3.1 Baseline implementation details

For CMA-ES, we use the open-source package pycma.³ The inputs to CMA-ES are identical to those provided to TuneNet, with the initial proposed physics parameters being used as the initial guess for the CMA algorithm. The CMA tolerance was set to 0.1.

For Greedy Entropy Search (EntSearch in this paper), we re-implemented the algorithm provided in Zhu et al. (2018). The underlying Gaussian Process (GP) sampling was implemented using the open-source Python package GPy,⁴ and we used an RBF GP kernel. Again, the initial proposed physics parameters were used as the initial guess for the algorithm. We discretize the parameter space into 50 bins, and use a gaussian process sampling population size of $n=100$. Because we do not calculate the value function for each policy, we cannot use the value as the stopping criterion as in Zhu et al. (2018). Instead, we stop iterating when the predicted parameter value does not change for two consecutive timesteps. This stopping parameter was determined empirically for our experiment after being shown to outperform two other stopping criteria: no stopping (which performed poorly because of noise in the GP sampling), and stopping after the parameter estimate changed by less than a threshold $\epsilon > 0$.

The initial guesses and simulated outcomes supplied to both CMA-ES and Greedy Entropy Search were identical to the inputs supplied to TuneNet, including all normalization transformations applied to the data.

The “Direct Prediction” network uses the same network architecture as TuneNet, but rather than stacking the observations from two models, the single model input is padded with zeros. This network is trained entirely from scratch separately from TuneNet (no weights are shared between the two models).