Learning Robot Arm Controls Using Augmented Random Search in Simulated Environments

Abstract

We investigate the learning of continuous action policy for controlling a six-axes robot arm. Traditional tabular Q-Learning can handle discrete actions well but less so for continuous actions since the tabular approach is constrained by the size of the state-value table. Recent advances in deep reinforcement learning and policy gradient learning abstract the look-up table using function approximators such as artificial neural networks. Artificial neural networks abstract loop-up policy tables as policy networks that can predict discrete actions as well as continuous actions. However, deep reinforcement learning and policy gradient learning were criticized for their complexity. It was reported in recent works that Augmented Random Search (ARS) has a better sample efficiency and a simpler hyper-parameter tuning. This motivates us to apply the technique to our robot-arm reaching tasks. We constructed a custom simulated robot arm environment using Unity Machine Learning Agents game engine, then designed three robot-arm reaching tasks. Twelve models were trained using ARS techniques. Another four models were trained using the state-of-the-art PG learning technique i.e., proximal policy optimization (PPO). Results from models trained using PPO provide a baseline from the policy gradient technique. Empirical results of models trained using ARS and PPO were analyzed and discussed.