Assist-as-needed robotic trainer based on reinforcement learning and its application to dart-throwing

Abstract

This paper proposes a novel robotic trainer for motor skill learning. It is user-adaptive inspired by the assist-as-needed principle well known in the field of physical therapy. Most previous studies in the field of the robotic assistance of motor skill learning have used predetermined desired trajectories, and it has not been examined intensively whether these trajectories were optimal for each user. Furthermore, the guidance hypothesis states that humans tend to rely too much on external assistive feedback, resulting in interference with the internal feedback necessary for motor skill learning. A few studies have proposed a system that adjusts its assistive strength according to the user’s performance in order to prevent the user from relying too much on the robotic assistance. There are, however, problems in these studies, in that a physical model of the user’s motor system is required, which is inherently difficult to construct. In this paper, we propose a framework for a robotic trainer that is user-adaptive and that neither requires a specific desired trajectory nor a physical model of the user’s motor system, and we achieve this using model-free reinforcement learning. We chose dart-throwing as an example motor-learning task as it is one of the simplest throwing tasks, and its performance can easily be and quantitatively measured. Training experiments with novices, aiming at maximizing the score with the darts and minimizing the physical robotic assistance, demonstrate the feasibility and plausibility of the proposed framework.

Introduction

Acquiring expertly skilled movements is generally a difficult task. Moreover, instructing novices in acquiring expertly skillful movements is also inherently difficult, because such movements are generated by unseen muscle (d’Avella and Bizzi, 2005, d’Avella et al., 2003, Murai et al., 2009) and neural activity (Hemami and Dariush, 2012, Ijspeert, 2008, Mylonas et al., 2012, Williamson, 1998).

This paper proposes a novel robotic trainer for motor skill learning. It is user-adaptive inspired by the assist-as-needed principle well known in the field of physical therapy (Cai et al., 2006, Emken and Reinkensmeyer, 2005, Jezernik et al., 2003).

Most previous studies in the field of the robotic assistance of motor skill learning have used predetermined desired trajectories, and it has not been examined intensively whether these trajectories were optimal for each user (Crespo and Reinkensmeyer, 2008, Duschau-Wicke et al., 2008, Emken and Reinkensmeyer, 2005).

Furthermore, the guidance hypothesis states that humans tend to rely too much on external assistive feedback, resulting in interference with the internal feedback necessary for motor skill learning (Schmidt & Wrisberg, 2008).

MIT-Manus (Krebs et al., 2004) and MIME (Lum et al., 2006) pioneered impedance control to rehabilitation robotics, and MIT-Manus (Krebs et al., 2003) include impedance selection based on the user’s performance. The impedance selection, however, has not been automatized, and instead physical therapists do the selection based on their knowledge and experience.

A few studies have proposed a system that adjusts its assistive strength according to the user’s performance in order to prevent the user from relying too much on the robotic assistance (Crespo and Reinkensmeyer, 2008, Emken and Reinkensmeyer, 2005). There are, however, problems in these studies, in that a physical model of the user’s motor system is required, which is inherently difficult to construct.

In this paper, we propose a framework for a robotic trainer that is user-adaptive and that neither requires a specific desired trajectory nor a physical model of the user’s motor system, and we achieve this using model-free reinforcement learning.

We chose dart-throwing as an example motor-learning task as it is one of the simplest throwing tasks, and its performance can be easily and quantitatively measured. Training experiments with novices aimed at maximizing the score with the darts and minimizing the physical robotic assistance. We demonstrate the feasibility and the plausibility of the proposed robotic trainer through experiments by comparing the results of four conditions: (1) without robot; (2) with non-adaptive fixed stiffness robot; (3) with adaptive robot; and (4) with non-adaptive decreasing stiffness robot.

This paper is organized as follows. Section  2 outlines the framework for our assist-as-needed robotic trainer. Section  3 describes how we applied this framework to develop the training system for dart-throwing. Section  4 describes our training experiments to validate the plausibility and the feasibility of the proposed training method. Section  5 presents the experimental results. Section  6 discusses the results while concluding remarks are provided in Section  7.