Robotic grasping and manipulation through human visuomotor learning

https://doi.org/10.1016/j.robot.2011.09.002Get rights and content

Abstract

A major goal of robotics research is to develop techniques that allow non-experts to teach robots dexterous skills. In this paper, we report our progress on the development of a framework which exploits human sensorimotor learning capability to address this aim. The idea is to place the human operator in the robot control loop where he/she can intuitively control the robot, and by practice, learn to perform the target task with the robot. Subsequently, by analyzing the robot control obtained by the human, it is possible to design a controller that allows the robot to autonomously perform the task. First, we introduce this framework with the ball-swapping task where a robot hand has to swap the position of the balls without dropping them, and present new analyses investigating the intrinsic dimension of the ball-swapping skill obtained through this framework. Then, we present new experiments toward obtaining an autonomous grasp controller on an anthropomorphic robot. In the experiments, the operator directly controls the (simulated) robot using visual feedback to achieve robust grasping with the robot. The data collected is then analyzed for inferring the grasping strategy discovered by the human operator. Finally, a method to generalize grasping actions using the collected data is presented, which allows the robot to autonomously generate grasping actions for different orientations of the target object.

Highlights

► Extended the robot skill synthesis via human learning paradigm to grasping actions. ► Discovered features generated by humans using the robot as an extension of the body. ► Exploiting human capacity to learn control tasks for synthesizing robot behaviors.

Introduction

Although humans are very skilled at manipulation, and in spite of the recent developments in robotic manipulation to match this skill, developing a robotic system that comes close to human hand dexterity is still elusive. Imitation and learning from observation/demonstration research in robotics aim at removing the burden of robot programming from experts by using human guidance to teach the robot a given skill such as manipulation. The simplest method to transfer a skill to a robot is to directly copy the motor commands of the demonstrator to the robot, the so called motor tape approach [1], which is effective for open-loop tasks. However, in general, it is not possible to adopt this approach. First, motor commands may not be available to the robot; even if available, the differences between the demonstrator and the robot often render the motor commands useless for the robot. Another approach which is commonly used is the so called direct teaching [2], [3], [4] where the robot behavior is shaped or molded by directly moving the joints of the robot to obtain the desired behavior. Direct teaching is not practical for complex tasks that may include non-negligible dynamics. Another alternative is the vision based imitation approach [5], [6], [7], [8], [9], [10], [11], [12], [13], [14] where the motion of a human demonstrator is mapped onto the kinematical structure of a robot. Current vision based imitation systems may fail to fulfill the precision and robustness requirements of many manipulation tasks as it is often not possible to accurately extract the finger configuration of the demonstrator through computer vision. To sidestep the problems related to computer vision, sophisticated motion capture systems can be deployed, which give accurate human motion data. However, due to the different dynamical properties of the robot and the human, the success of this approach heavily depends on the expert knowledge that goes in the transformation that takes the human motion and maps it onto the robot.

A new paradigm for obtaining skilled robot behavior is to utilize an intuitive teleoperation system where humans learn to control the robot joints to perform a given task [15]. Once the human manages to control the robot and complete the desired task with it, the control commands produced by the human–robot system during task execution can be used for designing controllers that operate autonomously. This places the initial burden of learning on the human instructor, but allows the robot to ultimately acquire the ability to perform the target skill without human guidance.

In this article, we report our progress on applying this paradigm to manipulation using an anthropomorphic robot hand. As the first task we study a two-balls manipulation task, where the robot is required to swap the so called Chinese balls without dropping them. We first briefly recap some results of the ball swapping task that we have published in conference proceedings earlier [15], [16], and then perform additional analysis. In particular, we present new results concerning the intrinsic dimensionality of the obtained ball swapping skill. As the second task, we focus on grasping and describe the derivation of a reaching and grasping controller using this paradigm. We give the details of the grasping interface we developed and present several results on the human control of the robot and actual robot grasp synthesis.

Section snippets

Method

To better exemplify the paradigm a bit of digression is in order. As soon as we hold a computer mouse, it almost becomes a part of our body; we use it fluently as we control our hands. This introspection has a neural basis [17]. In addition, experiments with monkeys have shown that arm and hand representations in the brain are very plastic. For example, as soon as a rake is grabbed with the purpose of reaching objects that are out of reach, the hand representation extends to represent the tool

Human–robot interface

For the ball swapping task, a 16DOF robot hand (Gifu Hand III, Dainichi Co. Ltd., Japan) was attached to a robotic arm (PA-10 robot, Mitsubishi Heavy Industries), which kept the hand close to horizontal, palm facing up posture. A fixed orientation was chosen and was used without change throughout. The task was to move the fingers of the robot hand so that the balls change their position without falling down. Humans can easily perform this task with their own hand, often requiring palm and thumb

Human–robot interface

For the grasping task, we used the same robotic hand (Gifu Hand III, Dainichi Co. Ltd., Japan) and arm (PA-10 robot, Mitsubishi Heavy Industries) as in the ball swapping task (see Fig. 5). However in this case, the arm robot was also actively controlled by the human; so the grasping task included reaching for grasping and the actual flexion of the finger around the target object so that a stable grasp is obtained.

For actuating the robot arm based on real-time human movements, an

Conclusion

In this study we have presented our ongoing work on extending the robot skill synthesis via human learning paradigm to grasping actions and provided additional analysis of the ball swapping skill. Although the grasping experiments conducted so far are preliminary, we were able to discover interesting features of the grasps generated by the human using the robot as an extension of the body, albeit the different kinematics of the robot. The data indicates that the human operator positioned the

Acknowledgments

We would like to thank Emre Ugur for the development of the simulator and his technical support throughout this project. B. Moore was supported by the Japan Society for the Promotion of Science (JSPS). E. Oztop was supported in part by the Global COE Program, Center of Human-Friendly Robotics Based on Cognitive Neuroscience at the Ministry of Education, Culture, Sports, Science and Technology, Japan.

Brian Moore earned his Ph.D. from the Johannes Kepler University. During this period, he was a member of the Research Institute for Symbolic Computation (RISC) and a research scientist at the Radon Institute for Computational and Applied Mathematics (RICAM) of the Austrian Academy of Sciences. Then he worked as a researcher at the ATR Cognitive Mechanisms Laboratories in Japan. He is now a researcher in the robotic lab at the Laval University.

References (28)

  • L. Berthouze, P. Bakker, Y. Kuniyoshi, Development of oculo-motor control through robotic imitation, in: IEEE/RSJ...
  • Y. Demiris et al.

    Imitation as a dual-route process featuring predictive and learning components: a biologically-plausible computational model

  • A. Ijspeert et al.

    Learning attractor landscapes for learning motor primitives

  • Y. Kuniyoshi, Y. Kuniyoshi, Y. Yorozu, M. Inaba, H. Inoue, From Visuo–Motor self learning to early imitation–a neural...
  • Cited by (12)

    Brian Moore earned his Ph.D. from the Johannes Kepler University. During this period, he was a member of the Research Institute for Symbolic Computation (RISC) and a research scientist at the Radon Institute for Computational and Applied Mathematics (RICAM) of the Austrian Academy of Sciences. Then he worked as a researcher at the ATR Cognitive Mechanisms Laboratories in Japan. He is now a researcher in the robotic lab at the Laval University.

    Erhan Oztop earned his Ph.D. from the University of Southern California in 2002. After he obtained his degree, he joined Computational Neuroscience Laboratories at the Advanced Telecommunications Research Institute International, (ATR) in Japan. There he worked as a researcher and later a group leader within the JST ICORP Computational Project during 2004–2008. From 2009 to 2011 he was a senior researcher at ATR Cognitive Mechanisms Laboratories where he led the Communication and Cognitive Cybernetics group. During this time he also held a visiting associate professor position at Osaka University and was affiliated with the National Institute of Information and Communication Technology, Kyoto, Japan. Currently, he is an engineering faculty member of Ozyegin University, Istanbul, Turkey, and is affiliated with ATR, Kyoto, Japan. His research interests include computational modeling of the brain mechanisms of action understanding and intelligent behavior, human-machine interface, robotics, machine learning and cognitive neuroscience.

    View full text