Teaching and learning of robot tasks via observation of human performance

https://doi.org/10.1016/j.robot.2004.03.005Get rights and content

Abstract

Within this paper, an approach for teaching a humanoid robot is presented that will enable the robot to learn typical tasks required in everyday household environments. Our approach, called Programming by Demonstration, which is implemented and successfully used in our institute to teach a robot system is presented. Firstly, we concentrate on an analysis of human actions and action sequences that can be identified when watching a human demonstrator. Secondly, sensor aid systems are introduced which augment the robot’s perception capabilities while watching a human’s demonstration and the robot’s execution of tasks respectively. The main focus is then layed on the knowledge representation in order to be able to abstract the problem solution strategies and to transfer them onto the robot system.

Introduction

Service robots, which are supposed to assist humans in their daily household work, must be adaptable and flexible. They must be adaptable on the one hand to the individual requirements and needs and on the other hand to the user’s environment. As each household environment is different depending on its geometric setup, its provided functionality and the requested tasks to be performed by the robot, ready-made programs will not be able to cope with such environments. So, flexibility will surely be one of the most important requirements for the robot’s design. It must be able to navigate through a changing environment, to adapt its recognition abilities to a particular scenery and to manipulate a wide range of objects. Furthermore, it is extremely important for the user to easily adapt the system to his needs, i.e. to teach the system what to do and how to do it. We think that the only approach that satisfies the latter condition are systems that automatically acquire relevant information for task execution by observation and multi-modal dialogues.

Learning of action sequences or operation plans is an abstract problem, which supposes a modeling of cognitive skills. When learning complex action sequences, basic manipulation skills or controlling techniques are not investigated. In fact, the aim is to generate an abstract description of the demonstration reflecting the user’s intention and modeling the problem solution as optimal as possible. Robot independence is an important issue because it would allow to exchange of robot programs between robots with different kinematics. Likewise, a given problem has to be suitably generalized, for example, distinguishing parameters specific for the particular demonstration and parameters specific for the problem concept. Both demand a certain insight in the environment and the user’s performance.

Obviously, systems providing this functionality require powerful sensor systems to gather as much information as possible by observing humans behavior or processing explicit instructions like commands or comments. They need a methodology to transform observed information for a specific task to a robot independent and flexible knowledge structure, and actuator systems using this knowledge structure to generate actions that will solve the acquired task in a certain target environment.

The following article shows how a robot may be programmed based on a human demonstration. After presenting related approaches a classification of typical human actions is given in Section 3. Compared to previous publications, e.g. [6], we will concentrate on the recognition of demonstrated tasks, the knowledge representation and the applicability for robot systems in Section 4. In order to observe human actions and their effects in the environment we use a training center which is equipped with additional sensor systems to obtain all relevant data for the learning process. This is presented in Section 5 together with the execution environment where tasks are performed by the robot.

Section snippets

State of the art

Several programming systems and approaches based on human demonstrations have been proposed during the past years. Many of them address special problems or a special subset of objects only. An overview and classification of the approaches can be found in [4], [16].

Basis for the mapping of a demonstration to a robot system are the task representation and task analysis. Often, the analysis of a demonstration takes place observing the changes in the scene. These changes can be described using

Classification of human demonstrations

Within the framework of Programming by Demonstration, the basis for successfully mapping task solutions from a human to a robot system is the ability of the programming system to recognize and to interpret human actions. In cognitive psychology, human activity is characterized by three features [2]: Direction, Decomposition and Operator selection.

It is important to know that cognition is mainly taken to be directed as well. Furthermore, humans tend to perceive activity as a clearly separated

Programming by Demonstration

The Programming by Demonstration approach consists of several stages starting with the user’s demonstration, building a generalized representation of the segmented data and finally executing the gathered task knowledge. The whole cycle is described in detail in [6].

This section discusses questions concerning background knowledge required for understanding human demonstration, for abstraction and representing certain fulfilled tasks and for transferring demonstrations across different

Training and execution center

The robot itself disposes of several sensors that realize external entities. However, those are limited when observing human activity and the environment in order to learn the performed task. Assuming that background knowledge about tasks and actions is limited, valuable information must be extracted by intimately observing the user during a demonstration. For real-world problems simplified demonstration environments will not lead to robot instructions that will be applicable in the general

Conclusion

In this article we presented an approach for teaching a robot new problem solution strategies. The tasks are demonstrated by humans in a training center which is equipped with several sensor aid systems to acquire as much information as needed to learn the performed tasks. Typical human actions were analyzed and a framework called Programming by Demonstration was implemented to segment and recognize these demonstrations. Macro-operators are a generalized structure to represent tasks as sets of

Acknowledgements

This work has been supported by the German collaborative research center ‘SFB Humanoid Robots’ granted by Deutsche Forschungsgemeinschaft (see http://www.sfb588.uni-karlsruhe.de).

Prof. Dr.-Ing. Rüdiger Dillmann received a Ph.D. at the University of Karlsruhe in 1980. Since 1987 he is Professor of the Department of Computer Science and since 2001 director of the research group, Industrial Applications of Informatics and Microsystems (IAIM) at the University of Karlsruhe. Since 2002 he is also director of the Research Center for Information Science (FZI), Karlsruhe. As a leader of these two institutes, Professor Dillmann supervises several research groups in the areas of

References (21)

  • J Zhang et al.

    Interactive assembly by a two-arm-robot agent

    Robotics and Autonomous Systems

    (1999)
  • A Alissandrakis et al.

    Imitation with ALICE: Learning to imitate corresponding actions across dissimilar embodiments

    IEEE Transactions on Systems, Man and Cybernetics

    (2002)
  • J. Anderson, Kognitive Psychologie. 2. Auflage, Spektrum der Wissenschaft Verlagsgesellschaft mbH, Heidelberg,...
  • C. Archibald, E. Petriu, Computational paradigm for creating and executing sensor-based robot skills, in: Proceedings...
  • R. Dillmann, O. Rogalla, M. Ehrenmann, R. Zöllner, M. Bordegoni, Learning robot behaviour and skills based on human...
  • R. Dillmann, R. Zöllner, M. Ehrenmann, O. Rogalla, Interactive natural programming of robots: introductory overview,...
  • M. Ehrenmann, R. Zöllner, O. Rogalla, S. Vacek, R. Dillmann, Programming service tasks in household environments by...
  • M. Ehrenmann, R. Zöllner, O. Rogalla, S. Vacek, R. Dillmann, Observation in programming by demonstration: training and...
  • H. Friedrich, Interaktive Programmierung von Manipulationssequenzen, Ph.D. Thesis, Universitat Karlsruhe,...
  • Y Kuniyoshi et al.

    Learning by watching: extracting reusable task knowledge from visual observation of human performance

    IEEE Transactions on Robotics and Automation

    (1994)
There are more references available in the full text version of this article.

Cited by (253)

  • The state of the art of search strategies in robotic assembly

    2022, Journal of Industrial Information Integration
    Citation Excerpt :

    Similarly, Nemec et al. [14] propose a learning strategy and an adaptive admittance control algorithm based on the dynamic motion primitive (DMP) framework, which is used as the bottom representation of the search trajectory derived from the PiH assembly search. At the same time, the algorithm uses Cartesian space trajectory to match the force and torque recorded in the LbD process rather than the position and orientation [103,104]. In addition, an iterative adjustment learning of search trajectory based on the characteristics of DMPs is proposed to improve task performance.

  • Emerging Frontiers in Human–Robot Interaction

    2024, Journal of Intelligent and Robotic Systems: Theory and Applications
  • Learning Skills From Demonstrations: A Trend From Motion Primitives to Experience Abstraction

    2024, IEEE Transactions on Cognitive and Developmental Systems
  • Teaching Assembly Tasks to Robots Using a 3D Simulation Environment

    2023, ACM International Conference Proceeding Series
View all citing articles on Scopus

Prof. Dr.-Ing. Rüdiger Dillmann received a Ph.D. at the University of Karlsruhe in 1980. Since 1987 he is Professor of the Department of Computer Science and since 2001 director of the research group, Industrial Applications of Informatics and Microsystems (IAIM) at the University of Karlsruhe. Since 2002 he is also director of the Research Center for Information Science (FZI), Karlsruhe. As a leader of these two institutes, Professor Dillmann supervises several research groups in the areas of robotics with special interest in intelligent, autonomous and mobile robotics, machine learning, machine vision, man-machine interaction, computer science in medicine and simulation techniques. Professor Dillmann is author or co-author of more than 100 scientific publications and several books. He is involved in many synergistic activities such as: Director of the German collaborative research center ‘Humanoid Robots’, IEEE-RAS chairman of the German chapter, Chairman of the German Society of Information Science (GI), section 4.3/1.4 ‘Robotic Systems’, Chairman of the German Association of Engineers (VDI-GMA), and Editor-in-Chief for the journal Robotics and Autonomous Systems, Elsevier.

View full text