Skip to main content
SpringerLink
Log in

Using probabilistic movement primitives in robotics

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Movement Primitives are a well-established paradigm for modular movement representation and generation. They provide a data-driven representation of movements and support generalization to novel situations, temporal modulation, sequencing of primitives and controllers for executing the primitive on physical systems. However, while many MP frameworks exhibit some of these properties, there is a need for a unified framework that implements all of them in a principled way. In this paper, we show that this goal can be achieved by using a probabilistic representation. Our approach models trajectory distributions learned from stochastic movements. Probabilistic operations, such as conditioning can be used to achieve generalization to novel situations or to combine and blend movements in a principled way. We derive a stochastic feedback controller that reproduces the encoded variability of the movement and the coupling of the degrees of freedom of the robot. We evaluate and compare our approach on several simulated and real robot scenarios.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

Notes

  1. http://www.ausy.tu-darmstadt.de/uploads/Team/AlexandrosParaschos/ProMP_toolbox.zip.

  2. This prior variance profile can be just set to \(\alpha \varvec{I}\), where \(\alpha \) is a small constant and \(\varvec{I}\) is the identity matrix.

  3. The third derivative of \(\varvec{\Psi }\) can be computed numerically.

  4. If inverse dynamics control (Peters et al. 2008) is used for the robot, the system reduces to a linear system where the terms \(\varvec{A}_{t}\), \(\varvec{B}_{t}\) and \(\varvec{c}_{t}\) are constant in time.

  5. As we multiply the noise by \(\varvec{B}{{\mathrm{dt}}}\), we need to divide the covariance \(\varvec{\Sigma }_{u}\) of the control noise \(\varvec{\epsilon }_{u}\) by \({{\mathrm{dt}}}\) to obtain this desired behavior.

  6. The observation noise is omitted as it represents independent noise which is not used for predicting the next state.

References

  • Bruno, D., Calinon, S., Malekzadeh, M. S., & Caldwell, D. G. (2015). Learning the stiffness of a continuous soft manipulator from multiple demonstrations. In Intelligent robotics and applications (pp. 185–195).

  • Buchli, J., Stulp, F., Theodorou, E., & Schaal, S. (2011). Learning variable impedance control. International Journal of Robotics Research, 30(7), 820–833.

    Article  Google Scholar 

  • Calinon, S. (2016). A tutorial on task-parameterized movement learning and retrieval. Intelligent Service Robotics, 9(1), 1–29.

    Article  Google Scholar 

  • Calinon, S., D’Halluin, F., Sauser, E. L., Caldwell, D. G., & Billard, A. G. (2010). Learning and reproduction of gestures by imitation. IEEE Robotics and Automation Magazine, 17, 44–54.

    Article  Google Scholar 

  • Calinon, S., Sardellitti, I., & Caldwell, D. G. (2010b). Learning-based control strategy for safe human–robot interaction exploiting task and robot redundancies. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 249–254).

  • Daniel, C., Neumann, G., & Peters, J. (2012). Learning concurrent motor skills in versatile solution spaces. In IEEE/RSJ international conference on intelligent robots and systems (IROS), (pp. 3591–3597).

  • da Silva, B., Konidaris, G., & Barto, A. (2012). Learning parameterized skills. In International conference on machine learning (pp. 1679–1686).

  • dAvella, A., & Bizzi, E. (2005). Shared and specific muscle synergies in natural motor behaviors. Proceedings of the National Academy of Sciences (PNAS), 102(3), 3076–3081.

    Article  Google Scholar 

  • Degallier, S., Righetti, L., Gay, S., & Ijspeert, A. (2011). Toward simple control for complex, autonomous robotic applications: Combining discrete and rhythmic motor primitives. Autonomous Robots, 31, 155–181.

    Article  Google Scholar 

  • Dominici, N., Ivanenko, Y. P., Cappellini, G., dAvella, A., Mondì, V., Cicchese, M., et al. (2011). Locomotor primitives in newborn babies and their development. Science, 334(6058), 997–999.

    Article  Google Scholar 

  • Ernesti, J., Righetti, L., Do, M., Asfour, T., & Schaal, S. (2012). Encoding of periodic and their transient motions by a single dynamic movement primitive. In IEEE-RAS international conference on humanoid robots (humanoids) (pp. 57–64).

  • Ewerton, M., Maeda, G., Peters, J., & Neumann, G. (2015). Learning motor skills from partially observed movements executed at different speeds. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 456–463).

  • Forte, D., Gams, A., Morimoto, J., & Ude, A. (2012). On-line motion synthesis and adaptation using a trajectory database. Robotics and Autonomous Systems, 60, 1327–1339.

    Article  Google Scholar 

  • Gams, A., Nemec, B., Ijspeert, A. J., & Ude, A. (2014). Coupling movement primitives: Interaction with the environment and bimanual tasks. IEEE Transactions on Robotics, 30(4), 816–830.

    Article  Google Scholar 

  • Higham, N. J. (1988). Computing a nearest symmetric positive semidefinite matrix. Linear Algebra and its Applications, 103, 103–118.

    Article  MathSciNet  MATH  Google Scholar 

  • Ijspeert, A. J. (2008). Central pattern generators for locomotion control in animals and robots: A review. Neural Networks, 21(4), 642–653.

    Article  Google Scholar 

  • Ijspeert, A. J., Nakanishi, J., Hoffmann, H., Pastor, P., & Schaal, S. (2013). Dynamical movement primitives: Learning attractor models for motor behaviors. Neural Computation, 25(2), 328–373.

    Article  MathSciNet  MATH  Google Scholar 

  • Ijspeert, A. J., Nakanishi, J., & Schaal, S. (2003). Learning attractor landscapes for learning motor primitives. In Advances in neural information processing systems (NIPS) (pp. 1547–1554).

  • Khansari-Zadeh, S. M., & Billard, A. (2011). Learning stable nonlinear dynamical systems with Gaussian mixture models. IEEE Transactions on Robotics, 27(5), 943–957.

    Article  Google Scholar 

  • Khansari-Zadeh, S. M., Kronander, K., & Billard, A. (2014). Modeling robot discrete movements with state-varying stiffness and damping: A framework for integrated motion generation and impedance control. In Robotics science and systems (R:SS).

  • Klug, S., Lens, T., von Stryk, O., Möhl, B., & Karguth, A. (2008). Biologically inspired robot manipulator for new applications in automation engineering. In Proceedings of robotik.

  • Kober, J., Muelling, K., Kroemer, O., Lampert, C. H., Scholkopf, B., & Peters, J. (2010). Movement templates for learning of hitting and batting. In International conference on robotics and automation (ICRA) (pp. 853–858).

  • Konidaris, G., Kuindersma, S., Grupen, R., & Barto, A. (2012). Robot learning from demonstration by constructing skill trees. International Journal of Robotics Research (IJRR), 31(3), 360–375.

    Article  Google Scholar 

  • Kormushev, P., Calinon, S., & Caldwell, D. G. (2010). Robot motor skill coordination with EM-based reinforcement learning. In International conference on intelligent robots and systems (IROS) (pp. 3232–3237).

  • Kulvicius, T., Ning, K., Tamosiunaite, M., & Worgotter, F. (2012). Joining movement sequences: Modified dynamic movement primitives for robotics applications exemplified on handwriting. IEEE Transactions on Robotics, 28(1), 145–157.

    Article  Google Scholar 

  • Lazaric, A., & Ghavamzadeh, M. (2010). Bayesian multi-task reinforcement learning. In International conference on machine learning (ICML) (pp. 599–606).

  • Li, W., & Todorov, E. (2010). Iterative linear quadratic regulator design for nonlinear biological movement systems. In International conference on informatics in control, automation and robotics (ICINCO) (pp. 222–229).

  • Maeda, G., Ewerton, M., Lioutikov, R., Amor, H., Peters, J., & Neumann, G. (2014). Learning interaction for collaborative tasks with probabilistic movement primitives. In International conference on humanoid robots (Humanoids) (pp. 527–534).

  • Matsubara, T., Hyon, S. H., & Morimoto, J. (2011). Learning parametric dynamic movement primitives from multiple demonstrations. Neural Networks, 24(5), 493–500.

    Article  Google Scholar 

  • Moro, F. L., Tsagarakis, N. G., & Caldwell, D. G. (2012). On the kinematic motion primitives (kMPs)—Theory and application. Frontiers in Neurorobotics, 6(10), 1–18.

    Google Scholar 

  • Muelling, K., Kober, J., & Peters, J. (2011). A biomimetic approach to robot table tennis. Adaptive Behavior Journal, 19(5), 359–376.

    Article  Google Scholar 

  • Mülling, K., Kober, J., Kroemer, O., & Peters, J. (2013). Learning to select and generalize striking movements in robot table tennis. The International Journal of Robotics Research, 32(3), 263–279.

    Article  Google Scholar 

  • Nakanishi, J., Morimoto, J., Endo, G., Cheng, G., Schaal, S., & Kawato, M. (2004). Learning from demonstration and adaptation of biped locomotion. Robotics and Autonomous Systems, 47, 79–91.

    Article  Google Scholar 

  • Neumann, G., Daniel, C., Paraschos, A., Kupcsik, A., & Peters, J. (2014). Learning modular policies for robotics. Frontiers in Computational Neuroscience, 8(62), 1.

    Google Scholar 

  • Neumann, G., Maass, W., & Peters, J. (2009). Learning complex motions by sequencing simpler motion templates. In International conference on machine learning (ICML) (pp. 753–760)

  • OHagan, A., & Forster, J. (2004). Kendalls advanced theory of statistics: Bayesian inference (2nd ed.). Arnold, New York. Technical report, ISBN 0-340-80752-0.

  • Paraschos, A., Daniel, C., Peters, J., & Neumann, G. (2013a). Probabilistic movement primitives. In Advances in neural information processing systems (NIPS) (pp. 2616–2624).

  • Paraschos, A., Neumann, G., & Peters, J. (2013b). A probabilistic approach to robot trajectory generation. In International conference on humanoid robots (humanoids) (pp. 477–483)

  • Pastor, P., Hoffmann, H., Asfour, T., & Schaal, S. (2009). Learning and generalization of motor skills by learning from demonstration. In International conference on robotics and automation (ICRA) (pp. 763–768)

  • Pastor, P., Righetti, L., Kalakrishnan, M., & Schaal, S. (2011). Online movement adaptation based on previous sensor experiences. In International conference on intelligent robots and systems (IROS) (pp. 365–371)

  • Peters, J., Mistry, M., Udwadia, F. E., Nakanishi, J., & Schaal, S. (2008). A unifying methodology for robot control with redundant DOFs. Autonomous Robots, 24(1), 1–12.

    Article  Google Scholar 

  • Righetti, L., & Ijspeert, A. J. (2006). Programmable central pattern generators: An application to biped locomotion control. In International conference on robotics and automation, (ICRA) (pp. 1585–1590).

  • Rozo, L., Calinon, S., Caldwell, D., Jiménez, P., & Torras, C. (2013). Learning collaborative impedance-based robot behaviors. In AAAI conference on artificial intelligence (pp. 1422–1428).

  • Rückert, E. A., Neumann, G., Toussaint, M., & Maass, W. (2012). Learned graphical models for probabilistic planning provide a new class of movement primitives. Frontiers in Computational Neuroscience, 6(97), 1.

    Google Scholar 

  • Rueckert, E., Mundo, J., Paraschos, A., Peters, J., & Neumann, G. (2015). Extracting low-dimensional control variables for movement primitives. In International conference on robotics and automation (ICRA) (pp. 1511–1518).

  • Schaal, S., Mohajerian, P., & Ijspeert, A. (2007). Dynamics systems vs. optimal control—A unifying view. Computational Neuroscience: Theoretical Insights into Brain Function, 165, 425–445.

    Google Scholar 

  • Schaal, S., Peters, J., Nakanishi, J., & Ijspeert, A. (2005). Learning movement primitives. In International symposium on robotics research (pp. 561–572).

  • Stark, H., & Woods, J. (2001). Probability and random processes with applications to signal processing (3rd ed.). Upper Saddle River: Prentice-Hall.

    Google Scholar 

  • Stengel, R. F. (2012). Optimal control and estimation. North Chelmsford, MA: Courier Corporation.

    MATH  Google Scholar 

  • Todorov, E. (2008). General duality between optimal control and estimation. Conference on Decision and Control, 5, 4286–4292.

    Google Scholar 

  • Todorov, E., & Jordan, M. (2002). Optimal feedback control as a theory of motor coordination. Nature Neuroscience, 5, 1226–1235.

    Article  Google Scholar 

  • Toussaint, M. (2009). Robot trajectory optimization using approximate inference. In International conference on machine learning (ICML) (pp. 1049–1056).

  • Ude, A., Gams, A., Asfour, T., & Morimoto, J. (2010). Task-specific generalization of discrete and periodic dynamic movement primitives. Transactions in Robotics, 5, 800–815.

    Article  Google Scholar 

  • Williams B., Toussaint, M., & Storkey, A. (2007). Modelling motion primitives and their timing in biologically executed movements. In Advances in neural information processing systems (NIPS) (pp. 1609–1616).

Download references

Acknowledgements

The research leading to these results has received funding from the European Community’s Framework Programme CoDyCo (FP7-ICT-2011-9 Grant No. 600716), CompLACS (FP7-ICT-2009-6 Grant No. 270327), GeRT (FP7-ICT-2009-4 Grant No. 248273), and ERC StG SKILLS4ROBOTS.

Author information

Authors and Affiliations

  1. Technische Universität Darmstadt, Hochschulstrasse 10, 64289, Darmstadt, Germany

    Alexandros Paraschos & Jan Peters

  2. Bosch Center for Artificial Intelligence, Robert-Bosch-Campus, 71272, Renningen, Germany

    Christian Daniel

  3. Max-Planck-Institut für Intelligente Systeme, Spemannstrasse 38, 72076, Tübingen, Germany

    Jan Peters

  4. Computational Learning for Autonomous Systems, School of Computer Science, University of Lincoln, Brayford Pool, LN6 7TS, Lincoln, UK

    Gerhard Neumann

Authors
  1. Alexandros Paraschos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Christian Daniel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jan Peters
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gerhard Neumann
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexandros Paraschos.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Paraschos, A., Daniel, C., Peters, J. et al. Using probabilistic movement primitives in robotics. Auton Robot 42, 529–551 (2018). https://doi.org/10.1007/s10514-017-9648-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-017-9648-7

Keywords

Search

Navigation