Skip to main content
SpringerLink
Log in

Gesture-Based Extraction of Robot Skill Parameters for Intuitive Robot Programming

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Despite a lot of research in the field, only very little experience exists with Teaching by Demonstration (TbD) in actual industrial use cases. In the factory of the future, it is necessary to rapidly reprogram flexible mobile manipulators to perform new tasks, when the need arises, for which a working system capable of TbD would be ideal. Contrary to current TbD approaches, that generally aim to recognize both action and where it is applied, we propose a division of labor, where the operator manually specifies the action the robot should perform, while gestures are used for specifying the relevant action parameter (e.g. on which object to apply the action). Using this two-step method has the advantages that there is no uncertainty of which action the robot will perform, it takes into account that the environment changes, so objects do not need to be at predefined locations, and the parameter specification is possible even for inexperienced users. Experiments with 24 people in 3 different environments verify that it is indeed intuitive, even for a robotics novice, to program a mobile manipulator using this method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Factory-in-a-Day project (2014). http://www.factory-in-a-day.seu/

  2. STAMINA project (2014). http://stamina-robot.eu

  3. TAPAS project (2014). http://tapas-project.eu

  4. Archibald, C.C.: A computational model for skills-oriented robot programming. PhD thesis, University of Ottawa, Ottawa (1995)

    Google Scholar 

  5. Bøgh, S., Nielsen, O.S., Pedersen, M.R., Krüger, V., Madsen, O.: Does your robot have skills? In: Proceedings of the 43rd International Symposium on Robotics (ISR), Taipei (2012)

  6. Bischoff, R., Kazi, A.: Perspectives on augmented reality based human-robot interaction with industrial robots. In: Intelligent Robots and Systems, 2004.(IROS 2004). Proceedings. 2004 IEEE/RSJ International Conference on, vol. 4, pp. 3226–3231 (2004)

  7. Björkelund, A., Edstrom, L., Haage, M., Malec, J., Nilsson, K., Nugues, P., Robertz, S., Storkle, D., Blomdell, A., Johansson, R., Linderoth, M., Nilsson, A., Robertsson, A., Stolt, A., Bruyninckx, H.: On the integration of skilled robot motions for productivity in manufacturing. In: 2011 IEEE International Symposium on Assembly and Manufacturing (ISAM). doi:10.1109/ISAM.2011.5942366 (2011)

  8. Bobick, A.: On human action. In: Moeslund, T.B., Hilton, A., Krüger, V., Sigal, L. (eds.) Visual analysis of humans, pp. 279–288. Springer, London (2011)

  9. Bobick, A.F.: Movement, activity and action: the role of knowledge in the perception of motion. Phil. Trans. R. Soc. B Biol. Sci. 352(1358), 1257–1265 (1997). PMID: 9304692 PMCID: 1692010

    Article  Google Scholar 

  10. Bruyninckx, H., De Schutter, J.: Specification of force-controlled actions in the “task frame formalism”-a synthesis. IEEE Trans. Robot. Autom. 12(4), 581–589 (1996). doi:10.1109/70.508440

    Article  Google Scholar 

  11. Chen, H., Sheng, W.: Transformative CAD based industrial robot program generation. Robot. Comput. Integr. Manuf. 27(5), 942–948 (2011). doi:10.1016/j.rcim.2011.03.006

    Article  Google Scholar 

  12. Ekvall, S., Kragic, D.: Robot learning from demonstration: a task-level planning approach. Int. J. Adv. Robot. Syst. 5(3), 223–234 (2008)

    Google Scholar 

  13. Finkemeyer, B., Kröger, T., Wahl, F.M.: Executing assembly tasks specified by manipulation primitive nets. Adv. Robot. 19(5), 591–611 (2005)

    Article  Google Scholar 

  14. Gadensgaard, D., Bourne, D.: Human/Robot multi-initiative setups for assembly cells. In: ICAS 2011, The Seventh International Conference on Autonomic and Autonomous Systems, pp. 1–6 (2011)

  15. Galindo, C., Fernández-Madrigal, J.A., Gonzílez, J., Saffiotti, A.: Robot task planning using semantic maps. Robot. Auton. Syst. 56(11), 955–966 (2008). doi:10.1016/j.robot.2008.08.007

    Article  Google Scholar 

  16. Gottschlich, S., Ramos, C., Lyons, D.: Assembly and task planning: a taxonomy. IEEE Robot. Autom. Mag. 1(3), 4–12 (1994)

    Article  Google Scholar 

  17. Hartmann, M.: DYNAPRO: Erfolgreich produzieren in turbulenten Märkten. Logis Verlag (1996)

  18. Hartmann, M.: DYNAPRO II: erfolgreich produzieren in turbulenten Märkten. Logis Verlag (1997)

  19. Hartmann, M.: DYNAPRO III: Erfolgreich produzieren in turbulenten Märkten. Logis Verlag (1998)

  20. Høilund, C., Krüger, V., Moeslund, T.: Evaluation of human body tracking system for gesture-based programming of industrial robots. In: Proceedings of the 2012 7th IEEE Conference on Industrial Electronics and Applications (ICIEA), pp. 477–480. IEEE (2012)

  21. Huckaby, J., Vassos, S., Christensen, H.I.: Planning with a task modeling framework in manufacturing robotics. In: 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 5787–5794 (2013)

  22. Hvilshøj, M., Bøgh, S., Nielsen, O.S., Madsen, O.: Autonomous industrial mobile manipulation (AIMM): past, present and future. Ind. Robot. Int. J. 39(2), 120–135 (2012). doi:10.1108/01439911211201582

    Article  Google Scholar 

  23. Inamura, T., Toshima, I., Tanie, H.: Embodied symbol emergence based on mimesis theory, vol. 23, pp. 4–5 (2004). http://citeseer.ist.psu.edu/viewdoc/summary?doi=10.1.1.100.7250

  24. Jen, Y.H., Taha, Z., Vui, L.J.: VR-Based robot programming and simulation system for an industrial robot. Int. J. Ind. Eng. Theory Appl. Pract. 15(3), 314–322 (2008)

    Google Scholar 

  25. Kang, S.B., Ikeuchi, K.: Toward automatic robot instruction from perception-temporal segmentation of tasks from human hand motion. IEEE Trans. Robot. Autom. 11(5), 670–681 (1995). doi:10.1109/70.466599

    Article  Google Scholar 

  26. Kang, S.B., Ikeuchi, K.: Toward automatic robot instruction from perception-mapping human grasps to manipulator grasps. IEEE Trans. Robot. Autom. 13(1), 81–95 (1997). doi:10.1109/70.554349

    Article  Google Scholar 

  27. Krüger, N., Piater, J., Wörgötter, F., Geib, C., Petrick, R., Steedman, M., Ude, A., Asfour, T., Kraft, D., Omrcen, D., et al: A formal definition of object-action complexes and examples at different levels of the processing hierarchy (2009). Technical report. http://www.paco-plus.org

  28. Kröger, T., Finkemeyer, B., Wahl, F.: Manipulation primitives — a universal interface between sensor-based motion control and robot programming. In: Schütz, D., Wahl, F. (eds.) Robotic systems for handling and assembly, Springer Tracts in Advanced Robotics, vol. 67, pp. 293–313. Springer, Berlin (2011)

  29. Kröger, T., Finkemeyer, B., Winkelbach, S., Eble, L.O., Molkenstruck, S., Wahl, F.: A manipulator plays jenga. IEEE Robot. Autom. Mag. 15(3), 79–84 (2008). doi:10.1109/MRA.2008.921547

    Article  Google Scholar 

  30. Krüger, V., Herzog, D., Baby, S., Ude, A., Kragic, D.: Learning actions from observations. IEEE Robot. Autom. Mag. 17(2), 30–43 (2010). doi:10.1109/MRA.2010.936961

    Article  Google Scholar 

  31. Krüger, V., Kragic, D., Ude, A., Geib, C.: The meaning of action: a review on action recognition and mapping. Adv. Robot. 21(13), 1473–1501 (2007)

    Google Scholar 

  32. Kulic, D., Kragic, D.: Learning action primitives. In: Moeslund, T.B., Hilton, A., Krüger V., Sigal, L. (eds.) Visual analysis of humans, pp. 333–354. Springer, London (2011)

  33. Lopes, M., Santos-Victor, J.: A developmental roadmap for learning by imitation in robots. IEEE Trans. Syst. Man Cybern. B (Cybernetics) 37, 308–321 (2007). doi:10.1109/TSMCB.2006.886949

    Article  Google Scholar 

  34. Madsen, O., Bøgh, S., Schou, C., Andersen, R., Damgaard, J., Pedersen, M.R., Krüger, V.: Integration of two autonomous mobile manipulators in a real-world industrial setting. In: IROS Workshop on Robotic Assistance Technologies in Industrial Settings (RATIS). Tokyo (2013)

  35. Mitsi, S., Bouzakis, K.D., Mansour, G., Sagris, D., Maliaris, G.: Off-line programming of an industrial robot for manufacturing. Int. J. Adv. Manuf. Technol. 26(3), 262–267 (2005). doi:10.1007/s00170-003-1728-5

    Article  Google Scholar 

  36. Neto, P., Mendes, N.: Direct off-line robot programming via a common CAD package. Robot. Auton. Syst. 61(8), 896–910 (2013). doi:10.1016/j.robot.2013.02.005

    Article  Google Scholar 

  37. Pedersen, M.R., Herzog, D., Krüger, V.: Intuitive skill-level programming of industrial handling tasks on a mobile manipulator. 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2014)

  38. Pedersen, M.R., Nalpantidis, L., Bobick, A., Krüger, V.: On the integration of hardware-abstracted robot skills for use in industrial scenarios. In: 2nd International IROS workshop on cognitive robotics systems: replicating human actions and activities. Tokyo (2013). http://renaud-detry.net/events/crs2013/papers/Pedersen.pdf

  39. Takashi, I.A.: Towards an assembly plan from observation: task recognition with polyhedral objects. IEEE Trans. Robot. Autom. 10(3), 368–385 (1994) http://citeseerx.ist.psu.edu/viewdoc/summary?doi= 10.1.1.12.8697

    Article  Google Scholar 

  40. Tenorth, M., Perzylo, A., Lafrenz, R., Beetz, M.: Representation and exchange of knowledge about actions, objects, and environments in the RoboEarth framework. IEEE Trans. Autom. Sci. Eng. 10 (3), 643–651 (2013). doi:10.1109/TASE.2013.2244883

    Article  Google Scholar 

  41. Waibel, M., Beetz, M., Civera, J., D’Andrea, R., Elfring, J., Galvez-Lopez, D., Haussermann, K., Janssen, R., Montiel, J.M.M., Perzylo, A., Schiessle, B., Tenorth, M., Zweigle, O., van de Molengraft, R.: RoboEarth. IEEE Robot. Autom. Mag. 18(2), 69–82 (2011). doi:10.1109/MRA.2011.941632

    Article  Google Scholar 

  42. Yeasin, M., Chaudhuri, S.: Toward automatic robot programming: learning human skill from visual data. IEEE Trans. Syst. Man Cybern. B Cybern. 30(1), 180–185 (2000)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Robotics, Vision and Machine Intelligence (RVMI) Lab, Department of Mechanical and Manufacturing Engineering, Aalborg University Copenhagen, AC Meyers Vaenge 15, 2450, Copenhagen, Denmark

    Mikkel Rath Pedersen & Volker Krüger

Authors
  1. Mikkel Rath Pedersen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Volker Krüger
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mikkel Rath Pedersen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Pedersen, M.R., Krüger, V. Gesture-Based Extraction of Robot Skill Parameters for Intuitive Robot Programming. J Intell Robot Syst 80 (Suppl 1), 149–163 (2015). https://doi.org/10.1007/s10846-015-0219-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-015-0219-x

Keywords

Search

Navigation