Skip to main content
SpringerLink
Log in

Transferring the semantic constraints in human manipulation behaviors to robots

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

In this study, we aim to help robots manipulate objects with the guidance of semantic constraints (the grasp location, grasp type, approaching way, trajectory constraint, grasp force, and opening width) which are learnt from human manipulation behaviors. In order to transfer the complex and uncertain relationships between the attributes of object and task and semantic constraints in human behaviors to robots. We propose a representation method of human behaviors in machine-understandable semantics and a collaborative reasoning mechanism. With the suggested semantic constraints from human behaviors, the robot manipulation can be completed under “consciousness” and proper for both the object and task. The shareability of the object attributes, primitive actions and semantic constraints makes the proposed method be generalized to new objects and even new tasks. Additionally, we find that the object’s parts can be grasped sequentially, according to the object’s state and the action to be performed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Alissandrakis A, Nehaniv C, Dautenhahn K (2002) Imitation with alice: learning to imitate corresponding actions across dissimilar embodiments. IEEE T Syst Man Cy-S Humans 32(4):482–496. https://doi.org/10.1109/TSMCA.2002.804820

    Article  Google Scholar 

  2. Nehaniv CL, Dautenhahn K (2007) Imitation and social learning in robots, humans and animals: behavioural, social and communicative dimensions. Cambridge Univ. Press. https://doi.org/10.1017/CBO9780511489808

  3. Ramirez-Amaro K, Beetz M, Cheng G (2015) Transferring skills to humanoid robots by extracting semantic representations from observations of human activities. Artif Intell 247:95–118. https://doi.org/10.1016/j.artint.2015.08.009

    Article  MathSciNet  MATH  Google Scholar 

  4. Cheng G, Ramirez-Amaro K, Beetz M (2019) Kuniyoshi Purposive Learning: robot reasoning about the meanings of human activities. Sci Robot 4(26), https://doi.org/10.1126/scirobotics.aav1530

  5. Katz G, Huang DW, Hauge T, Gentili R, Reggia J (2018) A novel parsimonious cause-effect reasoning algorithm for robot imitation and plan recognition. IEEE T Cogn Dev Syst 10(2):177–193. https://doi.org/10.1109/TCDS.2017.2651643

    Article  Google Scholar 

  6. Song D, Ek CH, Huebner K, Kragic D (2015) Task-based robot grasp planning using probabilistic inference. IEEE T Robot 31(3):546–561. https://doi.org/10.1109/TRO.2015.2409912

    Article  Google Scholar 

  7. Jain S, Argall B (2018) Recursive bayesian human intent recognition in shared-control robotics. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 1–5. https://doi.org/10.1109/IROS.2018.8593766

  8. Carvalho RN, Laskey KB, Costa PCG (2017) PR-OWL-a language for defining probabilistic ontologies. Int J Approx Reason 91:56–79. https://doi.org/10.1016/j.ijar.2017.08.011

    Article  MathSciNet  MATH  Google Scholar 

  9. Feix T, Bullock IM, Dollar AM (2014) Analysis of human grasping behavior: object characteristics and grasp type. IEEE T Haptics 7(3):311–323. https://doi.org/10.1109/TOH.2014.2326871

    Article  Google Scholar 

  10. Feix T, Bullock IM, Dollar AM (2014) Analysis of human grasping behavior: correlating tasks, objects and grasps. IEEE T Haptics 7(4):430–441. https://doi.org/10.1109/TOH.2014.2326867

    Article  Google Scholar 

  11. Klatzky RL, McCloskey B, Doherty S, Pellegrino J, Smith T (1987) Knowledge about hand shaping and knowledge about objects. J Motor Behav 19:187–213. https://doi.org/10.1080/00222895.1987.10735407

    Article  Google Scholar 

  12. Bullock IM, Zheng JZ, Rosa SDL, Guertler C, Dollar M (2013) Grasp frequency and usage in daily household and machine shop tasks. IEEE T Haptics 6(3):296–308. https://doi.org/10.1109/TOH.2013.6

    Article  Google Scholar 

  13. Dang H, Allen PK (2012) Semantic grasping: planning robotic grasps functionally suitable for an object manipulation task. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp 1311–1317. https://doi.org/10.1109/IROS.2012.6385563

  14. Antanas L, Moreno P, Neumann M, de Figueiredo RP, Kersting K, Santos-Victor J, Raedt L (2019) Semantic and geometric reasoning for robotic grasping: a probabilistic logic approach. Auton Robot 43 (6):1393–1418. https://doi.org/10.1007/s10514-018-9784-8

    Article  Google Scholar 

  15. Wu H, Wu X, Ma Q, Tian G (2019) Cloud robot: semantic map building for intelligent service task. Appl Intell 49(2):319–334. https://doi.org/10.1007/s10489-018-1277-0

    Article  Google Scholar 

  16. Dourlens S, Ramdane-Cherif A, Monacelli E (2013) Multi levels semantic architecture for multimodal interaction. Appl Intell 38(4):586–599. https://doi.org/10.1007/s10489-012-0387-3

    Article  Google Scholar 

  17. Ahmed T, Srivastava A (2017) An automated approach to estimate human interest. Appl Intell 47 (4):1186–1207. https://doi.org/10.1007/s10489-017-0947-7

    Article  Google Scholar 

  18. Min HJ, Cho SB (2010) . Adaptive Behaviors of Reactive Mobile Robot with Bayesian inference in nonstationary environments 33(3):264–277. https://doi.org/10.1007/s10489-009-0164-0

    Article  Google Scholar 

  19. Carvalho RN, Matsumoto S, Laskey KB, Costa PCG, Ladeira M, Santos LL (2013) Probabilistic ontology and knowledge fusion for procurement fraud detection in brazil. https://doi.org/10.1007/978-3-642-35975-0_2

  20. Carvalho RN, Haberlin R, Costa PCG, Laskey KB, Chang K (2011) Modeling a probabilistic ontology for maritime domain awareness. In: 14th International Conference on Information Fusion, pp 1–8

  21. Farhadi A, Endres I, Hoiem D, Forsyth D (2009) Describing objects by their attributes. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 1778–1785. https://doi.org/10.1109/CVPR.2009.5206772

  22. Yuke Z, Alireza F, Li FF (2014) Reasoning about object affordances in a knowledge base representation. In: 2014 European Conference on Computer Vision (ECCV), pp 408–424. https://doi.org/10.1007/978-3-319-10605-2_27

  23. Castellano G, Fanelli AM, Sforza G, et al. (2016) Shape annotation for intelligent image retrieval. Appl Intell 44(1):179–195. https://doi.org/10.1007/s10489-015-0693-7

    Article  Google Scholar 

  24. Kamakura N, Matsuo M, Ishii H, Mitsuboshi F, Miura Y (1980) Patterns of static prehension in normal hands. AM J Occup Ther 34:437–445. https://doi.org/10.5014/ajot.34.7.437

    Article  Google Scholar 

  25. Hesse C, Deubel H (2010) Advance planning in sequential pick-and-place tasks. J Neurophysiol 104 (1):508–516. https://doi.org/10.1152/jn.00097.2010

    Article  Google Scholar 

  26. Cai M, Kitani KM, Sato Y (2018) Understanding hand-object manipulation by modeling the contextual relationship between actions, grasp types and object attributes. arXiv:1807.08254v1

  27. Gupta K, Burschka D, Bhavsar A (2018) Classifying object manipulation actions based on grasp-types and motion-constraints. arXiv:1806.07574

  28. Motik B, Grau BC, Horrocks I, Wu Z, Fokoue A, Lutz C (2012) OWL 2 web ontology language profiles (second edition). https://www.w3.org/TR/owl2-profiles/

  29. Goldstein EB (2015) Cognitive Psychology: Connecting Mind, Research and Everyday Experience, 4th edn. Cengage Learning, Stamford

    Google Scholar 

  30. Erol K, Hendler J, Nau D (1994) HTN planning: complexity and expressivity. In: 12th National Conference on Artificial Intelligence, pp 1123–1128

  31. Carvalho RN, Ladeira M, Santos L, Matsumoto S, Costa PCG (2007) UnBBayes-MEBN: comments on implementing a probabilistic ontology tool. In: International Conference-applied Computing, pp 211–218

  32. Matsumoto S, Carvalho RN, Costa PCG, Laskey KB, Santos LL, Ladeira M (2011) There’s no more need to be a night OWL: on the PR-OWL for a MEBN tool before nightfall In: Introduction to the Semantic Web: Concepts, Technologies and Applications, iConcept press

  33. Horrocks I, Patel-Schneider PF, Boley H, Tabet S, Grosof B, Dean M (2004) SWRL: a semantic web rule language combining OWL and RuleML. https://www.w3.org/Submission/SWRL/

  34. Harris S, Seaborne A (2013) SPARQL 1.1 query language. https://www.w3.org/TR/sparql11-query/

Download references

Acknowledgements

This work is supported by the Key Program for Joint Funds of the National Natural Science Foundation of China under Grant U1813215, and the National Natural Science Foundation of China under Grant 61773239.

Author information

Authors and Affiliations

  1. School of Control Science and Engineering, Shandong University, Jinan, China

    Cici Li & Guohui Tian

Authors
  1. Cici Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Guohui Tian
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guohui Tian.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Li, C., Tian, G. Transferring the semantic constraints in human manipulation behaviors to robots. Appl Intell 50, 1711–1724 (2020). https://doi.org/10.1007/s10489-019-01580-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-019-01580-8

Keywords

Search

Navigation