Skip to main content
SpringerLink
Log in

Natural object manipulation using anthropomorphic robotic hand through deep reinforcement learning and deep grasping probability network

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Human hands can perform complex manipulation of various objects. It is beneficial if anthropomorphic robotic hands can manipulate objects like human hands. However, it is still a challenge due to the high dimensionality and a lack of machine intelligence. In this work, we propose a novel framework based on Deep Reinforcement Learning (DRL) with Deep Grasping Probability Network (DGPN) to grasp and relocate various objects with an anthropomorphic robotic hand much like a human hand. DGPN is used to predict the probability of successful human-like natural grasping based on the priors of human grasping hand poses and object touch areas. Thus, our DRL with DGPN rewards natural grasping hand poses according to object geometry for successful human-like manipulation of objects. The proposed DRL with DGPN is evaluated by grasping and relocating five objects including apple, light bulb, cup, bottle, and can. The performance of our DRL with DGPN is compared with the standard DRL without DGPN. The results show that the standard DRL only achieves an average success rate of 22.60%, whereas our DRL with DGPN achieves 89.40% for the grasping and relocation tasks of the objects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20

Similar content being viewed by others

References

  1. Andrychowicz O, Baker B, Chociej M, Józefowicz R, McGrew B, Pachocki J, Petron A, Plapert M, Powell G, Zaremba W (2020) Learning dexterous in-hand manipulation. Int J Robotics Res 39(1):3–20. https://doi.org/10.1177/0278364919887447

    Article  Google Scholar 

  2. Piazza C, Grioli G, Catalano M, Bicchi A (2019) A century of robotic hands. Ann Rev Control Robotics Autonomous Syst 2(1):1–32. https://doi.org/10.1146/annurev-control-060117-105003

    Article  Google Scholar 

  3. Kontoudis G, Liarokapis M, Zisimatos A, Mavrogiannis C, Kyriakopoulos K (2015) Open-source, anthropomorphic, underactuated robot hands with a selectively lockable differential mechanism: Towards affordable prostheses. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Hamburg, pp 5857–5862. https://doi.org/10.1109/IROS.2015.7354209

    Book  Google Scholar 

  4. Deimel R, Brock O (2016) A novel type of compliant and underactuated robotic hand for dexterous grasping. Int J Robotics Res 35(1–3):161–185. https://doi.org/10.1177/0278364915592961

    Article  Google Scholar 

  5. Smit G, Plettenburg D, Van der Helm F (2015) The lightweight Delft cylinder hand: first multi-articulating hand that meets the basic user requirements. IEEE Trans Neural Syst Rehab Eng 23(3):431–440. https://doi.org/10.1109/TNSRE.2014.2342158

    Article  Google Scholar 

  6. Starke J, Eichmann C, Ottenhaus S, Asfour T (2020) Human-inspired representation of object-specific. Int J Humanoid Robotics 17(02):2050008-1–2050008-20. https://doi.org/10.1142/S0219843620500085

    Article  Google Scholar 

  7. Ozawa R, Tahara K (2017) Grasp and dexterous manipulation of multi-fingered robotic hands: a review from a control view point. Adv Robot 31(19–20):1030–1050. https://doi.org/10.1080/01691864.2017.1365011

    Article  Google Scholar 

  8. Rosell J, Suárez R, Rosales C, Pérez A (2011) Autonomous motion planning of a hand-arm robotic system based on captured human-like hand postures. Autonomous Robots 31(87):87–102. https://doi.org/10.1007/s10514-011-9232-5

    Article  Google Scholar 

  9. Gu S, Holly E, Lillicrap T, Levine S (2017) Deep Reinforcement Learning for Robotic Manipulation with Asynchronous off-policy Updates. IEEE International Conference on Robotics and Automation (ICRA), Singapore, pp 3389–3396. https://doi.org/10.1109/ICRA.2017.7989385

    Book  Google Scholar 

  10. Riedmiller M, Hafner R, Lampe T, Neunert M, Van de Wiele T, Mnih V, Heess N, Springenberg T (2018) Learning by playing – solving sparse reward tasks from scratch. In proceedings of the 35 th international conference on machine, Stockholm

  11. Sanz P, Inesta J, Pobil A (1999) Planar grasping characterization based on curvature-symmetry fusion. Appl Intell 10:25–36. https://doi.org/10.1023/A:1008381314159

    Article  Google Scholar 

  12. Fenjiro Y, Benbrahim H (2018) Deep reinforcement learning overview of the state of the art. J Automation Mobile Robotics Intell Syst 12(3):20–39. https://doi.org/10.14313/JAMRIS_3-2018/15

    Article  Google Scholar 

  13. Ficucielo F (2019) Synergy-based control of Underactuated anthropomorphic hands. IEEE Trans Ind Inf 15(2):1144–1152. https://doi.org/10.1109/TII.2018.2841043

    Article  Google Scholar 

  14. Zhao X, Ding S, An Y, Jia W (2019) Applications of asynchronous deep reinforcement learning based on dynamic updating weights. Appl Intell 49:581–591. https://doi.org/10.1007/s10489-018-1296-x

    Article  Google Scholar 

  15. Mohsen F, Wang J, Al-Sabahi K (2020) A hierarchical self-attentive neural extractive summarizer via reinforcement learning (HSASRL). Appl Intell 50:2633–2646. https://doi.org/10.1007/s10489-020-01669-5

    Article  Google Scholar 

  16. Rajeswaran A, Kumar V, Gupta A, Vezzani G, Schulman J, Todorov E, Levine S (2018) Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations. in Proceedings of Robotics. Science and Systems, Pittsburgh

    Google Scholar 

  17. Peng X, Abbeel P, Levine S, Van de Panne M (2018) DeepMimic: example-guided deep reinforcement learning of physics-based character skills. ACM Trans Graph 37(4):1–14. https://doi.org/10.1145/3197517.3201311

    Article  Google Scholar 

  18. Ficuciello F, Migliozzi A, Laudante G, Falco P, Siciliano B (2019) Vision-based grasp learning of an anthropomorphic hand-arm system in a synergy-based control framework. Science robotics 4(26):1–11. https://doi.org/10.1126/scirobotics.aao4900

    Article  Google Scholar 

  19. Nazemi A (2019) A new collaborate Neuro-dynamic framework for solving convex second order cone programming problems with an application in multi-fingered robotic hands. Appl Intell 49:3512–3523. https://doi.org/10.1007/s10489-019-01462-z

    Article  Google Scholar 

  20. Li Y, Pollard N (2005) A shape matching algorithm for synthesizing humanlike enveloping grasps. 5th IEEE-RAS International Conference on Humanoid Robots, Tsukuba, pp 442–449. https://doi.org/10.1109/ICHR.2005.1573607

    Book  Google Scholar 

  21. Rubert C, Leon B, Morales A, Sancho-Bru J (2018) Characterisation of grasp quality metrics. J Intell Robotic Syst 89(3-4):319–342. https://doi.org/10.1109/ICRA.2014.6907393

    Article  Google Scholar 

  22. Lau M, Dev K, Shi W, Dorsey J, Rushmeier H (2016) Tactile mesh saliency. ACM Trans Graph 35(4):1–11. https://doi.org/10.1145/2897824.2925927

    Article  Google Scholar 

  23. Brahmbhatt S, Ham C, Kemp C, Hays J (2019) ContactDB: Analyzing and Predicting Grasp Contact via Thermal Imaging. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, pp 8701–8711. https://doi.org/10.1109/CVPR.2019.00891

    Book  Google Scholar 

  24. Wang Z, Li Z, Wang B, Liu H (2016) Robot grasp detection using multimodal deep convolutional neural networks. Adv Mech Eng 8(9):1–12. https://doi.org/10.1177/1687814016668077

    Article  Google Scholar 

  25. Levine S, Pastor P, Krizhevsky A, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robotics Res 37(4–5):421–436. https://doi.org/10.1177/0278364917710318

    Article  Google Scholar 

  26. Mahler J, Liang J, Niyaz S, Laskey M, Doan R, Liu X, Ojea J, Goldberg K (2017) Dex-net 2.0: deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. RSS: robotics: science and systems, Cambridge, Massachusetts, USA

  27. Todorov E, Erez T, Tassa Y (2012) MuJoCo: a physics engine for model-based control. IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura, pp 5026–5033. https://doi.org/10.1109/IROS.2012.6386109

    Book  Google Scholar 

  28. Kumar V, Xu Z, Todorov E (2013) Fast, strong and compliant pneumatic actuation for dexterous tendon-driven hands. IEEE International Conference on Robotics and Automation, Karlsruhe, pp 1512–1519. https://doi.org/10.1109/ICRA.2013.6630771

    Book  Google Scholar 

  29. Brahmbhatt S, Handa A, Hays J, Fox D (2019) ContactGrasp: Fuctional Multi-finger Grasp Synthesis from Contact. 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), Macau, China, doi: https://doi.org/10.1109/IROS40897.2019.8967960

  30. Arulkumaran K, Deisenroth M, Brundage M, Bharath A (2017) Deep reinforcement learning: a brief survey. IEEE Signal Process Magazine 34(6):26–38. https://doi.org/10.1109/MSP.2017.2743240

    Article  Google Scholar 

  31. Ravichandiran S (2018) Hands-on reinforcement learning with python. In: Shetty S (ed) . Packt Publishing Ltd., Birmingham, pp 7–18

    Google Scholar 

  32. Kakade S (2001) A natural policy gradient. In NIPS 01: proceedings of the 14th international conference on neural information processing systems: natural and synthetic, Vancouver, British Columbia, Canada

  33. Reinhard J (2007) Machine learning of motor skills for robotics. Ph.D. Dissertation, University of Southern California, USA

  34. Rajeswaran A, Lowrey K, Todorov E, Kakade S (2017) Towards generalization and simplicity in continuous control. In advances in neural information processing systems 30 (NIPS 2017), Los Angeles, United States

  35. Nair A, McGrew B, Andrychowicz M, Zaremba W, Abbeel P (2018) Overcoming Exploration in Reinforcement Learning with Demonstrations. IEEE International Conference on Robotic and Automation (ICRA), Brisbane, pp 6292–6299. https://doi.org/10.1109/ICRA.2018.8463162

    Book  Google Scholar 

  36. Zhao L, Zhou Y, Lu H, Fujita H (2019) Parallel computing method of deep belief networks and its application to traffic flow prediction. Knowl-Based Syst 163:972–987. https://doi.org/10.1016/j.knosys.2018.10.025

    Article  Google Scholar 

  37. Balasubramanian R, Santos V (eds) (2014) The human hand as an inspiration for robot hand development. Springer International Publishing, Switzerland, pp 219–247. https://doi.org/10.1007/978-3-319-03017-3

    Book  Google Scholar 

  38. Xu Z, Todorov E (2016) Design of a highly biomimetic anthropomorphic robotic hand towards artificial limb regeneration. IEEE International Conference on Robotics and Automation (ICRA), Stockholm, pp 3485–3492. https://doi.org/10.1109/ICRA.2016.7487528

    Book  Google Scholar 

  39. Gupta A, Eppner C, Levine S, Abbeel P (2016) Learning dexterous manipulation for a soft robotic hand from human demonstrations. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Daejeon, South Korea, pp. 3786–3793. https://doi.org/10.1109/IROS.2016.7759557

Download references

Acknowledgments

This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (MEST) (NRF-2019R1A2C1003713). Edwin Valarezo Añazco gratefully acknowledges to “Escuela Superior Politécnica del Litoral (ESPOL)” and its excellence program “Walter Valdano Raffo.”

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, College of Electronics and Information, Kyung Hee University, Yongin-si, Republic of Korea

    Edwin Valarezo Añazco, Patricio Rivera Lopez, Nahyeon Park, Jiheon Oh, Gahyeon Ryu, Mugahed A. Al-antari & Tae-Seong Kim

Authors
  1. Edwin Valarezo Añazco
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Patricio Rivera Lopez
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Nahyeon Park
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jiheon Oh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gahyeon Ryu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Mugahed A. Al-antari
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Tae-Seong Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tae-Seong Kim.

Ethics declarations

Conflict of interest

The authors declare no conflicts of interest regarding publishing this article.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Valarezo Añazco, E., Rivera Lopez, P., Park, N. et al. Natural object manipulation using anthropomorphic robotic hand through deep reinforcement learning and deep grasping probability network. Appl Intell 51, 1041–1055 (2021). https://doi.org/10.1007/s10489-020-01870-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-020-01870-6

Keywords

Search

Navigation