Skip to main content
SpringerLink
Log in

Learn to grasp unknown objects in robotic manipulation

  • Original Research Paper
  • Published:
Intelligent Service Robotics Aims and scope Submit manuscript

Abstract

Grasping unfamiliar objects (unknown during training) based on limited prior knowledge is a challenging task in robotic manipulation. Recent solutions typically require predefined information of target objects, task-specific training data, or a huge experience data with training time-consuming to achieve usable generalization ability. This paper introduces a robotic grasping strategy based on the model-free deep reinforcement learning, named Deep Reinforcement Grasp Policy. The developed system demands minimal training time and limited simple objects in simulation and generalizes efficiently on novel objects in real-world scenario. Without requiring any type of prior object awareness or task-specific training data. Our scalable visual grasping system is entirely self-learning approach. The model trains end-to-end policies (from only visual observations to decisions-making) to seek optimal grasp strategy. A perception network utilizes a convolutional neural network that maps visual observations to grasp action as dense pixel-wise Q-values represent the location and orientation of a primitive action executed by a robot. In simulation and physical experiments, a six-DOF robot manipulator with a two-finger gripper is utilized to validate the developed method. The empirical results demonstrated successfully based only on minimal previous knowledge of a few hours of simulated training and simple objects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Jonschkowski R, Eppner C, Höfer S, Martín-Martín R, Brock O (2016) Probabilistic multi-class segmentation for the amazon picking challenge. In: IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 1–7. https://doi.org/10.1109/iros.2016.7758087

  2. Tekin B, Sinha SN, Fua P (2018) Real-time seamless single shot 6d object pose prediction. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 292–301. https://doi.org/10.1109/cvpr.2018.00038

  3. Tremblay J, To T, Sundaralingam B, Xiang Y, Fox D, Birchfield S (2018) Deep object pose estimation for semantic robotic grasping of household objects. arXiv preprint. https://arxiv.org/abs/1809.10790v1

  4. Nguyen TT, Nguyen ND, Nahavandi S (2020) Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications. IEEE Trans Cybern. https://doi.org/10.1109/tcyb.2020.2977374

    Article  Google Scholar 

  5. Kalashnikov D et al (2018) Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint. https://arxiv.org/abs/1806.10293

  6. Levine S, Pastor P, Krizhevsky A, Ibarz J, Quillen D (2018) Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res 37(4–5):421–436. https://doi.org/10.1177/0278364917710318

    Article  Google Scholar 

  7. Sheng W, Thobbi A, Gu Y (2014) An integrated framework for human–robot collaborative manipulation. IEEE Trans Cybern 45(10):2030–2041. https://doi.org/10.1109/tcyb.2014.2363664

    Article  Google Scholar 

  8. Mnih V et al (2013) Playing atari with deep reinforcement learning. arXiv preprint. https://arxiv.org/abs/1312.5602

  9. Lillicrap TP et al (2015) Continuous control with deep reinforcement learning. arXiv preprint. https://arxiv.org/abs/1509.02971

  10. Rohmer E, Singh SP, Freese M (2013) V-REP: a versatile and scalable robot simulation framework. In: IEEE/RSJ international conference on intelligent robots and systems. IEEE, pp 1321–1326. https://doi.org/10.1109/iros.2013.6696520

  11. Herzog A et al (2014) Learning of grasp selection based on shape-templates. Auton Robots 36(1–2):51–65. https://doi.org/10.1007/s10514-013-9366-8

    Article  Google Scholar 

  12. Goldfeder C, Ciocarlie M, Dang H, Allen PK (2009) The Columbia grasp database. In: IEEE international conference on robotics and automation. IEEE, pp 1710–1716. https://doi.org/10.1109/robot.2009.5152709

  13. Mahler J et al (2017) Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint. https://arxiv.org/abs/1703.09312

  14. ten Pas A, Gualtieri M, Saenko K, Platt R (2017) Grasp pose detection in point clouds. Int J Robot Res 36(13–14):1455–1473. https://doi.org/10.1177/0278364917735594

    Article  Google Scholar 

  15. Han H, Paul G, Matsubara T (2017) Model-based reinforcement learning approach for deformable linear object manipulation. In: 13th IEEE conference on automation science and engineering (CASE). IEEE, pp 750–755. https://doi.org/10.1109/coase.2017.8256194

  16. Van der Merwe M, Lu Q, Sundaralingam B, Matak M, Hermans T (2020) Learning continuous 3D reconstructions for geometrically aware grasping. In: IEEE international conference on robotics and automation (ICRA), pp 11516–11522. https://doi.org/10.1109/ICRA40945.2020.9196981

  17. Zaidi L, Corrales JA, Bouzgarrou BC, Mezouar Y, Sabourin L (2017) Model-based strategy for grasping 3D deformable objects using a multi-fingered robotic hand. Robot Auton Syst 95:196–206. https://doi.org/10.1016/j.robot.2017.06.011

    Article  Google Scholar 

  18. Mahler J et al (2019) Learning ambidextrous robot grasping policies. Sci Robot. https://doi.org/10.1126/scirobotics.aau4984

    Article  Google Scholar 

  19. Mousavian A, Eppner C, Fox D (2019) 6-dof graspnet: variational grasp generation for object manipulation. In: Proceedings of the IEEE international conference on computer vision, pp 2901–2910. https://doi.org/10.1109/iccv.2019.00299

  20. Wang C et al. (2019) Densefusion: 6d object pose estimation by iterative dense fusion. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 3343–3352. https://doi.org/10.1109/cvpr.2019.00346

  21. Deng X, Xiang Y, Mousavian A, Eppner C, Bretl T, Fox D (2020) Self-supervised 6d object pose estimation for robot manipulation. In: IEEE international conference on robotics and automation (ICRA), pp 3665–3671. https://doi.org/10.1109/ICRA40945.2020.9196714

  22. Lenz I, Lee H, Saxena A (2015) Deep learning for detecting robotic grasps. Int J Robot Res 34(4–5):705–724. https://doi.org/10.1177/0278364914549607

    Article  Google Scholar 

  23. Redmon J, Angelova A (2015) Real-time grasp detection using convolutional neural networks. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 1316–1322. https://doi.org/10.1109/icra.2015.7139361

  24. Bohg J, Morales A, Asfour T, Kragic D (2013) Data-driven grasp synthesis—a survey. IEEE Trans Robot 30(2):289–309. https://doi.org/10.1109/tro.2013.2289018

    Article  Google Scholar 

  25. Hebert P et al (2012) Combined shape, appearance and silhouette for simultaneous manipulator and object tracking. In: IEEE international conference on robotics and automation. IEEE, pp 2405–2412. https://doi.org/10.1109/icra.2012.6225084

  26. Fang K et al (2020) Learning task-oriented grasping for tool manipulation from simulated self-supervision. Int J Robot Res 39(2–3):202–216

    Article  Google Scholar 

  27. Matsumoto E, Saito M, Kume A, Tan J (2020) End-to-end learning of object grasp poses in the Amazon Robotics Challenge. In: Advances on robotic item picking. Springer, pp 63–72. https://doi.org/10.1007/978-3-030-35679-8_6

  28. Breyer M, Furrer F, Novkovic T, Siegwart R, Nieto J (2019) Flexible robotic grasping with sim-to-real transfer based reinforcement learning. IEEE Robot Autom Lett 4(2):1549–1556. https://doi.org/10.1109/LRA.2019.2896467

    Article  Google Scholar 

  29. Liang H, Lou X, Choi C (2021) Learning visual affordances with target-orientated deep Q-network to grasp objects by harnessing environmental fixtures. arXiv preprint. https://arxiv.org/abs/1910.03781

  30. Gualtieri M, Ten Pas A, Saenko K, Platt R (2016) High precision grasp pose detection in dense clutter. In: IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 598–605. https://doi.org/10.1109/iros.2016.7759114

  31. Pinto L, Gupta A (2016) Supersizing self-supervision: learning to grasp from 50k tries and 700 robot hours. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 3406–3413. https://doi.org/10.1109/icra.2016.7487517

  32. Gualtieri M, Ten Pas A, Platt R (2018) Pick and place without geometric object models. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 7433–7440. https://doi.org/10.1109/icra.2018.8460553

  33. Xiang G, Su J (2019) Task-oriented deep reinforcement learning for robotic skill acquisition and control. IEEE Trans Cybern. https://doi.org/10.1109/tcyb.2019.2949596

    Article  Google Scholar 

  34. Sadeghi F, Levine S (2016) Cad2rl: real single-image flight without a single real image. arXiv preprint. https://arxiv.org/abs/1611.04201

  35. Luo B, Yang Y, Liu D (2018) Adaptive Q-learning for data-based optimal output regulation with experience replay. IEEE Trans Cybern 48(12):3337–3348. https://doi.org/10.1109/tcyb.2018.2821369

    Article  Google Scholar 

  36. Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 3389–3396. https://doi.org/10.1109/icra.2017.7989385

  37. Ghadirzadeh A, Maki A, Kragic D, Björkman M (2017) Deep predictive policy training using reinforcement learning. In: IEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, pp 2351–2358. https://doi.org/10.1109/iros.2017.8206046

  38. Van Hasselt H, Guez A, Silver D (2015) Deep reinforcement learning with double Q-learning. arXiv preprint. https://arxiv.org/abs/1509.06461

  39. Chen Z, Lin M, Jia Z, Jian S (2020) Towards generalization and data efficient learning of deep robotic grasping. arXiv preprint. https://arxiv.org/abs/2007.00982

  40. Jo H, Song J-B (2020) Object-independent grasping in heavy clutter. Appl Sci 10(3):804. https://doi.org/10.3390/app10030804

    Article  Google Scholar 

  41. Sauvet B, Lévesque F, Park S, Cardou P, Gosselin C (2019) Model-based grasping of unknown objects from a random pile. Robotics 8(3):79. https://doi.org/10.3390/robotics8030079

    Article  Google Scholar 

  42. Song S, Zeng A, Lee J, Funkhouser T (2020) Grasping in the wild: learning 6DoF closed-loop grasping from low-cost demonstrations. IEEE Robot Autom Lett 5(3):4978–4985. https://doi.org/10.1109/LRA.2020.3004787

    Article  Google Scholar 

  43. Mnih V et al (2015) Human-level control through deep reinforcement learning. Nature 518(7540):529–533. https://doi.org/10.1038/nature14236

    Article  Google Scholar 

  44. Zeng A et al (2018) Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In: IEEE international conference on robotics and automation (ICRA). IEEE, pp 1–8. https://doi.org/10.1177/0278364919868017

  45. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation, p 1605. arXiv e-prints. https://arxiv.org/abs/1605.06211

  46. Huang G, Liu Z, Van Der Maaten L, Weinberger KQ (2016) Densely connected convolutional networks. arXiv preprint. https://arxiv.org/abs/1608.06993

  47. Deng J, Dong W, Socher R, Li L-J, Li K, Fei-Fei L (2009) Imagenet: a large-scale hierarchical image database. In: IEEE conference on computer vision and pattern recognition. IEEE, pp 248–255. https://doi.org/10.1109/cvprw.2009.5206848

  48. Paszke A et al (2017) Automatic differentiation in pytorch

  49. Schaul T, Quan J, Antonoglou I, Silver D (2015) Prioritized experience replay. arXiv preprint. https://arxiv.org/abs/1511.05952

  50. Popov I et al (2017) Data-efficient deep reinforcement learning for dexterous manipulation. arXiv preprint. https://arxiv.org/abs/1704.03073

Download references

Acknowledgements

This work was supported by NSERC Discovery Program under Grant RGPIN-2017-05762 and MITACS Accelerate under IT14727.

Author information

Authors and Affiliations

  1. University of Ontario Institute of Technology, Oshawa, Canada

    Abdulrahman Al-Shanoon & Haoxiang Lang

  2. Kennesaw State University, Marietta, USA

    Ying Wang

  3. Senturing Technologies, Ltd., Vancouver, Canada

    Yunfei Zhang

  4. Xiamen University, Xiamen, China

    Wenxin Hong

Authors
  1. Abdulrahman Al-Shanoon
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Haoxiang Lang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ying Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yunfei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Wenxin Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Haoxiang Lang.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Shanoon, A., Lang, H., Wang, Y. et al. Learn to grasp unknown objects in robotic manipulation. Intel Serv Robotics 14, 571–582 (2021). https://doi.org/10.1007/s11370-021-00380-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11370-021-00380-9

Keywords

Search

Navigation