Skip to main content
SpringerLink
Log in

A Graph-Based Deep Reinforcement Learning Approach to Grasping Fully Occluded Objects

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Grasping in cluttered scenes is an important issue in robotic manipulation. The cooperation of grasping and pushing actions based on reinforcement learning is an effective means to obtain the target object when it is completely blocked or there is no suitable grasping position around it. When exploring invisible objects, many existing methods depend excessively on model design and redundant grasping actions. We propose a graph-based deep reinforcement learning model to efficiently explore invisible objects and improve the performance for cooperative grasping and pushing tasks. Our model first extracts the state features and then estimates the Q value with different graph Q-Nets according to whether the target object is found. The graph-based Q-learning model contains an encoder, a graph reasoning module and a decoder. The encoder is used to integrate the state features such that the features of one region include those of other regions. The graph reasoning module captures the internal relationships of features between different regions through graph convolution networks. The decoder maps the features transformed by reasoning to the original state features. Our method achieves a 100% success rate in the task of exploring the target object and a success rate of more than 90% in the task of grasping and pushing cooperatively in simulation experiment, which performs better than many existing state-of-the-art methods. Our method is an effective means to help robots obtain completely occluded objects by grasping and pushing cooperation in the cluttered scenes. The verification experiment on the real robot further shows the generalization and practicability of our proposed model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Data Availability

Data openly available in a public repository. The data that support the findings of this study are openly available at https://github.com/ttongjiayuan/the-dataset-of-grasping-occluded-objects.

References

  1. Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps. Int J Robot Res. 2015;34(4–5):705–24.

    Article  Google Scholar 

  2. Redmon J, Angelova A. Real-time grasp detection using convolutional neural networks. In: 2015 IEEE International Conference on Robotics and Automation (ICRA). IEEE; 2015. p. 1316–22.

  3. ten Pas A, Gualtieri M, Saenko K, Platt R. Grasp pose detection in point clouds. Int J Robot Res. 2017;36(13–14):1455–73.

    Google Scholar 

  4. Ni P, Zhang W, Bai W, Lin M, Cao Q. A new approach based on two-stream cnns for novel objects grasping in clutter. J Intell Robot Syst. 2019;94(1):161–77.

    Article  Google Scholar 

  5. Yang Y, Liang H, Choi C. A deep learning approach to grasping the invisible. IEEE Robotics and Automation Letters. 2020;5(2):2232–9.

    Article  Google Scholar 

  6. Shao Q, Hu J, Wang W, Fang Y, Liu W, Qi J, Ma J. Suction grasp region prediction using self-supervised learning for object picking in dense clutter. In: 2019 IEEE 5th International Conference on Mechatronics System and Robots (ICMSR). IEEE; 2019.  p. 7–12.

  7. Bhagat S, Banerjee H, Ho Tse ZT, Ren H. Deep reinforcement learning for soft, flexible robots: Brief review with impending challenges. Robotics. 2019;8(1):4.

    Article  Google Scholar 

  8. Berscheid L, Meßner P, Kröger T. Robot learning of shifting objects for grasping in cluttered environments. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2019. p. 612–18.

  9. Zeng A, Song S, Welker S, Lee J, Rodriguez A, Funkhouser T. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2018. p. 4238–245.

  10. Boularias A, Bagnell JA, Stentz A. Learning to manipulate unknown objects in clutter by reinforcement. Twenty-Ninth AAAI Conference on Artificial Intelligence. 2015.

  11. Zhang H, Lan X, Zhou X, Tian Z, Zhang Y, Zheng N. Visual manipulation relationship recognition in object-stacking scenes. Pattern Recogn Lett. 2020;140:34–42.

    Article  Google Scholar 

  12. Zuo G, Tong J, Liu H, Chen W, Li J. Graph-based visual manipulation relationship reasoning network for robotic grasping. Front Neurorobot. 2021;15.

  13. Zhang H, Lan X, Bai S, Zhou X, Tian Z, Zheng N. Roi-based robotic grasp detection for object overlapping scenes. In: 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2019. p. 4768–75.

  14. Ren Y, Zhu C, Xiao S. Deformable faster E-CNN with aggregating multi-layer features for partially occluded object detection in optical remote sensing images. Remote Sens. 2018;10.9:1470.

  15. Boroushaki T, Leng J, Clester I, Rodriguez A, Adib F. Robotic grasping of fully-occluded objects using RF perception. In: 2021 IEEE International Conference on Robotics and Automation (ICRA). IEEE; 2021. p. 923–29.

  16. Cui Y, Ooga JI, Ogawa A, Matsubara T. Probabilistic active filtering with gaussian processes for occluded object search in clutter. Appl Intell. 2020;50(12):4310–24.

    Article  Google Scholar 

  17. Danielczuk M, Kurenkov A, Balakrishna A, Matl M, Wang D, Martín-Martín R, ...  Goldberg K. Mechanical search: Multi-step retrieval of a target object occluded by clutter. In: 2019 International Conference on Robotics and Automation (ICRA). IEEE; 2019. p. 1614–621.

  18. Danielczuk M, Angelova A, Vanhoucke V, Goldberg K. X-ray: Mechanical search for an occluded object by minimizing support of learned occupancy distributions. In: 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE; 2020. p. 9577–584.

  19. Kipf TN, Welling M. Semi-supervised classification with graph convolutional networks. 2016. arXiv preprint arXiv:1609.02907.

  20. Chen Y, Rohrbach M, Yan Z, Shuicheng Y, Feng J, Kalantidis Y. Graph-based global reasoning networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. p. 433–42.

  21. Satorras VG, Estrach JB. Few-shot learning with graph neural networks. In: International Conference on Learning Representations. 2018.

  22. Kampffmeyer M, Chen Y, Liang X, Wang H, Zhang Y, Xing EP. Rethinking knowledge graph propagation for zero-shot learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. p. 11487–96.

  23. Kipf TN, Welling M. Variational graph auto-encoders. 2016. arXiv preprint arXiv:1611.07308.

  24. Jiang J, Dun C, Huang T, Lu Z. Graph convolutional reinforcement learning. In: International Conference on Learning Representations. 2019.

  25. Zambaldi V, Raposo D, Santoro A, Bapst V, Li Y, Babuschkin I, ... Battaglia P. Relational deep reinforcement learning. 2018. arXiv preprint arXiv:1806.01830.

  26. Li R, Jabri A, Darrell T, Agrawal P. Towards practical multi-object manipulation using relational reinforcement learning. In: 2020 IEEE International Conference on Robotics and Automation (ICRA). IEEE; 2020. p. 4051–58.

  27. Mnih V, Kavukcuoglu K, Silver D, Rusu AA, Veness J, Bellemare MG, ... Hassabis D. Human-level control through deep reinforcement learning. Nature. 2015;518(7540):529–33.

  28. Rohmer MFE, Singh SPN. V-REP: a versatile and scalable robot simulation framework. In: IROS. 2013.

  29. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE International Conference on Computer Vision. 2017. p. 618–26.

Download references

Funding

This work was supported by the National Natural Science Foundation of China (61873008), Beijing Natural Science Foundation (4192010) and National Key R & D Plan (2018YFB1307004).

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Beijing University of Technology, Beijing, China

    Guoyu Zuo, Jiayuan Tong, Zihao Wang & Daoxiong Gong

  2. Beijing Key Laboratory of Computing Intelligence and Intelligent Systems, Beijing, 100124, China

    Guoyu Zuo, Jiayuan Tong, Zihao Wang & Daoxiong Gong

Authors
  1. Guoyu Zuo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiayuan Tong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zihao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Daoxiong Gong
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guoyu Zuo.

Ethics declarations

Ethics Approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Conflict of Interest

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zuo, G., Tong, J., Wang, Z. et al. A Graph-Based Deep Reinforcement Learning Approach to Grasping Fully Occluded Objects. Cogn Comput 15, 36–49 (2023). https://doi.org/10.1007/s12559-022-10047-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-022-10047-x

Keywords

Search

Navigation