Skip to main content
SpringerLink
Log in

A novel vision-based multi-task robotic grasp detection method for multi-object scenes

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Grasping a specified object from multi-object scenes is an essential ability for intelligent robots. This ability depends on the affiliation between the grasp position and the object category. Most existing multi-object grasp detection methods considering the affiliation rely on object detection results, thus limiting the improvement of robotic grasp detection accuracy. This paper proposes a decoupled single-stage multi-task robotic grasp detection method based on the Faster R-CNN framework for multi-object scenes. The designed network independently detects the category of an object and its possible grasp positions by using one loss function. A new grasp matching strategy is designed to determine the relationship between object categories and predicted grasp positions. The VMRD grasp dataset is used to test the performance of the proposed method. Compared with other grasp detection methods, the proposed method achieves higher object detection accuracy and grasp detection accuracy.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Hou X Y, Ao W, Song Q, et al. FUSAR-Ship: building a high-resolution SAR-AIS matchup dataset of Gaofen-3 for ship detection and recognition. Sci China Inf Sci, 2020, 63: 140303

    Article  Google Scholar 

  2. Zhang W T, Jiang J W, Shao Y X, et al. Snapshot boosting: a fast ensemble framework for deep neural networks. Sci China Inf Sci, 2020, 63: 112102

    Article  Google Scholar 

  3. Xie G, Shangguan A Q, Fei R, et al. Motion trajectory prediction based on a CNN-LSTM sequential model. Sci China Inf Sci, 2020, 63: 212207

    Article  Google Scholar 

  4. Xie J, Pang Y W, Cholakkal H, et al. PSC-Net: learning part spatial co-occurrence for occluded pedestrian detection. Sci China Inf Sci, 2021, 64: 120103

    Article  MathSciNet  Google Scholar 

  5. Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps. Int J Robot Res, 2015, 34: 705–724

    Article  Google Scholar 

  6. Redmon J, Angelova A. Real-time grasp detection using convolutional neural networks. In: Proceedings of IEEE International Conference on Robotics and Automation, 2015. 1316–1322

  7. Zhou X W, Lan X G, Zhang H B, et al. Fully convolutional grasp detection network with oriented anchor box. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018. 7223–7230

  8. Song Y, Gao L, Li X, et al. A novel robotic grasp detection method based on region proposal networks. Robot Comput-Integrated Manuf, 2020, 65: 101963

    Article  Google Scholar 

  9. Mahler J, Goldberg K. Learning deep policies for robot bin picking by simulating robust grasping sequences. In: Proceedings of Conference on Robot Learning, 2017. 515–524

  10. Zeng A, Song S, Yu K-T, et al. Robotic pick-and-place of novel objects in clutter with multi-affordance grasping and cross-domain image matching. In: Proceedings of IEEE International Conference on Robotics and Automation, 2018. 1–8

  11. Zhang H, Lan X, Bai S, et al. ROI-based robotic grasp detection for object overlapping scenes. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019. 4768–4775

  12. Park D, Seo Y, Shin D, et al. A single multi-task deep neural network with post-processing for object detection with reasoning and robotic grasp detection. In: Proceedings of IEEE International Conference on Robotics and Automation, 2020. 7300–7306

  13. Ren S Q, He K M, Girshick R, et al. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell, 2017, 39: 1137–1149

    Article  Google Scholar 

  14. Asif U, Bennamoun M, Sohel F A. RGB-D object recognition and grasp detection using hierarchical cascaded forests. IEEE Trans Robot, 2017, 33: 547–564

    Article  Google Scholar 

  15. Wang Z C, Li Z Q, Wang B, et al. Robot grasp detection using multimodal deep convolutional neural networks. Adv Mech Eng, 2016, 8: 1–12

    Google Scholar 

  16. Kumra S, Kanan C. Robotic grasp detection using deep convolutional neural networks. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, 2017. 769–776

  17. Guo D, Sun F C, Kong T, et al. Deep vision networks for real-time robotic grasp detection. Int J Adv Robotic Syst, 2017, 14: 1–8

    Google Scholar 

  18. Guo D, Sun F, Liu H, et al. A hybrid deep architecture for robotic grasp detection. In: Proceedings of IEEE International Conference on Robotics and Automation, 2017. 1609–1614

  19. Gualtieri M, Pas A T, Saenko K, et al. High precision grasp pose detection in dense clutter. In: Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems, 2016. 598–605

  20. Guo D, Kong T, Sun F, et al. Object discovery and grasp detection with a shared convolutional neural network. In: Proceedings of IEEE International Conference on Robotics and Automation, 2016. 2038–2043

  21. Jiang Y, Moseson S, Saxena A. Efficient grasping from RGBD images: learning using a new rectangle representation. In: Proceedings of IEEE International Conference on Robotics and Automation, 2011. 3304–3311

  22. He K M, Zhang X Y, Ren S Q, et al. Deep residual learning for image recognition. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016. 770–778

  23. Lin T Y, Dollar P, Girshick R, et al. Feature pyramid networks for object detection. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2017. 936–944

  24. Yang X, Sun H, Fu K, et al. Automatic ship detection in remote sensing images from Google Earth of complex scenes based on multiscale rotation dense feature pyramid networks. Remote Sens, 2018, 10: 132

    Article  Google Scholar 

  25. Song Y, Pan Q K, Gao L, et al. Improved non-maximum suppression for object detection using harmony search algorithm. Appl Soft Computing, 2019, 81: 105478

    Article  Google Scholar 

  26. Zhang H, Lan X, Wan L, et al. RPRG: toward real-time robotic perception, reasoning and grasping with one multi-task convolutional neural network. 2018. ArXiv:1809.07081

Download references

Acknowledgements

This work was supported by China Postdoctoral Science Foundation (Grant No. 2021M692778), National Key Research and Development Project of China (Grant No. 2018AAA0101704), Natural Science Foundation of Hubei Province (Grant No. 2021CFB368), and Research Project of Hubei Provincial Department of Education (Grant No. Q20201105).

Author information

Authors and Affiliations

  1. College of Computer Science and Technology, Zhejiang University, Hangzhou, 310058, China

    Yanan Song

  2. Institute of Computing Innovation, Zhejiang University, Hangzhou, 311215, China

    Yanan Song

  3. State Key Laboratory of Digital Manufacturing Equipment and Technology, Huazhong University of Science and Technology, Wuhan, 430074, China

    Liang Gao, Xinyu Li & Weiming Shen

  4. School of Management, Wuhan University of Science and Technology, Wuhan, 430081, China

    Kunkun Peng

Authors
  1. Yanan Song
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Liang Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xinyu Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Weiming Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kunkun Peng
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Weiming Shen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, Y., Gao, L., Li, X. et al. A novel vision-based multi-task robotic grasp detection method for multi-object scenes. Sci. China Inf. Sci. 65, 222104 (2022). https://doi.org/10.1007/s11432-021-3558-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11432-021-3558-y

Keywords

Search

Navigation