Abstract
Autonomous robot grasping in multi-object scenarios poses significant challenges, requiring precise grasp candidate detection, determination of object-grasp affiliations. To solve these challenges, this research presents a novel approach to address these challenges by developing a dedicated grasp detection model called GraspAnything, which is extended from SegmentAnything model. The GraspAnything model, based on SegmentAnything (SAM) model, receives bounding boxes as prompts and simultaneously outputs the mask of objects and all possible grasp poses for parallel jaw gripper. A grasp decoder module is added to the SAM model to enable grasp detection functionality. Experiment results have demonstrate the effectiveness of our model in grasp detection tasks. The implications of this research extend to various industrial applications, such as object picking and sorting, where intelligent robot grasping can significantly enhance efficiency and automation. The developed models and approaches contribute to the advancement of autonomous robot grasping in complex, multi-object environments.
S. Liu and Z. Lei—The first two authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Kirillov, A., et al.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015–4026 (2023)
Morrison, D., Corke, P., Leitner, J.: Closing the loop for robotic grasping: a real-time, generative grasp synthesis approach. arXiv preprint arXiv:1804.05172 (2018)
Mahler, J., Matl, M., Liu, X., Li, A., Gealy, D., Goldberg, K.: Dex-net 3.0: computing robust vacuum suction grasp targets in point clouds using a new analytic model and deep learning. In: 2018 IEEE International Conference on robotics and automation (ICRA), pp. 5620–5627. IEEE (2018)
Kumra, S., Joshi, S., Sahin, F.: GR-ConvNet v2: a real-time multi-grasp detection network for robotic grasping. Sensors 22(16), 6208 (2022)
Morgan, A.S., Wen, B., Liang, J., et al.: Vision-driven compliant manipulation for reliable, high-precision assembly tasks. arXiv preprint arXiv:2106.14070 (2021)
Di Lillo, P., Arrichiello, F., Di Vito, D., Antonelli, G.: BCI-controlled assistive manipulator: developed architecture and experimental results. IEEE Trans. Cogn. Dev. Syst. 13(1), 91–104 (2020)
Zhang, H., Lan, X., Bai, S., et al.: A multi-task convolutional neural network for autonomous robotic grasping in object stacking scenes. In: Proceedings of the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 6435–6442. IEEE (2019)
Dong, M., Zhang, J.: A review of robotic grasp detection technology. Robotica 2023, 1–40 (2023)
Caldera, S., Rassau, A., Chai, D.: Review of deep learning methods in robotic grasp detection. Multimodal Technol. Interact. 2(3), 57 (2018)
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards real-time object detection with region proposal networks. Adv. Neural. Inf. Process. Syst. 28, 91–99 (2015)
Ainetter, S., Böhm, C., Dhakate, R., et al.: Depth-aware object segmentation and grasp detection for robotic picking tasks. arXiv preprint arXiv:2111.11114 (2021)
Redmon, J., Angelova, A.: Real-time grasp detection using convolutional neural networks. In: Proceedings of the 2015 IEEE International Conference on Robotics and Automation (ICRA), pp. 1316–1322. IEEE (2015)
Zhang, H., Lan, X., Zhou, X., Tian, Z., Zhang, Y., Zheng, N.: Visual manipulation relationship network for autonomous robotics. In: 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids), pp. 118–125. IEEE (2018)
Ainetter, S., Fraundorfer, F.: End-to-end trainable deep neural network for robotic grasp detection and semantic segmentation from RGB. In: 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 13452–13458. IEEE (2021)
Depierre, A., Dellandréa, E., Chen, L.: Jacquard: a large scale dataset for robotic grasp detection. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3511–3516. IEEE (2018)
Gilles, M., et al.: MetaGraspNetV2: all-in-one dataset enabling fast and reliable robotic bin picking via object relationship reasoning and dexterous grasping. IEEE Trans. Autom. Sci. Eng. (2023)
Cao, H., Chen, G., Li, Z., Feng, Q., Lin, J., Knoll, A.: Efficient grasp detection network with Gaussian-based grasp representation for robotic manipulation. IEEE/ASME Trans. Mechatron. (2022)
Holomjova, V., Starkey, A.J., Meißner, P.: GSMR-CNN: an end-to-end trainable architecture for grasping target objects from multi-object scenes. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3808–3814. IEEE (2023)
Zhou, X., Lan, X., Zhang, H., Tian, Z., Zhang, Y., Zheng, N.: Fully convolutional grasp detection network with oriented anchor box. In: 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7223–7230. IEEE (2018)
Song, Y., Gao, L., Li, X., Shen, W.: A novel robotic grasp detection method based on region proposal networks. Robot. Comput.-Integr. Manuf. 65, 101963 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Liu, S., Lei, Z., Zhu, H., Ma, J., Lin, Z. (2025). SegmentAnything-Based Approach to Scene Understanding and Grasp Generation. In: Li, H., et al. Social Robotics. ICSR + InnoBiz 2024. Lecture Notes in Computer Science(), vol 15170. Springer, Singapore. https://doi.org/10.1007/978-981-96-1151-5_3
Download citation
DOI: https://doi.org/10.1007/978-981-96-1151-5_3
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-96-1150-8
Online ISBN: 978-981-96-1151-5
eBook Packages: Computer ScienceComputer Science (R0)