Abstract
Robotic grasping plays a crucial role in manipulation tasks. However, due to the complexity of human-robot interaction, service robots still face significant challenges in handling task-oriented operations in real-world environments. To address this issue and better meet practical interaction needs, we propose a multimodal combination framework for robotic grasping. It leverages language texts to facilitate communication and detects and grasps target objects based on point clouds and feedback. The framework comprises several multimodal components, including ChatGPT, stereo cameras, and wearable devices, to complete instruction processing, grasp detection, and motion execution. To enable effective interaction, ChatGPT facilitates basic communication and responds to instructions between humans and robots. Additionally, the robot can detect the 6-DoF grasp of objects based on point clouds obtained by stereo cameras. These grasps are combined with the feedback provided by ChatGPT to further meet the requirement from human. Finally, we utilize wearable devices to teach robots generalized motor skills. This enables the robot to learn corresponding movements and perform them effectively in various scenarios, further improving its manipulation abilities. The experimental results from simulated conversations and real-scene tasks highlight that our proposed framework provides logical communication, stable grasping, and effective motion.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Suwa, S., et al.: Home-care professionals’ ethical perceptions of the development and use of home-care robots for older adults in Japan. Int. J. Hum.-Comput. Interact. 36, 1295–1303 (2020)
ten Pas, A., Gualtieri, M., Saenko, K., Platt, R.: Grasp pose detection in point clouds. Int. J. Robot. Res. 36(13–14, SI), 1455–1473 (2017)
Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners, pp. 2205–11916 (2022)
Madaan, A., Zhou, S., Alon, U., Yang, Y., Neubig, G.: Language models of code are few-shot commonsense learners. arXiv e-prints, 2210–07128 (2022)
Liang, J., et al.: Code as policies: language model programs for embodied control. arXiv e-prints, 2209–07753 (2022)
Liang, H., et al.: PointNetGPD: detecting grasp configurations from point sets. In: 2019 IEEE International Conference on Robotics and Automation (ICRA), pp. 3629–3635 (2019)
Charles, R.Q., Su, H., Kaichun, M., Guibas, L.J.: PointNet: deep learning on point sets for 3d classification and segmentation. In: 30TH IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), pp. 77–85 (2017)
Ni, P., Zhang, W., Zhu, X., Cao, Q.: PointNet++ grasping: learning an end-to-end spatial grasp generation algorithm from sparse point clouds. In: 2020 IEEE International Conference on Robotics and Automation (ICRA), pp. 3619–3625 (2020)
Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet plus plus: deep hierarchical feature learning on point sets in a metric space. In: Advances in Neural Information Processing Systems 30 (NIPS 2017), vol. 30 (2017)
Ambhore, S.: A comprehensive study on robot learning from demonstration. In: 2020 2nd International Conference on Innovative Mechanisms for Industry Applications (ICIMIA), pp. 291–299 (2020)
Ginesi, M., Meli, D., Calanca, A., Dall’Alba, D., Sansonetto, N., Fiorini, P.: Dynamic movement primitives: volumetric obstacle avoidance. In: 2019 19th International Conference on Advanced Robotics (ICAR), pp. 234–239 (2019)
Acknowledgments
This work was supported in part by the National key R&D program of China 2018AAA0100803, in part by the National Natural Science Foundation 62276090 and 62203150, in part by the Key Research and Development Program of Jiangsu under grants BK20192004B, in part by the China Postdoctoral Science Foundation under Grant 2021M701051, in part by the Jiangsu Province Excellent Post-doctoral Program 2022ZB192, in part by the Changzhou Basic Research Program (Application Program) CJ20220051, in part by the Guangdong Forestry Science and Technology Innovation Project under grant 2020KJCX005.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Huang, C., Wang, Z., Zhu, H., Li, J., Liu, X. (2023). Design and Implementation of a Multimodal Combination Framework for Robotic Grasping. In: Ronzhin, A., Sadigov, A., Meshcheryakov, R. (eds) Interactive Collaborative Robotics. ICR 2023. Lecture Notes in Computer Science(), vol 14214. Springer, Cham. https://doi.org/10.1007/978-3-031-43111-1_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-43111-1_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43110-4
Online ISBN: 978-3-031-43111-1
eBook Packages: Computer ScienceComputer Science (R0)