Language Guided Grasping of Unknown Concepts Based on Knowledge System

Abstract

Language guided grasping in cultter environment has become a hot topic for robotic arms, where the robot is supposed to grasp a specific object according to the language instruction. However, existing studies typically fail on grasping unknown objects that are not recognized by the detection model, which post higher requirements of the learning unknown objects from external knowledge. In this paper, unlike traditional target detection algorithms, our system identifies unknown objects through a knowledge-driven approach. We present a knowledge-learning driven system that guides the robot in learning new concepts to grasp unknown objects from pre-built multimodal knowledge base. Specifically, we first retrieve the image knowledge from Internet and text description knowledge from human experts to construct the multimodal knowledge base. Based on the question context, the model will query the object concept knowledge from the knowledge base. Once the concept is known, the model will ground the object by a multimodal fusion module. In contrast, to address the issue of novel concept, we propose a knowledge fusion module that combines BERT, GCN and ResNet to acquire semantic text knowledge, structured text knowledge, and image knowledge derived from the knowledge base. Subsequently, we utilized the re-calibration multimodal fusion module to ground unknown objects. Finally, conditioned on the grounding context, we employ a well-trained grasp network to derive the accurate grasping pose to perform the grasping task. Experiments conducted on the simulation and real-world scenarios demonstrate that our knowledge-learning driven system is effective in assisting robots to learn unknown concepts and achieve successful grasping tasks. A demo video is available online (https://youtu.be/Fsc4nZyY9LI).