Spatial semantic hybrid map building and application of mobile service robot

Abstract

In a mobile service robot, map building is a basis for navigation that allows the robot to perform a given task. When robots carry on mission planning and execution, they not only need structural information about the environment to navigation and self localization, but also need to possess a deeper knowledge to endow a robot with higher degrees of autonomy and intelligence. In this paper, the QR code based artificial labels are used to provide semantic concepts and relations of objects and rooms in indoor environments, which can solve the complexity and limitations of semantic recognition and scene understanding only with robot’s vision. Semantic knowledge offered by artificial labels is obtained to incarnate semantic representations of the environment, which are formed as the semantic map. In order to maintain the navigation abilities of the robot, the spatial modeling method of a human is imitated to describe the relations of function area as a global topological map by the spectrum clustering algorithm. The semantic map is combined with topological representations to build a spatial semantic hybrid map. Using the hybrid map, “semantic” path planning, the elementary management of objects, and the context based robot localization are studied to show the hybrid map’s practicability. The results of several experiments show that the spatial semantic hybrid map comes up to capture the human point-of-view of robot environments, enable a high-level reasonable service path, and achieve function-driven navigation.

Introduction

As service robots go deeper and deeper into our work and life, the service category for humans becomes more and more wide, such as tea giving and water sending, garbage removal and transportation, cargo handling, letter transmission and so on. Therefore environment model building for service tasks of the robot has become the hotspot of current map building research. When robots carry out mission planning and execution, they not only need structural information about the environment for navigation and self localization, but also need to understand the semantic information of the environment to possess certain social skills. Social skills require the ability to interact with humans using a human-like language which means that the robot must be endowed with a representation of the environment involving objects concepts and their semantic relations. For the task of bringing a drink, for example, the decomposition steps of the action plan are a semantic sequence: “go to the kitchen — find the refrigerator — open the refrigerator — take a drink”. The robot needs to build a semantic map. It can reflect room features and the attributive relation between rooms and objects (such as the ascription relationship of the kitchen, the refrigerator and the drink.), and make a semantic annotation for the properties and operational method of service objects. Using the above information, the robots can easily and efficiently carry out automatic reasoning about service-related knowledge.

However, the tasks of localization and navigation are still the mainstream of map building. The metric map, topological map, metric–topological map or representation-based map have focused on the description of spatial structures, but not considered the functional characteristics of the working environment of the robots, or the attributive relationship between rooms and objects. Typically, humans seem to perceive space in terms of high-level information such as functions, states & descriptions, relationships etc. Thus, a human-compatible representation would have to encode similar information. In order to give robots the real high-level information, we should be inspired from the representation of human’s habits to describe the space environment, and study semantic maps including the description of an item’s information, a description of the room function and a description of the attributive relationship between rooms and objects, which guarantee the completion of a service robot’s services task.

Aiming at a robot’s indoor service tasks, an environment cognition method named as QR code(Quick Response Code) based spatial semantic hybrid map building, is proposed. The semantic maps  [1], [2], [3], [4] come up to capture the human point-of-view of robot environments, which enable high-level and more intelligent robot development and human–robot interaction as well. The QR code technology  [5], [6] is used to solve the complexity and limitations of semantic recognition and scene understanding which is only relying on the robot’s vision. To build the spacial map, the following steps are used. Firstly, the robot’s vision is used to formulate an undirected weight graph based on a SIFT feature matching algorithm. Then the spectral clustering algorithm is used to construct a global topological graph of the indoor unknown environment with room partition function. In indoor environment, the information taken by the visual is complex, and information quality is easily impacted by light, shelter, etc. So the result of image processing is very unstable. Even though the SIFT featured matching algorithm has outstanding advantages, it still has certain limitations. To resolve this problem, two-dimensional artificial object labels based on QR code technology are pasted on large objects. This method can not only enhance the accuracy and speed of the environment model building, but also get an item’s information of functional properties to form an object database, and establish attributive relations between rooms and objects. Based on the information of the QR code, semantic descriptions of the environment are obtained by the robot to build the semantic map. In order to maintain the navigation abilities of the robot, topological representations are added to the semantic map which possess hybrid map characters. It should be noted that the QR code labels are only marked on large objects, such as refrigerators, TV set cabinets, beds, etc. For example, there is a refrigerator and an operating board in a kitchen, a sofa and TV set cabinet in a living room, a bookshelf and a desk in the study room, a bed and a wardrobe in the bedroom, and so on. The arrangement of an office is much simpler, including some desks and file cabinets. Thus the small number of labels pasted on the large objects can solve the cognitive problem of the functions of rooms and objects. Small object ownership relations can be written in QR code labels. As small object relationships with large objects are basically determined, and the relation between large objects and rooms are also unchanged, it is possible to do this in a real-world application.

A semantic hybrid map makes path planning, localization and navigation of a robot possess semantic characters, and then a robot possessing human intelligence is applied to achieve service tasks such as object management. This method can improve a mobile robot’s understanding of the environment from a geometrical structure, vague, passive level, to a semantic, accurate, active cognitive level. As artificial labels are used to give some surrounding information, the environment explored by robots is treated as a semi-unknown environment.

The rest of this paper is organized as follows. The related works of map building are introduced in Section  2. In Section  3, the spectral clustering algorithm is used to realize function area segmentation, and the global topological map is acquired to describe the unknown environment. Then the semantic information of objects and rooms are added to the global topological map based on QR code labels, which are discussed in Section  4. Two illustrated examples, semantic path planning and context based robot localization, are presented in Section  5 to show the advantages of the proposed semantic map. These real applications are used to describe how semantic maps improve the hierarchy of task planning and the accuracy of robot localization. As QR code labels are crucial to obtaining semantic information, the design and identification of QR code and the orientation adjustment strategy are presented in Section  6. In Section  7, the experimental results are shown to validate the proposed method. The conclusions are included in Section  8.