Spatial semantic hybrid map building and application of mobile service robot
Highlights
► QR code technology is used for the robot to build the semantic hybrid map. ► Semantic information is supplied to the traditional map of the robot. ► Semantic path planning and the elementary management of objects are realized. ► Using the method mentioned in this paper, the robot can build a semantic map autonomously.
Introduction
As service robots go deeper and deeper into our work and life, the service category for humans becomes more and more wide, such as tea giving and water sending, garbage removal and transportation, cargo handling, letter transmission and so on. Therefore environment model building for service tasks of the robot has become the hotspot of current map building research. When robots carry out mission planning and execution, they not only need structural information about the environment for navigation and self localization, but also need to understand the semantic information of the environment to possess certain social skills. Social skills require the ability to interact with humans using a human-like language which means that the robot must be endowed with a representation of the environment involving objects concepts and their semantic relations. For the task of bringing a drink, for example, the decomposition steps of the action plan are a semantic sequence: “go to the kitchen — find the refrigerator — open the refrigerator — take a drink”. The robot needs to build a semantic map. It can reflect room features and the attributive relation between rooms and objects (such as the ascription relationship of the kitchen, the refrigerator and the drink.), and make a semantic annotation for the properties and operational method of service objects. Using the above information, the robots can easily and efficiently carry out automatic reasoning about service-related knowledge.
However, the tasks of localization and navigation are still the mainstream of map building. The metric map, topological map, metric–topological map or representation-based map have focused on the description of spatial structures, but not considered the functional characteristics of the working environment of the robots, or the attributive relationship between rooms and objects. Typically, humans seem to perceive space in terms of high-level information such as functions, states & descriptions, relationships etc. Thus, a human-compatible representation would have to encode similar information. In order to give robots the real high-level information, we should be inspired from the representation of human’s habits to describe the space environment, and study semantic maps including the description of an item’s information, a description of the room function and a description of the attributive relationship between rooms and objects, which guarantee the completion of a service robot’s services task.
Aiming at a robot’s indoor service tasks, an environment cognition method named as QR code(Quick Response Code) based spatial semantic hybrid map building, is proposed. The semantic maps [1], [2], [3], [4] come up to capture the human point-of-view of robot environments, which enable high-level and more intelligent robot development and human–robot interaction as well. The QR code technology [5], [6] is used to solve the complexity and limitations of semantic recognition and scene understanding which is only relying on the robot’s vision. To build the spacial map, the following steps are used. Firstly, the robot’s vision is used to formulate an undirected weight graph based on a SIFT feature matching algorithm. Then the spectral clustering algorithm is used to construct a global topological graph of the indoor unknown environment with room partition function. In indoor environment, the information taken by the visual is complex, and information quality is easily impacted by light, shelter, etc. So the result of image processing is very unstable. Even though the SIFT featured matching algorithm has outstanding advantages, it still has certain limitations. To resolve this problem, two-dimensional artificial object labels based on QR code technology are pasted on large objects. This method can not only enhance the accuracy and speed of the environment model building, but also get an item’s information of functional properties to form an object database, and establish attributive relations between rooms and objects. Based on the information of the QR code, semantic descriptions of the environment are obtained by the robot to build the semantic map. In order to maintain the navigation abilities of the robot, topological representations are added to the semantic map which possess hybrid map characters. It should be noted that the QR code labels are only marked on large objects, such as refrigerators, TV set cabinets, beds, etc. For example, there is a refrigerator and an operating board in a kitchen, a sofa and TV set cabinet in a living room, a bookshelf and a desk in the study room, a bed and a wardrobe in the bedroom, and so on. The arrangement of an office is much simpler, including some desks and file cabinets. Thus the small number of labels pasted on the large objects can solve the cognitive problem of the functions of rooms and objects. Small object ownership relations can be written in QR code labels. As small object relationships with large objects are basically determined, and the relation between large objects and rooms are also unchanged, it is possible to do this in a real-world application.
A semantic hybrid map makes path planning, localization and navigation of a robot possess semantic characters, and then a robot possessing human intelligence is applied to achieve service tasks such as object management. This method can improve a mobile robot’s understanding of the environment from a geometrical structure, vague, passive level, to a semantic, accurate, active cognitive level. As artificial labels are used to give some surrounding information, the environment explored by robots is treated as a semi-unknown environment.
The rest of this paper is organized as follows. The related works of map building are introduced in Section 2. In Section 3, the spectral clustering algorithm is used to realize function area segmentation, and the global topological map is acquired to describe the unknown environment. Then the semantic information of objects and rooms are added to the global topological map based on QR code labels, which are discussed in Section 4. Two illustrated examples, semantic path planning and context based robot localization, are presented in Section 5 to show the advantages of the proposed semantic map. These real applications are used to describe how semantic maps improve the hierarchy of task planning and the accuracy of robot localization. As QR code labels are crucial to obtaining semantic information, the design and identification of QR code and the orientation adjustment strategy are presented in Section 6. In Section 7, the experimental results are shown to validate the proposed method. The conclusions are included in Section 8.
Section snippets
Relevant works
In the past few decades, many researchers had considered the problem of building accurate maps of the environment with the data gathered by a mobile robot, which can be divided into two types: perceiving map building for navigation and cognizing map building for artificial intelligence.
Global topological map building with the function area segmentation strategy
To achieve object management, operational services, and interaction with human friendliness, a robot needs to express the environment referring to the space understanding of people. People use the concept “room” to describe the indoor environment. The relations of a “room” are described as a global recapitulative map for a person to remember the structure of the surroundings. Thus a robot should first create a global topological map with room segmentation to represent the space. A room is
The necessity of using QR code based artificial labels
Feature matching based on the SIFT algorithm is the basis of function area division and topology map building. SIFT is a very important and effective algorithm in computer vision, which has the feature of rotation and scale invariance, as well as high matching reliability. But we also found that in practice, the SIFT operator needs higher computational complexity, its processing speed is slow, and the matching algorithm is sensitive to nonlinear transformation of noise and image intensity.
The service task realization based on semantic hybrid map
It is a common task for a service robot to transmit and manage objects in an indoor environment. The transfer task is based on the condition that the robot knows the location of the objects. For example, if a person issues a task of taking a cold drink, the robot should search for the cold drink first. There is no ownership information of the objects in navigation oriented traditional maps, so the robot should use its vision to search the object all over the area. This searching method is named
Recognition of QR code based two-dimensional artificial object label
When the robot obtains an image, first of all, it carries out the two-dimensional artificial label’s recognition. If an artificial label is discovered, the robot will read the QR code information from the artificial label.
The circular detection is the basis of the artificial label’s recognition. The artificial labels designed here have more unique features. The background of the label is white, and the colorized rings have a higher saturation value. As the structure of the label is complex, the
The experiment of the semantic hybrid map building
Experiments are done using a Leader 1-DX mobile robot platform (shown as Fig. 10(a)) which was developed by our lab and equips laser sensor and Point Grey Research’s Bumblebee®2 Stereo Vision camera systems. It can collect indoor environmental information abundantly. The experimental environment was a quasi-structured home environment shown in Fig. 10(b). The object labels were pasted on TV cabinets, sink, stove, beds and other items. In order to verify the adaptability of artificial labels,
Conclusions
Many robotics researchers have paid attention to semantic map building nowadays. Object identification and function cognition in the environment is the key to semantic adding of the above-mentioned map. However, the problem of transforming the low level natural character to the advanced semantic character has not been completely solved by only using the image process algorithm to identify natural features. Thus it is difficult to obtain the semantic information of objects and rooms’ function &
Acknowledgments
This work has partly been supported by the National Nature Science Foundation under Grant 61075092, National Youth Foundation under Grant 61203330 and 61104009, Shandong Nature Science Foundation under Grant ZR2012FM031, ZR2011FM007 and ZR2010FM011, and the Innovation Foundation of Shandong University2011JC017 and 2012TS078, Shandong Postdoctoral Innovation Foundation201203058. The authors also acknowledge the anonymous reviewers and the receiving Editor for many constructive comments.
Wu Hao was born in Jinan (China) in 1972. She received her M.S. and Ph.D. in Control Theory and Control Engineering from Shandong University, in 1997 and 2011, respectively. Currently, she is an associate professor at Shandong University. Her research mainly focuses on the navigation of service robots, being the author of about 20 journal and conference papers.
References (33)
- et al.
Towards semantic maps for mobile robots
Robotics and Autonomous System
(2008) - et al.
Subjective local maps for hybrid metric-topological SLAM
Robotics and Autonomous Systems
(2009) - et al.
Cognitive maps for mobile robots-an object based approach
Robotics and Autonomous Systems
(2007) - et al.
Curious George: an attentive semantic robot
Robotics and Autonomous Systems
(2008) - et al.
Towards 3D Point cloud based object maps for household environments
Robotics and Autonomous Systems
(2008) - et al.
Semantic mapping using mobile robots
IEEE Transactions on Robotics
(2008) - H. Zender, P. Jensfelt, O. Martinez-Mozos, G.-J. Kruijff, W. Burgard, An integrated robotic system for spatial...
- Tingqi Wang, Qijun Chen, Object semantic map representation for indoor mobile robots, in: Proceedings of the 2011...
- R. Katsuki, J. Ota, T. Mizuta, T. Kito, T. Arai, et al. Design of an artificial mark to determine 3D pose by monocular...
- R. Katsuki, J. Ota, Y. Tamura, T. Mizuta, T. Kito, et al. Handling of objects with marks by a robot, in: Proceedings of...
Improved techniques for grid mapping with Rao–Blackwellized particle filters
IEEE Transactions on Robotics
Toward a unified Bayesian approach to hybrid metric-topological SLAM
IEEE Transactions on Robotics
Cognitive maps in rats and men
Psychological Review
Cited by (26)
A framework for multi-robot control in execution of a Swarm Production System
2023, Computers in IndustryExploring the cognitive process for service task in smart home: A robot service mechanism
2020, Future Generation Computer SystemsCitation Excerpt :The navigation controller is the main medium to link service robot and IS, which is used for transporting and updating the environment information such as semantic map, environmental map, and planning path. Remarkable, the global environment map and semantic map include room function, object properties and their affiliation [51], which is growth and portability, and is used to assist robot cognitive environment. Also, the planning algorithms are encapsulated in navigation controller.
A novel mobile robot localization approach based on topological maps using classification with reject option in omnidirectional images
2017, Expert Systems with ApplicationsCitation Excerpt :Other methods are a combination of both geometric and topological methods. Good examples of this hybrid approach are presented in Rady (2016); Wu et al. (2014). In the outdoor environment, navigation systems such as the Global Positioning System (GPS) provide precise positions and can be applied in mobile robot localization.
Dynamic Path Planning Algorithm for Unmanned Ship Based on Deep Reinforcement Learning
2022, Communications in Computer and Information Science
Wu Hao was born in Jinan (China) in 1972. She received her M.S. and Ph.D. in Control Theory and Control Engineering from Shandong University, in 1997 and 2011, respectively. Currently, she is an associate professor at Shandong University. Her research mainly focuses on the navigation of service robots, being the author of about 20 journal and conference papers.
Tian Guo-hui was born in Hejian (China) in 1969. He received his M.S. in Industry Automation from Shandong University in 1993, and his Ph.D. in Automatic Control Theory and Application from Dongbei University in 1997. He was a postdoctoral researcher at the Engineering Department of Tokyo University from 2003 to 2005. Currently, he is a professor and a doctoral tutor at Shandong University. His research mainly focuses on service robots and smart space. He is the author of about 40 journal and conference papers.
Li Yan was born in Jinan (China) in 1981. He received his Ph.D. in math from Shandong University in 2008. He has a postdoctoral research at the Electrical and Computer Engineering Department of Utah State University from 2009 to 2010. Currently, he is an associate professor at Shandong University. His research mainly focuses on service robots, being the author of about 30 journal and conference papers.
Zhou Feng-yu was born in Shandong (China) in 1969. He received his M.S. in Automatic Control from Shandong Science and Technology University in 1999, and his Ph.D. in Control Theory and Control Engineering from Tianjin University in 2008. Currently, he is a professor at Shandong University. His research mainly focuses on service robots and smart space. He is the author of about 40 journal and conference papers.
Duan Peng was born in Shandong (China) in 1988. He received his M.S. in Automatic Control from Shandong University in 2012. He is working towards a Ph.D. in the Department of Control Science and Engineering, Shandong University, on the subject of map build of on service robots.