Semantic mapping for mobile robotics tasks: A survey
Introduction
The above quoted metaphor was used by the coiner of the phrase logical geography in his attempt to elucidate the term [1], however today’s robotics specialists have realized that they face the same problem as the local villagers, yet the other way round. Nowadays one may argue that the problem of simultaneously localization and mapping (SLAM) has been solved, still the output of such a process is only perceivable by a man bearing compass and units of measurement. Accordingly, contemporary mobile robots behave like machine cartographers, unable to liaise with local villagers, that is the human inhabitants, who know by wont to navigate through the own environment. Thus, the majority of the existing mapping approaches aim to construct a globally consistent metric map of the robot’s operating environment. The robots bear state of the art instrumentation that allows, on the one hand, the construction of the map and, on the other hand, the own localization with respect to this map and, thus, to determine their global pose with remarkable accuracy. Based on this capability, the robots can plan a path and navigate towards a goal, which should be also a specified metric position in the global map reference frame. Howbeit, for a robot to apprehend the environment the way a human does and, consequently, to lead a stranger from place to place, a different skill than any geometrical map can provide is required. The robots to come should be endowed with capacities to understand their surroundings in a human-centric term, i.e. to be able to tell the difference between a room and a corridor or to discriminate the different functionality a kitchen and a living room have. Therefore, the formation of maps augmented by semasiological attributes involving human concepts, such as types of rooms, objects and their spatial arrangement, is considered a compulsory attribute for the future robots that should be designed to operate in environments inhabited by humans.
A solution to this problem is offered by semantic mapping, a qualitative description of the robot’s surroundings, aiming to augment the navigation capabilities and the task-planning, as well as to bridge the gap in human–robot interaction (HRI), see e.g. [2], [3], [4]. Especially the work in [4] addresses semantic mapping with emphasis on HRI by using natural language, thus enabling the most direct way for robots to socialize with humans. Thence, semantic mapping is a flourished pioneering area encouraging the elaboration of several doctoral dissertations [5], [6]. The term semantic derives from the Greek word , standing for significant, which in turn derives from the verb , meaning to signify, that successively stems from the noun , that is sign. Thus, semantics is related to the study between signs and the things to which they refer, that is their meaning. The latter is oriented to the identification of the way that two or more entities interact, behave towards, and deal with each other [7]. Thereby, the semantic mapping targets to the identification and the record of the signs and the symbols that contain meaningful concepts for humans, during the robot’s wander in human-inhabited areas. Consequently, a semantic map is an enhanced representation of the environment, which entails both geometrical information and high level qualitative features. Speculating the ability of the artificial agents to semantically perceive the own environment and accurately recall the learned spatial memories, the fundamental communication link between human and robots can be established. Therefore, for a successful HRI the robots must retain cognitive interpretation capacities about space, i.e. they should involve semantic attributes about the objects and the places encountered, in association with the geometrical perception of the surroundings. Moreover, the semantic information existing in the abient need to be organized in a such a fashion that the artificial agent can appropriately perceive and represent its environment. The most suitable way to organize all these information is by means of a map, namely a semantic map. Due to the fact that contemporary robots use to navigate in their environments by computing their pose within metric maps, the vast amount of the semantic mapping methods reported in the literature use these metric maps to add semantic information on top of it [2], [4]. Therefore, a semantic map comprises high level features that model the human concepts about places, objects, shapes and even the relationships of all these, whilst a metric map retains all those geometrical features the robot should be aware of in order to safely navigate within its surroundings. Yet, it should be further noted that works have been reported on semantic mapping, which do not use a metric map to determine the type of a place, specially the ones using vision [8], [9].
The goal of the review paper in hand is to provide insights of the semantic mapping, to study the distinct components encompassing, to give a categorization of the related literature, to mention the possible applications in mobile robotics and, lastly, to refer to the methods and databases available for benchmarking. In order to support this goal, a quality-based taxonomy of the existing mapping strategies is attempted here, which should highlight the dominant attributes such methods retain. An illustrative representation of the described taxonomy of the most frequent components the semantic mapping approaches possess is depicted in Fig. 1. The primary characteristics constitute the condiciones sine quibus non a method producing a complete semantic map should satisfy. Of such are the modalities utilized to reason about the observed scene constituting an element apt to distinguish the abundance of different methods. In particular in many methods only single cues–e.g. objects–are utilized to infer about a place, while some other methodologies exploit multiple cues–such as objects, places and shapes–to produce semasiological clues about an area. Another frequented feature in many semantic mapping techniques is the temporal coherence such a map reveals, which renders it useful for high-level activities, viz. task planning or HRI. An additional important attribute a typical semantic mapping method possesses is the existence of a respective topological map, that is an abstraction of the explored environment in terms of a graph. The nodes of such a graph are organized in a geometrical manner, so as to simultaneously preserve conceptual knowledge about the explored places. These graphs could be either unconstrained ones retaining only geometrical characteristics or they could possess several constrains in accordance with the semantic attributes that they enclose. The existence of a 2D or a 3D metric map of the explored environment–either indoors or outdoors–is a complementary component, which frequently supplements the attributes implemented by the semasiological methods. According to the scale, to which each method is expanded, the metric map could be either a single scene or a progressively created map, that is the pose is referred to a local or a global coordinate system, respectively.
Section snippets
Antecedents
Among the several modalities employed in robot navigation, vision is the most dominant. This is mainly due to the fact that scholars are able to straightforwardly replicate the own vision-based navigation experiences onto their experimental agents. The first twenty years of vision-based robot navigation are surveyed in [10]. This work was conducted over one decade ago and it concludes that although then (2002) there were an adequate accumulated expertise to send a mobile robot from one
Scale based categorization
Owing to the fact that in many occasions a semantic map is built on top of a metric one, a straightforward clustering of the existing techniques can be based on the scale the underlying method retains. Thus, semantic mapping paradigms have been employed both in indoors and outdoors cases. Moreover, the methods developed for indoors situations are further distinguished into single-scene and large-scale ones. The single-scene class gleans those methods that reason about an instance frame with
Applications
How do you put into practice semantic maps? This section intends to review the application areas reported hitherto. The principal ambition of semantic mapping, as outlined earlier in this survey, is to provide the robots with a depiction comprehensible by humans. Yet, with the aim to include a human in mobile robot tasks, remote information should be efficiently presented to human beings. The various works examined in this section aspire to cover this aspect from different perspectives. In
Benchmarking and validation datasets
Any semantic mapping computation set-up constitutes a complex system comprising multiple subordinate modules. Such a system typically retains an increased number of parameters that influence the overall accuracy of the employed methods. That have been said, the demand of various realistic datasets to be utilized both for development and assessment of the produced algorithms is of great importance. Moreover, the establishment of objective metrics is awkward, should we bear in mind that it
Open issues and questions
About ten years ago, in a survey on socially interactive robots [128] it was deduced that it is of great moment for the robots to be compatible with the humans requirements, to match the application demands, to be understandable and to allow the interactional comfort the human expects. Although the introduction of semantic mapping in mobile robots have narrowed the distance in all these aspects since then, there is still a long way to go, as well as other open issues and questions are in need
Epilogue
From the plethora of the laborious research that has already been conducted towards this direction, it is revealed that the semantic mapping is an active and ceaselessly growing research area. Although it is a relatively new topic of interest, various aspects of it have emerged through the recent years. This survey intended to summarize and categorize the existing methods. Accordingly, the primary characteristics of the semantic mapping were outlined and clustered corresponding to the
Dr. Ioannis Kostavelis is a Postdoctoral Research Associate at the Democritus University of Thrace, Department of Production and Management Engineer in the Robotics and Automation Laboratory. He holds a diploma in Production and Management Engineering from the Democritus University of Thrace and an M.Sc. in Informatics from the Aristotle University of Thessaloniki. He fulfilled his Ph.D. studies under the hood of the Laboratory of Robotics and Automation, at the Department of Production and
References (129)
- et al.
Learning spatially semantic representations for cognitive robot navigation
Robot. Auton. Syst.
(2013) - et al.
Conceptual spatial representations for indoor mobile robots
Robot. Auton. Syst.
(2008) Modeling spatial knowledge
Cogn. Sci.
(1978)- et al.
A robot exploration and mapping strategy based on a semantic hierarchy of spatial representations
Robot. Auton. Syst.
(1991) The spatial semantic hierarchy
Artif. Intell.
(2000)- et al.
Map-based navigation in mobile robots: I. A review of localization strategies
Cogn. Syst. Res.
(2003) - et al.
Map-based navigation in mobile robots: II. A review of map-learning and path-planning strategies
Cogn. Syst. Res.
(2003) Learning metric-topological maps for indoor mobile robot navigation
Artif. Intell.
(1998)- et al.
Collision risk assessment for autonomous robots by offline traversability learning
Robot. Auton. Syst.
(2012) - et al.
Robots in the kitchen: Exploiting ubiquitous sensing and actuation
Robot. Auton. Syst.
(2008)
A detailed analysis of a new 3d spatial feature vector for indoor scene classification
Robot. Auton. Syst.
Indoor scene recognition by a mobile robot through adaptive object detection
Robot. Auton. Syst.
Towards 3d point cloud based object maps for household environments
Robot. Auton. Syst.
Supervised semantic labeling of places using information extracted from sensor data
Robot. Auton. Syst.
Extracting semantic indoor maps from occupancy grids
Robot. Auton. Syst.
Cognitive maps for mobile robotsan object based approach
Robot. Auton. Syst.
Semantic labeling for indoor topological mapping using a wearable catadioptric system
Robot. Auton. Syst.
Bayesian space conceptualization and place classification for semantic maps in mobile robotics
Robot. Auton. Syst.
The more you learn, the less you store: Memory-controlled incremental svm for visual place recognition
Image Vis. Comput.
Curious George: An attentive semantic robot
Robot. Auton. Syst.
Abstractions
Dialogue
Large-scale semantic mapping and reasoning with heterogeneous modalities
Semantic Labeling of Places with Mobile Robots, vol.~61
Semantic mapping with mobile robots
A discriminative approach to robust visual place recognition
Vision for mobile robot navigation: A survey
Trans. Pattern Anal. Mach. Intell.
Probabilistic Robotics, vol.~1
Support-theoretic subgraph preconditioners for large-scale slam
Improved techniques for grid mapping with rao-blackwellized particle filters
IEEE Trans. Robot.
An efficient fastslam algorithm for generating maps of large-scale cyclic environments from raw laser range measurements
Simultaneous localization and mapping: part I
Robot. Autom. Mag., IEEE
Simultaneous localization and mapping (slam): Part II
Robot. Autom. Mag.
Visual topological slam and global localization
A new approach for large-scale localization and mapping: Hybrid metric-topological slam
Incremental topo-metric slam using vision and robot odometry
Snapshots for semantic maps
Learning the terrain and planning a collision-free trajectory for indoor post-disaster environments
Automated place classification using object detection
Place classification using visual object categorization and global information
Labelme: A database and web-based tool for image annotation
Int. J. Comput. Vis.
Efficient organized point cloud segmentation with connected components
Categorization of indoor places using the kinect sensor
Sensors
3d mapping with semantic knowledge
Autonomous semantic mapping for robots performing everyday manipulation tasks in kitchen environments
Model-based and learned semantic object labeling in 3d point cloud maps of kitchen environments
Tables, counters, and shelves: Semantic mapping of surfaces in 3d
Rapid multi-robot exploration with topometric maps
Cited by (344)
Vision-IMU multi-sensor fusion semantic topological map based on RatSLAM
2023, Measurement: Journal of the International Measurement ConfederationA novel framework to improve motion planning of robotic systems through semantic knowledge-based reasoning
2023, Computers and Industrial EngineeringLifelong mapping in the wild: Novel strategies for ensuring map stability and accuracy over time evaluated on thousands of robots
2023, Robotics and Autonomous SystemsLive semantic data from building digital twins for robot navigation: Overview of data transfer methods
2023, Advanced Engineering Informatics
Dr. Ioannis Kostavelis is a Postdoctoral Research Associate at the Democritus University of Thrace, Department of Production and Management Engineer in the Robotics and Automation Laboratory. He holds a diploma in Production and Management Engineering from the Democritus University of Thrace and an M.Sc. in Informatics from the Aristotle University of Thessaloniki. He fulfilled his Ph.D. studies under the hood of the Laboratory of Robotics and Automation, at the Department of Production and Management Engineering, Democritus University of Thrace, under the supervision of Prof. Antonios Gasteratos. His research has been supported through several research projects funded by the European Space Agency, the European Commission and the Greek government. His current research interests include machine vision systems for robotic applications augmented with machine learning strategies, targeting on the construction of semantic maps suitable for high level robot navigation. More details about him are available at http://robotics.pme.duth.gr/kostavelis/.
Antonios Gasteratos is an Associate Professor at the Department of Production and Management Engineering, Democritus University of Thrace (DUTH), Greece. He teaches the courses of Robotics, Automatic Control Systems, Measurements Technology and Electronics. He holds a B.Eng. and a Ph.D. from the Department of Electrical and Computer Engineering, DUTH, Greece. During 1999–2000 he was a Visiting Researcher at the Laboratory of Integrated Advanced Robotics (LIRA-Lab), DIST, University of 34 Genoa, Italy. He has served as a reviewer for numerous Scientific Journals and International Conferences. His research interests are mainly in mechatronics and in robot vision. He has published more than 160 papers in books, journals and conferences. He is a senior member of the IEEE. More details about him are available at http://robotics.pme.duth.gr/.