Semantic mapping for mobile robotics tasks: A survey

https://doi.org/10.1016/j.robot.2014.12.006Get rights and content

Highlights

  • Two level navigation.

  • Cognitive navigation.

  • Spatial semantics.

Abstract

The evolution of contemporary mobile robotics has given thrust to a series of additional conjunct technologies. Of such is the semantic mapping, which provides an abstraction of space and a means for human–robot communication. The recent introduction and evolution of semantic mapping motivated this survey, in which an explicit analysis of the existing methods is sought. The several algorithms are categorized according to their primary characteristics, namely scalability, inference model, temporal coherence and topological map usage. The applications involving semantic maps are also outlined in the work at hand, emphasizing on human interaction, knowledge representation and planning. The existence of publicly available validation datasets and benchmarking, suitable for the evaluation of semantic mapping techniques is also discussed in detail. Last, an attempt to address open issues and questions is also made.

Introduction

The above quoted metaphor was used by the coiner of the phrase logical geography in his attempt to elucidate the term  [1], however today’s robotics specialists have realized that they face the same problem as the local villagers, yet the other way round. Nowadays one may argue that the problem of simultaneously localization and mapping (SLAM) has been solved, still the output of such a process is only perceivable by a man bearing compass and units of measurement. Accordingly, contemporary mobile robots behave like machine cartographers, unable to liaise with local villagers, that is the human inhabitants, who know by wont to navigate through the own environment. Thus, the majority of the existing mapping approaches aim to construct a globally consistent metric map of the robot’s operating environment. The robots bear state of the art instrumentation that allows, on the one hand, the construction of the map and, on the other hand, the own localization with respect to this map and, thus, to determine their global pose with remarkable accuracy. Based on this capability, the robots can plan a path and navigate towards a goal, which should be also a specified metric position in the global map reference frame. Howbeit, for a robot to apprehend the environment the way a human does and, consequently, to lead a stranger from place to place, a different skill than any geometrical map can provide is required. The robots to come should be endowed with capacities to understand their surroundings in a human-centric term, i.e. to be able to tell the difference between a room and a corridor or to discriminate the different functionality a kitchen and a living room have. Therefore, the formation of maps augmented by semasiological attributes involving human concepts, such as types of rooms, objects and their spatial arrangement, is considered a compulsory attribute for the future robots that should be designed to operate in environments inhabited by humans.

A solution to this problem is offered by semantic mapping, a qualitative description of the robot’s surroundings, aiming to augment the navigation capabilities and the task-planning, as well as to bridge the gap in human–robot interaction (HRI), see e.g.  [2], [3], [4]. Especially the work in  [4] addresses semantic mapping with emphasis on HRI by using natural language, thus enabling the most direct way for robots to socialize with humans. Thence, semantic mapping is a flourished pioneering area encouraging the elaboration of several doctoral dissertations  [5], [6]. The term semantic derives from the Greek word σημαντικòς[sēmantikos], standing for significant, which in turn derives from the verb σημαὶνϵιν[sēmainein], meaning to signify, that successively stems from the noun ση̃μα[sēma], that is sign. Thus, semantics is related to the study between signs and the things to which they refer, that is their meaning. The latter is oriented to the identification of the way that two or more entities interact, behave towards, and deal with each other  [7]. Thereby, the semantic mapping targets to the identification and the record of the signs and the symbols that contain meaningful concepts for humans, during the robot’s wander in human-inhabited areas. Consequently, a semantic map is an enhanced representation of the environment, which entails both geometrical information and high level qualitative features. Speculating the ability of the artificial agents to semantically perceive the own environment and accurately recall the learned spatial memories, the fundamental communication link between human and robots can be established. Therefore, for a successful HRI the robots must retain cognitive interpretation capacities about space, i.e. they should involve semantic attributes about the objects and the places encountered, in association with the geometrical perception of the surroundings. Moreover, the semantic information existing in the abient need to be organized in a such a fashion that the artificial agent can appropriately perceive and represent its environment. The most suitable way to organize all these information is by means of a map, namely a semantic map. Due to the fact that contemporary robots use to navigate in their environments by computing their pose within metric maps, the vast amount of the semantic mapping methods reported in the literature use these metric maps to add semantic information on top of it  [2], [4]. Therefore, a semantic map comprises high level features that model the human concepts about places, objects, shapes and even the relationships of all these, whilst a metric map retains all those geometrical features the robot should be aware of in order to safely navigate within its surroundings. Yet, it should be further noted that works have been reported on semantic mapping, which do not use a metric map to determine the type of a place, specially the ones using vision  [8], [9].

The goal of the review paper in hand is to provide insights of the semantic mapping, to study the distinct components encompassing, to give a categorization of the related literature, to mention the possible applications in mobile robotics and, lastly, to refer to the methods and databases available for benchmarking. In order to support this goal, a quality-based taxonomy of the existing mapping strategies is attempted here, which should highlight the dominant attributes such methods retain. An illustrative representation of the described taxonomy of the most frequent components the semantic mapping approaches possess is depicted in Fig. 1. The primary characteristics constitute the condiciones sine quibus non a method producing a complete semantic map should satisfy. Of such are the modalities utilized to reason about the observed scene constituting an element apt to distinguish the abundance of different methods. In particular in many methods only single cues–e.g. objects–are utilized to infer about a place, while some other methodologies exploit multiple cues–such as objects, places and shapes–to produce semasiological clues about an area. Another frequented feature in many semantic mapping techniques is the temporal coherence such a map reveals, which renders it useful for high-level activities, viz. task planning or HRI. An additional important attribute a typical semantic mapping method possesses is the existence of a respective topological map, that is an abstraction of the explored environment in terms of a graph. The nodes of such a graph are organized in a geometrical manner, so as to simultaneously preserve conceptual knowledge about the explored places. These graphs could be either unconstrained ones retaining only geometrical characteristics or they could possess several constrains in accordance with the semantic attributes that they enclose. The existence of a 2D or a 3D metric map of the explored environment–either indoors or outdoors–is a complementary component, which frequently supplements the attributes implemented by the semasiological methods. According to the scale, to which each method is expanded, the metric map could be either a single scene or a progressively created map, that is the pose is referred to a local or a global coordinate system, respectively.

Section snippets

Antecedents

Among the several modalities employed in robot navigation, vision is the most dominant. This is mainly due to the fact that scholars are able to straightforwardly replicate the own vision-based navigation experiences onto their experimental agents. The first twenty years of vision-based robot navigation are surveyed in  [10]. This work was conducted over one decade ago and it concludes that although then (2002) there were an adequate accumulated expertise to send a mobile robot from one

Scale based categorization

Owing to the fact that in many occasions a semantic map is built on top of a metric one, a straightforward clustering of the existing techniques can be based on the scale the underlying method retains. Thus, semantic mapping paradigms have been employed both in indoors and outdoors cases. Moreover, the methods developed for indoors situations are further distinguished into single-scene and large-scale ones. The single-scene class gleans those methods that reason about an instance frame with

Applications

How do you put into practice semantic maps? This section intends to review the application areas reported hitherto. The principal ambition of semantic mapping, as outlined earlier in this survey, is to provide the robots with a depiction comprehensible by humans. Yet, with the aim to include a human in mobile robot tasks, remote information should be efficiently presented to human beings. The various works examined in this section aspire to cover this aspect from different perspectives. In

Benchmarking and validation datasets

Any semantic mapping computation set-up constitutes a complex system comprising multiple subordinate modules. Such a system typically retains an increased number of parameters that influence the overall accuracy of the employed methods. That have been said, the demand of various realistic datasets to be utilized both for development and assessment of the produced algorithms is of great importance. Moreover, the establishment of objective metrics is awkward, should we bear in mind that it

Open issues and questions

About ten years ago, in a survey on socially interactive robots  [128] it was deduced that it is of great moment for the robots to be compatible with the humans requirements, to match the application demands, to be understandable and to allow the interactional comfort the human expects. Although the introduction of semantic mapping in mobile robots have narrowed the distance in all these aspects since then, there is still a long way to go, as well as other open issues and questions are in need

Epilogue

From the plethora of the laborious research that has already been conducted towards this direction, it is revealed that the semantic mapping is an active and ceaselessly growing research area. Although it is a relatively new topic of interest, various aspects of it have emerged through the recent years. This survey intended to summarize and categorize the existing methods. Accordingly, the primary characteristics of the semantic mapping were outlined and clustered corresponding to the

Dr. Ioannis Kostavelis is a Postdoctoral Research Associate at the Democritus University of Thrace, Department of Production and Management Engineer in the Robotics and Automation Laboratory. He holds a diploma in Production and Management Engineering from the Democritus University of Thrace and an M.Sc. in Informatics from the Aristotle University of Thessaloniki. He fulfilled his Ph.D. studies under the hood of the Laboratory of Robotics and Automation, at the Department of Production and

References (129)

  • A. Swadzba et al.

    A detailed analysis of a new 3d spatial feature vector for indoor scene classification

    Robot. Auton. Syst.

    (2014)
  • P. Espinace et al.

    Indoor scene recognition by a mobile robot through adaptive object detection

    Robot. Auton. Syst.

    (2013)
  • R.B. Rusu et al.

    Towards 3d point cloud based object maps for household environments

    Robot. Auton. Syst.

    (2008)
  • O. Martinez~Mozos et al.

    Supervised semantic labeling of places using information extracted from sensor data

    Robot. Auton. Syst.

    (2007)
  • Z. Liu et al.

    Extracting semantic indoor maps from occupancy grids

    Robot. Auton. Syst.

    (2014)
  • S. Vasudevan et al.

    Cognitive maps for mobile robotsan object based approach

    Robot. Auton. Syst.

    (2007)
  • A. Rituerto et al.

    Semantic labeling for indoor topological mapping using a wearable catadioptric system

    Robot. Auton. Syst.

    (2014)
  • S. Vasudevan et al.

    Bayesian space conceptualization and place classification for semantic maps in mobile robotics

    Robot. Auton. Syst.

    (2008)
  • A. Pronobis et al.

    The more you learn, the less you store: Memory-controlled incremental svm for visual place recognition

    Image Vis. Comput.

    (2010)
  • D. Meger et al.

    Curious George: An attentive semantic robot

    Robot. Auton. Syst.

    (2008)
  • G. Ryle

    Abstractions

    Dialogue

    (1962)
  • A. Pronobis et al.

    Large-scale semantic mapping and reasoning with heterogeneous modalities

  • Ó.M. Mozos

    Semantic Labeling of Places with Mobile Robots, vol.~61

    (2010)
  • A. Pronobis

    Semantic mapping with mobile robots

    (2011)
  • M.-W. Inc., Merriam-Webster’s collegiate dictionary, Merriam-Webster,...
  • A. Pronobis et al.

    A discriminative approach to robust visual place recognition

  • A. Ranganathan, F. Dellaert, Semantic modeling of places using objects, in: Proceedings of the 2007 Robotics: Science...
  • G.N. DeSouza et al.

    Vision for mobile robot navigation: A survey

    Trans. Pattern Anal. Mach. Intell.

    (2002)
  • S. Thrun et al.

    Probabilistic Robotics, vol.~1

    (2005)
  • Y.-D. Jian et al.

    Support-theoretic subgraph preconditioners for large-scale slam

  • G. Grisetti et al.

    Improved techniques for grid mapping with rao-blackwellized particle filters

    IEEE Trans. Robot.

    (2007)
  • D. Hahnel et al.

    An efficient fastslam algorithm for generating maps of large-scale cyclic environments from raw laser range measurements

  • H. Durrant-Whyte et al.

    Simultaneous localization and mapping: part I

    Robot. Autom. Mag., IEEE

    (2006)
  • T. Bailey et al.

    Simultaneous localization and mapping (slam): Part II

    Robot. Autom. Mag.

    (2006)
  • A. Angeli et al.

    Visual topological slam and global localization

  • J.-L. Blanco et al.

    A new approach for large-scale localization and mapping: Hybrid metric-topological slam

  • S. Bazeille et al.

    Incremental topo-metric slam using vision and robot odometry

  • C.W. Nielsen et al.

    Snapshots for semantic maps

  • I. Kostavelis et al.

    Learning the terrain and planning a collision-free trajectory for indoor post-disaster environments

  • P. Viswanathan et al.

    Automated place classification using object detection

  • P. Viswanathan et al.

    Place classification using visual object categorization and global information

  • B.C. Russell et al.

    Labelme: A database and web-based tool for image annotation

    Int. J. Comput. Vis.

    (2008)
  • A. Trevor et al.

    Efficient organized point cloud segmentation with connected components

  • O.M. Mozos et al.

    Categorization of indoor places using the kinect sensor

    Sensors

    (2012)
  • A. Nüchter et al.

    3d mapping with semantic knowledge

  • N. Blodow et al.

    Autonomous semantic mapping for robots performing everyday manipulation tasks in kitchen environments

  • R.B. Rusu et al.

    Model-based and learned semantic object labeling in 3d point cloud maps of kitchen environments

  • A.J. Trevor et al.

    Tables, counters, and shelves: Semantic mapping of surfaces in 3d

  • A.J. Trevor, J.G. Rogers, C. Nieto-Granda, H.I. Christensen, Feature-based mapping with grounded landmark and place...
  • A. Cowley et al.

    Rapid multi-robot exploration with topometric maps

  • Cited by (344)

    • Vision-IMU multi-sensor fusion semantic topological map based on RatSLAM

      2023, Measurement: Journal of the International Measurement Confederation
    View all citing articles on Scopus

    Dr. Ioannis Kostavelis is a Postdoctoral Research Associate at the Democritus University of Thrace, Department of Production and Management Engineer in the Robotics and Automation Laboratory. He holds a diploma in Production and Management Engineering from the Democritus University of Thrace and an M.Sc. in Informatics from the Aristotle University of Thessaloniki. He fulfilled his Ph.D. studies under the hood of the Laboratory of Robotics and Automation, at the Department of Production and Management Engineering, Democritus University of Thrace, under the supervision of Prof. Antonios Gasteratos. His research has been supported through several research projects funded by the European Space Agency, the European Commission and the Greek government. His current research interests include machine vision systems for robotic applications augmented with machine learning strategies, targeting on the construction of semantic maps suitable for high level robot navigation. More details about him are available at http://robotics.pme.duth.gr/kostavelis/.

    Antonios Gasteratos is an Associate Professor at the Department of Production and Management Engineering, Democritus University of Thrace (DUTH), Greece. He teaches the courses of Robotics, Automatic Control Systems, Measurements Technology and Electronics. He holds a B.Eng. and a Ph.D. from the Department of Electrical and Computer Engineering, DUTH, Greece. During 1999–2000 he was a Visiting Researcher at the Laboratory of Integrated Advanced Robotics (LIRA-Lab), DIST, University of 34 Genoa, Italy. He has served as a reviewer for numerous Scientific Journals and International Conferences. His research interests are mainly in mechatronics and in robot vision. He has published more than 160 papers in books, journals and conferences. He is a senior member of the IEEE. More details about him are available at http://robotics.pme.duth.gr/.

    View full text