Inferring robot goals from violations of semantic knowledge

Abstract

A growing body of literature shows that endowing a mobile robot with semantic knowledge and with the ability to reason from this knowledge can greatly increase its capabilities. In this paper, we present a novel use of semantic knowledge, to encode information about how things should be, i.e. norms, and to enable the robot to infer deviations from these norms in order to generate goals to correct these deviations. For instance, if a robot has semantic knowledge that perishable items must be kept in a refrigerator, and it observes a bottle of milk on a table, this robot will generate the goal to bring that bottle into a refrigerator. The key move is to properly encode norms in an ontology so that each norm violation results in a detectable inconsistency. A goal is then generated to bring the world back in a consistent state, and a planner is used to transform this goal into actions. Our approach provides a mobile robot with a limited form of goal autonomy: the ability to derive its own goals to pursue generic aims. We illustrate our approach in a full mobile robot system that integrates a semantic map, a knowledge representation and reasoning system, a task planner, and standard perception and navigation routines.

Introduction

Mobile robots intended for service and personal use are being increasingly endowed with the ability to represent and use semantic knowledge about the environment where they operate [1], [2]. This knowledge encodes general information about the entities in the world and their relations, for instance, that a kitchen is a type of room which is used for cooking and which typically contains a refrigerator, a stove, and a sink; that milk is a perishable food; and that perishable food is stored in a refrigerator. Once this knowledge is available to a robot, it can be exploited to better understand the environment or plan actions [3], [4], [5], [6], [7], assuming of course that this knowledge is a faithful representation of the properties of the environment. There is, however, an interesting issue which has received less attention so far: what happens if this knowledge turns out to be in conflict with the robot’s observations?

Suppose for concreteness that the robot observes a bottle of milk lying on a table. This observation conflicts with the semantic knowledge that milk, a perishable item, should be stored in a refrigerator. The robot has three options to resolve this contradiction: (a) to verify its perceptions, e.g., by looking for clues which may indicate that the observed object is not a milk bottle; (b) to update its semantic knowledge base, e.g., by adding a subclass of milk that is not perishable; or (c) to modify the environment, e.g., by putting the bottle in the refrigerator. While some works have addressed the first two options [6], [8], [9], [10], the last one has not received much attention so far. Interestingly, the last option leverages the distinctive ability of robots to modify their physical environment. The goal of this paper is to investigate this option.

Our investigation proceeds in four steps. First, we address the problem of how to encode normative knowledge in a robot, that is, semantic knowledge on how things should be. For this we use a hybrid semantic map  [8], which combines traditional robot maps with description logics  [11], and enrich it with the notion of “normative” concepts. Second, we study how the robot can automatically detect violations of its normative knowledge, and isolate the causes of these violations. For this we rely on our encoding of norms to transform norm violations into logical inconsistencies. This allows us to use the mechanisms of description logics to detect an inconsistency and to identify the objects and relations which are involved in it. Third, we discuss how to go from the detection of a violation to a recovery strategy. We define a mechanism to automatically generate a goal, which represents the intention to achieve a specific state of the world that satisfies the violated norm. If this goal is fed to a standard task planner, it will result in a plan to execute the actions needed to bring the world back to a consistent state — provided of course that the robot has the right action repertoire.

Troubles rarely come alone, so our fourth and last step is to extend the above mechanism to the case of multiple violations of norms. This extension is not straightforward because of several reasons: (i) standard inference systems based on tableau methods do not behave well with multiple, simultaneous inconsistencies; (ii) violations may be inter-dependent, and solving one violation may produce another one; and (iii) some violations may be more important than others. We propose an algorithm that alternates violation detection, goal generation, and simulated recovery until a feasible sequence of recovery plans is found, also taking into account user defined priorities. This algorithm enables a mobile robot to generate a “to do list” in order to keep its workspace, as it perceives it, consistent with respect to a set of given norms.

In the rest of this paper we describe the above steps in more detail. We complement the formal descriptions with algorithms and examples to allow other researchers to reproduce our results. We also report a proof-of-concept experiment that shows the concrete applicability of our approach to real robotic systems. It should be emphasized that in this work we focus on the detection of norms violations and on the goal inference mechanism: the development of perception and action capabilities and the possible use of semantic knowledge in that context are beyond the scope of this paper.

In the next section we review some related work. Section  3 introduces our semantic map. In Section  4 we present the above first three steps, while Section  5 deals with the extension to the case of multiple concurrent norm violations. Section  6 reports the proof-of-concept experiment. We then discuss our results in Section  7 and conclude.¹