Elsevier

Knowledge-Based Systems

Volume 204, 27 September 2020, 106174
Knowledge-Based Systems

Home service robot task planning using semantic knowledge and probabilistic inference

https://doi.org/10.1016/j.knosys.2020.106174Get rights and content

Abstract

In the face of unstructured home environment, home service robots are inevitably confronted with uncertainty and incompleteness of environment information. How to make the home service robot obtain enough environment information and plan a discrete sequence of actions through task planning is the key problem of robot intelligence. In this paper, a hierarchical task network based on semantic knowledge and probabilistic inference method is proposed. We use the object location ontology, the location relation between dynamic and static objects to build semantic knowledge of home environment, and build the probability model between dynamic and static objects, as well as between static objects and home scenes. The location of the object is determined by the semantic knowledge and the probability model. Hierarchical task network is selected as an engine of task planner, which can be provided with the location information to improve the autonomy and effectiveness of robot task planning. In order to prevent task execution failure and enhance the adaptability of robot to unstructured home environment, a mechanism of task execution diagnosis and replanning is designed. Experimental results in simulation and real home environment demonstrate that our method can effectively improve the performance of service robot task planning and generate better task execution sequence.

Introduction

With the development of robotics, home service robots have gradually entered homes to provide services for human beings, and become a new type of intelligence device for improving the quality of human life, as well as a good partner of human beings. The problem of service robot task planning attracts the attention of many researchers, who have done a lot of research on task planning of home service robot [1], [2], [3]. The service robot faces highly unstructured home environment, as well as dynamic and static objects, which increase the difficulty of service robot task planning.

In order to complete the task planning of service robot, on the one hand, the task planning is to plan high-level actions such as grasp, pour and place. On the other hand, it needs to plan low-level details of motor control, for example, joint angle, hand posture, approach direction for the grasp. In this paper, we mainly focus on high-level actions and achieve a sequence of actions to guide the robot to complete a complex task and achieve the desired goal in unstructured home environment. As home service robots are starting to perform everyday manipulation tasks, such as sending water, cleaning up, setting a table, making food, food heating, delivering objects, etc., it is necessary that their control programs become knowledgeable than they are today. For everyday manipulation task, robot needs to know or reason about where the target objects related to manipulation task, and how to conduct their actions on objects in an appropriate manner to accomplish these tasks. These inferences can only be made when robots can acquire the necessary knowledge, including knowledge about the appearance of objects, their attributes, where they can be found, and what may happen if specific operations are performed on them.

Most task planning generation methods [4], [5], [6] focus on the deterministic information of the world environment, such as the object location. However, there is a certain degree of uncertainty and dynamics in home environment, especially for some operable objects, which can be divided into dynamic objects and static objects. Dynamic objects are usually small and easy to be moved by people or robots, such as cups, books, bowls. While static objects are usually large objects whose locations are generally unchanged, such as beds, sofas and refrigerators. In home environment, the location of these static and dynamic objects is not a definite model, but a dynamic one. Although there is a certain location relation between the object and the home scene in home environment, this relation is generally dynamic, not a fixed model. For example, In the real home environment, it is very likely that a robot will see a refrigerator and a sink when entering a kitchen, although the probability of the service robot observing a bed and a TV is obviously low. Task planning is a complex process with uncertainty. Therefore, the higher-level form of probabilistic reasoning with semantic knowledge should be proposed to derive probabilistic information that is related to the existing objects in the home environment, predicting their location and the relation between the dynamic objects and static objects.

Therefore, in order to enhance the automaticity of task planning of home service robot and make it adapt to unstructured home environment better, we propose a hierarchical task network(HTN) [7] planning model that utilizes semantic knowledge and probabilistic inference to assist task planning in the home environment. In our proposed method, task planning utilizes semantic knowledge and involves the uncertainty of the location of the object in the home environment. Our contribution is presented as follows:

  • (1)

    We establish the object location ontology of home environment and the dynamic and static objects relation model. Because there are uncertainties between static objects and home scenes, as well as between dynamic and static objects, so, we establish the probability model between them.

  • (2)

    The semantic inference rules are established to infer the object location and the relation between dynamic and static objects, and the obtained information is used as the input of task planner to assist the service robot in task planning.

  • (3)

    When home service robot performs tasks in the real home environment, it may fail because of the dynamic characteristics of home environment. Therefore, a diagnosis mechanism of planning execution is designed to realize the replanning of robot tasks and complete the given tasks.

The rest of this paper is organized as follows. The related work is shown in Section 2. Section 3 describes the details of our method. Section 4 describes the experimental results. Finally, Section 5 describes the conclusion and future work.

Section snippets

Related work

Most researches that have dealt with generation of task planning sequence for mobile robot from an initial state to a goal state. Previous studies have focused on task planning on the condition of known or determined state of the world environment. World representation based on semantic knowledge is probably one of the most important issues in the robotics literature. This environment can be described by a predefined semantic model, on which task planning can be accomplished by determined

Our approach

Automatic task planning is a branch of artificial intelligence, which studies the planning process in the computing environment. In home environment, planning process aims to choose and order a sequence of actions to finish a task, a key issue for home service robots. According to commands of user, home service robots conduct a repeated execution of admissible actions belonging to a set A from an initial state S0 to a goal state Sg. So a task planning problem P={S0,Sg,A} can be addressed by

Experiments

This section evaluates the effectiveness of our proposed task planning method in home environment. It is generally very difficult to compare our method with other state of the art methods. This is because that different task planning methods are developed for different environments, such as known/unknown, small/large, market/office/home, and so, it is hard to evaluate which algorithm has better performance. Moreover, there is no same and well-established environment or robot system for

Conclusion and future work

For the unstructured home environment, we manage to establish hierarchical information network based on semantic knowledge and probabilistic inference model of home environment which is applied to assist home service robot in task planning. Semantic knowledge composed of object location ontology, the location relation between dynamic and static objects. To deal with uncertainty in home environment, we also build the probability model between dynamic and static objects, as well as between static

CRediT authorship contribution statement

Zhongli Wang: Conceptualization, Methodology, Software, Writing - original draft, Writing - review & editing. Guohui Tian: Supervision. Xuyang Shao: Investigation, Software, Validation, Data curation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work is supported by the National Key R&D Program of China (2018YFB1307101) and the National Natural Science Foundation of China (U1813215).

References (40)

  • GeorgievskiI. et al.

    HTN planning: Overview, comparison, and beyond

    Artificial Intelligence

    (2015)
  • JiZ. et al.

    Towards automated task planning for service robots using semantic knowledge representation

  • DjezairiS. et al.

    Layered task planning for mobile manipulators

    Comput. Electr. Eng.

    (2019)
  • ChenT. et al.

    Neural task planning with AND-OR graph representations

    IEEE Trans. Multimed.

    (2018)
  • ErolK. et al.

    UMCP: a sound and complete procedure for hierarchical task-network planning

  • M. Bollini, J. Barry, D. Rus, Bakebot: Baking cookies with the pr2, in: The PR2 Workshop: Results, Challenges and...
  • BeetzM. et al.

    Robotic roommates making pancakes

  • MillerS. et al.

    A geometric approach to robotic laundry folding

    Int. J. Robot. Res.

    (2012)
  • Z. Xue, J. Zoellner, R. Dillmann, Automatic optimal grasp planning based on found contact points, in: IEEE/ASME Int....
  • M. Beetz, et al. Know rob 2.0 A 2nd generation knowledge processing framework for cognition-enabled robotic agents, in:...
  • Cited by (35)

    • Adaptive and intelligent robot task planning for home service: A review

      2023, Engineering Applications of Artificial Intelligence
      Citation Excerpt :

      The overall knowledge-based framework of object search for robot service planning that includes the knowledge layer, the planning layer, and the service layer is delineated in Fig. 13 Task re-planning strategies that consider changes in target position were designed to improve the adaptability of robots in the real world (Wang and Tian, 2022; Wang et al., 2020). The overall framework of the hybrid task planning approach (Wang and Tian, 2022) is shown in Fig. 14.

    • Hybrid offline and online task planning for service robot using object-level semantic map and probabilistic inference

      2022, Information Sciences
      Citation Excerpt :

      Because generation of action sequences is the focus of this paper, and object detection, object grasping method and navigation are not the focus of our research, so we select some classic algorithms to ensure that our experiments are carried out. In previous work [41], we counted the relations between dynamic and static objects. We add the category of dynamic objects on the original basis to carry out the experiment in this paper, as shown in Fig. 7.

    • Service planning oriented efficient object search: A knowledge-based framework for home service robot

      2022, Expert Systems with Applications
      Citation Excerpt :

      The advances in key technologies of robots, such as object detection (Cao, Liao, Song, Chen, & Li, 2020), robot navigation and localization (Kamil et al., 2017; Zhang, Wang, & Chen, 2019b), robot scene classification (Romero-González, Martínez-Gómez, García-Varea, & Rodriguez-Ruiz, 2017), and so on, have brought the promise for the service robot to execute daily tasks in the household environment. In the unstructured family environment, the robot is expected to have the ability of service planning (Galindo, Fernández-Madrigal, González, & Saffiotti, 2008; Kim et al., 2017; Wang, Tian, & Shao, 2020), which is still a challenging problem at present. The service planning is the first step of service execution.

    • Service skill improvement for home robots: Autonomous generation of action sequence based on reinforcement learning

      2021, Knowledge-Based Systems
      Citation Excerpt :

      Han [15] proposes a method for obtaining textual knowledge by integrating neural networks and knowledge graph. Wang [16] proposes a method of task planning with semantic knowledge and probabilistic inference as the assistance. Oswaldo [17] presents a NN-based model that extracts actions from narrative texts.

    View all citing articles on Scopus
    1

    Since 1880.

    View full text