Home service robot task planning using semantic knowledge and probabilistic inference
Introduction
With the development of robotics, home service robots have gradually entered homes to provide services for human beings, and become a new type of intelligence device for improving the quality of human life, as well as a good partner of human beings. The problem of service robot task planning attracts the attention of many researchers, who have done a lot of research on task planning of home service robot [1], [2], [3]. The service robot faces highly unstructured home environment, as well as dynamic and static objects, which increase the difficulty of service robot task planning.
In order to complete the task planning of service robot, on the one hand, the task planning is to plan high-level actions such as grasp, pour and place. On the other hand, it needs to plan low-level details of motor control, for example, joint angle, hand posture, approach direction for the grasp. In this paper, we mainly focus on high-level actions and achieve a sequence of actions to guide the robot to complete a complex task and achieve the desired goal in unstructured home environment. As home service robots are starting to perform everyday manipulation tasks, such as sending water, cleaning up, setting a table, making food, food heating, delivering objects, etc., it is necessary that their control programs become knowledgeable than they are today. For everyday manipulation task, robot needs to know or reason about where the target objects related to manipulation task, and how to conduct their actions on objects in an appropriate manner to accomplish these tasks. These inferences can only be made when robots can acquire the necessary knowledge, including knowledge about the appearance of objects, their attributes, where they can be found, and what may happen if specific operations are performed on them.
Most task planning generation methods [4], [5], [6] focus on the deterministic information of the world environment, such as the object location. However, there is a certain degree of uncertainty and dynamics in home environment, especially for some operable objects, which can be divided into dynamic objects and static objects. Dynamic objects are usually small and easy to be moved by people or robots, such as cups, books, bowls. While static objects are usually large objects whose locations are generally unchanged, such as beds, sofas and refrigerators. In home environment, the location of these static and dynamic objects is not a definite model, but a dynamic one. Although there is a certain location relation between the object and the home scene in home environment, this relation is generally dynamic, not a fixed model. For example, In the real home environment, it is very likely that a robot will see a refrigerator and a sink when entering a kitchen, although the probability of the service robot observing a bed and a TV is obviously low. Task planning is a complex process with uncertainty. Therefore, the higher-level form of probabilistic reasoning with semantic knowledge should be proposed to derive probabilistic information that is related to the existing objects in the home environment, predicting their location and the relation between the dynamic objects and static objects.
Therefore, in order to enhance the automaticity of task planning of home service robot and make it adapt to unstructured home environment better, we propose a hierarchical task network(HTN) [7] planning model that utilizes semantic knowledge and probabilistic inference to assist task planning in the home environment. In our proposed method, task planning utilizes semantic knowledge and involves the uncertainty of the location of the object in the home environment. Our contribution is presented as follows:
- (1)
We establish the object location ontology of home environment and the dynamic and static objects relation model. Because there are uncertainties between static objects and home scenes, as well as between dynamic and static objects, so, we establish the probability model between them.
- (2)
The semantic inference rules are established to infer the object location and the relation between dynamic and static objects, and the obtained information is used as the input of task planner to assist the service robot in task planning.
- (3)
When home service robot performs tasks in the real home environment, it may fail because of the dynamic characteristics of home environment. Therefore, a diagnosis mechanism of planning execution is designed to realize the replanning of robot tasks and complete the given tasks.
The rest of this paper is organized as follows. The related work is shown in Section 2. Section 3 describes the details of our method. Section 4 describes the experimental results. Finally, Section 5 describes the conclusion and future work.
Section snippets
Related work
Most researches that have dealt with generation of task planning sequence for mobile robot from an initial state to a goal state. Previous studies have focused on task planning on the condition of known or determined state of the world environment. World representation based on semantic knowledge is probably one of the most important issues in the robotics literature. This environment can be described by a predefined semantic model, on which task planning can be accomplished by determined
Our approach
Automatic task planning is a branch of artificial intelligence, which studies the planning process in the computing environment. In home environment, planning process aims to choose and order a sequence of actions to finish a task, a key issue for home service robots. According to commands of user, home service robots conduct a repeated execution of admissible actions belonging to a set from an initial state to a goal state . So a task planning problem can be addressed by
Experiments
This section evaluates the effectiveness of our proposed task planning method in home environment. It is generally very difficult to compare our method with other state of the art methods. This is because that different task planning methods are developed for different environments, such as known/unknown, small/large, market/office/home, and so, it is hard to evaluate which algorithm has better performance. Moreover, there is no same and well-established environment or robot system for
Conclusion and future work
For the unstructured home environment, we manage to establish hierarchical information network based on semantic knowledge and probabilistic inference model of home environment which is applied to assist home service robot in task planning. Semantic knowledge composed of object location ontology, the location relation between dynamic and static objects. To deal with uncertainty in home environment, we also build the probability model between dynamic and static objects, as well as between static
CRediT authorship contribution statement
Zhongli Wang: Conceptualization, Methodology, Software, Writing - original draft, Writing - review & editing. Guohui Tian: Supervision. Xuyang Shao: Investigation, Software, Validation, Data curation.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Acknowledgments
This work is supported by the National Key R&D Program of China (2018YFB1307101) and the National Natural Science Foundation of China (U1813215).
References (40)
- et al.
Semantic reasoning in service robots using expert systems
Robot. Auton. Syst.
(2019) - et al.
Robot task planning using semantic maps
Robot. Auton. Syst.
(2008) - et al.
Realization of task intelligence for service robots in an unstructured environment
Annu. Rev. Control
(2017) - et al.
Integrating robot task planner with common-sense knowledge base to improve the efficiency of planning
Procedia Comput. Sci.
(2013) - et al.
Using temporal logics to express search control knowledge for planning
Artificial Intelligence
(2000) - et al.
Representations for robot knowledge in the KnowRob framework
Artificial Intelligence
(2017) - et al.
Knowledge-based instruction of manipulation tasks for industrial robotics
Robot. Comput.-Integr. Manuf.
(2015) Model-free incremental learning of the semantics of manipulation actions
Robot. Auton. Syst.
(2015)- et al.
Robot learning of manipulation activities with overall planning through precedence graph
Robot. Auton. Syst.
(2019) - et al.
Planning with actively eliciting preferences
Knowl.-Based Syst.
(2019)
HTN planning: Overview, comparison, and beyond
Artificial Intelligence
Towards automated task planning for service robots using semantic knowledge representation
Layered task planning for mobile manipulators
Comput. Electr. Eng.
Neural task planning with AND-OR graph representations
IEEE Trans. Multimed.
UMCP: a sound and complete procedure for hierarchical task-network planning
Robotic roommates making pancakes
A geometric approach to robotic laundry folding
Int. J. Robot. Res.
Cited by (35)
Adaptive and intelligent robot task planning for home service: A review
2023, Engineering Applications of Artificial IntelligenceCitation Excerpt :The overall knowledge-based framework of object search for robot service planning that includes the knowledge layer, the planning layer, and the service layer is delineated in Fig. 13 Task re-planning strategies that consider changes in target position were designed to improve the adaptability of robots in the real world (Wang and Tian, 2022; Wang et al., 2020). The overall framework of the hybrid task planning approach (Wang and Tian, 2022) is shown in Fig. 14.
Hybrid offline and online task planning for service robot using object-level semantic map and probabilistic inference
2022, Information SciencesCitation Excerpt :Because generation of action sequences is the focus of this paper, and object detection, object grasping method and navigation are not the focus of our research, so we select some classic algorithms to ensure that our experiments are carried out. In previous work [41], we counted the relations between dynamic and static objects. We add the category of dynamic objects on the original basis to carry out the experiment in this paper, as shown in Fig. 7.
Efficient semantic place categorization by a robot through active line-of-sight selection
2022, Knowledge-Based SystemsContext change and triggers for human intention recognition
2022, Procedia Computer ScienceService planning oriented efficient object search: A knowledge-based framework for home service robot
2022, Expert Systems with ApplicationsCitation Excerpt :The advances in key technologies of robots, such as object detection (Cao, Liao, Song, Chen, & Li, 2020), robot navigation and localization (Kamil et al., 2017; Zhang, Wang, & Chen, 2019b), robot scene classification (Romero-González, Martínez-Gómez, García-Varea, & Rodriguez-Ruiz, 2017), and so on, have brought the promise for the service robot to execute daily tasks in the household environment. In the unstructured family environment, the robot is expected to have the ability of service planning (Galindo, Fernández-Madrigal, González, & Saffiotti, 2008; Kim et al., 2017; Wang, Tian, & Shao, 2020), which is still a challenging problem at present. The service planning is the first step of service execution.
Service skill improvement for home robots: Autonomous generation of action sequence based on reinforcement learning
2021, Knowledge-Based SystemsCitation Excerpt :Han [15] proposes a method for obtaining textual knowledge by integrating neural networks and knowledge graph. Wang [16] proposes a method of task planning with semantic knowledge and probabilistic inference as the assistance. Oswaldo [17] presents a NN-based model that extracts actions from narrative texts.
- 1
Since 1880.