Comparison of strategy learning methods in Farmer–Pest problem for various complexity environments without delays
Highlights
► We propose the multi-dimensional domain allowing comparison of learning algorithms. ► We compare efficaciousness of several agent strategy learning algorithms in the proposed domain. ► We show that methods other than reinforcement learning can be used for agent strategy generation. ► We show that in specific conditions, supervised learning can improve performance of agents much faster that reinforcement learning.
Introduction
The goal of this paper is to compare efficaciousness of several agent strategy learning algorithms in a new learning environment designed especially for such a comparison.
The problem of learning naturally appears in multi-agent systems which are efficient architectures for decentralized problem solving. In complex or changing environments it is very difficult, sometimes even impossible, to design all system details a priori. To overcome this problem one can apply a learning algorithm which allows to adapt the system to the environment. To apply learning in a multi-agent system, one should choose a method of learning, which fits well to the problem. There are many algorithms developed so far. However, in multi-agent systems most applications use reinforcement learning or evolutionary computations.
To help the system developer to choose appropriate learning algorithm, these algorithms should be compared in a controlled environment to choose the best method. This paper presents a new domain called Farmer–Pest problem, which is designed to compare various agent learning methods. It is an optimization with feedback problem. It has multiple dimensions and therefore can be easily configured to better adapt to specific needs. It may be simple or very complex, with reward delay or without it. Changes are introduced by tuning environment parameters. It makes the environment more flexible than other environments used by researchers to test agent learning (see Section 2). This flexibility is necessary to compare learning methods in various conditions and settings, without violating the core ideas and rules of the problem.
This paper is an extended version of [1]. We present more experiments and they are described in a more detailed form. The aim of this paper is to investigate selected learning algorithms by comparing their performance in environments of configurable complexity in terms of the number of possible factors and the distribution of their values. We present comparison of reinforcement learning algorithm (SARSA) and three supervised learning algorithms (Naïve Bayes, C4.5 and Ripper).
In our research, we make the following contributions to the state of the art: we propose the multi-dimensional domain allowing comparison leaning algorithms; we show that methods other than reinforcement learning can be used for strategy generation; we compare learning algorithms in configurations with various complexity and show that in specific conditions, supervised learning can improve performance of agents much faster that reinforcement learning.
In the following sections we overview existing environments for learning agents and describe the proposed problem domain. Next, the learning agent architecture is described followed by the presentation of methods applied and experimental results. Finally, conclusions and the further work are outlined.
Section snippets
Environments for learning agents
Good survey of learning in multi-agent systems working in various domains can be found in [2], [3]. Below three example problems are presented.
Very popular in multi-agent systems is a soccer domain. The environment consist of a soccer field with two goals and a ball. Two teams of simulated or real robots are controlled by the agents. The performance is measured by the difference of scored goals. In [4] genetic programming is utilized to learn behavior-based team coordination. In [5] C4.5
The Farmer–Pest problem
The Farmer–Pest problem borrows the concept from the specific aspect of real world, in which farmers struggle to protect their fields and crops from pests. Each farmer (this is the only type of agent in the problem) can manage multiple fields. On each field, a multiple types of pests can appear. Each pest has a specific set of attributes, e.g. number of legs, color. Values of these attributes depend on the pest type. To protect the field, the farmer can take the advantage of multiple means
Architecture of learning agent
In this section we present the learning agent architecture, which is used in the experiments. It is presented in Fig. 1(b). The agent consists of four modules:
- Processing Module
is responsible for basic agent activities, storing training data, executing learning process, and using learned knowledge.
- Learning Module
is responsible for execution of learning algorithm and giving answers for problems with use of learned knowledge.
- Training Data
is a storage for examples used for learning.
- Generated
Methods
In this section we present details of the methods applied in the system. We describe two types of learning algorithms that were tested in experiments: reinforcement learning and supervised learning. Next we present Boltzmann selection that is applied to chose actions for execution. Finally, selected implementation details are shown.
Experiments
With use of the Farmer–Pest problem we are able to make comparison of several learning algorithms. Here we present broader results than initial ones presented in [1]. We have chosen three dimensions to define various versions of the environment: number of attributes and their domains, number of pest types and attribute distributions. We are able to show that various conditions favor different learning algorithms.
Conclusion and further research
In this paper we present comparison of performance of the reinforcement and supervised learning algorithms: SARSA, Naïve Bayes, C4.5 and Ripper. These algorithms were used by agents taking part in a simulation of Farmer–Pest Problem which is A scalable multi-dimensional problem domain for testing agent learning algorithms. This environment provides a large number of configurable dimensions which enables preparation of different testing conditions. This allows to test algorithms more thoroughly
Acknowledgments
This research was funded in part by the Polish Ministry of Science and Higher Education grant number N N516 366236. Authors would like to thank students M. Mlostek and M. Pulchny, who prepared software for the experiments.
This research was also partially supported by The European Union by means of European Social Fund, PO KL Priority IV: Higher Education and Research, Activity 4.1: Improvement and Development of Didactic Potential of the University and Increasing Number of Students of the
Bartlomiej Sniezynski received his Ph.D. (2004) degree in Computer Science from AGH University of Science and Technology, Poland. In 2004 he worked as a Postdoctoral Fellow at the Machine Learning and Inference Laboratory, George Mason University, Fairfax, VA, USA, where he worked in professor R.S. Michalski's team. Currently he is an assistant professor at the Department of Computer Science, AGH. His research interests include machine learning, multi-agent systems and knowledge engineering.
References (21)
- et al.
International Conference on Computational Science, ICCS 2011 Farmer–Pest Problem: A Multidimensional Problem Domain for Comparison of Agent Learning Methods
Procedia Computer Science
(2011) Fast effective rule induction
- et al.
Parallel multi-frontal solver for p adaptive finite element modeling of multi-physics computational problems
Journal of Computational Science
(2010) - et al.
Cooperative multi-agent learning: The state of the art
Autonomous Agents and Multi-Agent Systems
(2005) - et al.
Learning in multiagent systems
(1999) - et al.
Co-evolving soccer softbot team coordination with genetic programming
- et al.
On behavior classification in adversarial environments
in: Distributed Autonomous Robotic Systems 4
(2000) - et al.
Reinforcement learning for robocup-soccer keepaway
Adaptive Behavior
(2005) Multi-agent reinforcement learning: Independent vs. cooperative agents
- et al.
Evolving behavioral strategies in predators and prey
Adaptation and Learning in Multiagent Systems
(1996)
Cited by (10)
Agent-based simulations, adaptive algorithms and solvers
2017, Journal of Computational ScienceCitation Excerpt :With the perspective of two special issues already published and the new one in press, we can draw some historical perspective on the evolution of the research topics performed within the three mentioned groups. The soft-computing papers were indeed represented in the first special issues [1–4], with some local drop down to only one paper in the second issue [8], and back again to three research papers in this edition [13–15]. The hard-computing papers with the emphasis on adaptive algorithms were represented in the first issue by only one paper [5], and this number increase to two papers in the second edition [9,10], and now we have two papers again [16,17].
Agent-based simulations, adaptive algorithms and solvers
2015, Journal of Computational ScienceCitation Excerpt :The first special issue followed the ABS-AAS workshop in frame of International Conference on Computational Science (ICCS) held in Singapore, in June 2011. The papers in the first special issue was categorized into two sections, including Soft Computing papers [1–3] and Hard Computing papers [4–7]. This time we categorize the papers submitted to the second special issue into three groups, including agent-based simulations, adaptive algorithms and solvers.
Training example generation method for supervised learning agents in sequential scenarios
2014, Procedia Computer ScienceAgent-based adaptation system for service-oriented architectures using supervised learning
2014, Procedia Computer ScienceWillingness of farmers for vocational training on new professional farmers and its influencing factors
2020, Revista Argentina de Clinica PsicologicaAdaptive Service Management in Mobile Cloud Computing by Means of Supervised and Reinforcement Learning
2018, Journal of Network and Systems Management
Bartlomiej Sniezynski received his Ph.D. (2004) degree in Computer Science from AGH University of Science and Technology, Poland. In 2004 he worked as a Postdoctoral Fellow at the Machine Learning and Inference Laboratory, George Mason University, Fairfax, VA, USA, where he worked in professor R.S. Michalski's team. Currently he is an assistant professor at the Department of Computer Science, AGH. His research interests include machine learning, multi-agent systems and knowledge engineering.
Jacek Dajda received his Ph.D. (2008) degree in Computer Science from AGH University of Science and Technology, Poland. At present, he works as Assistant Professor at AGH University of Science and Technology located in Krakow, Poland. His research interests involve software engineering with an emphasis on software architectures and frameworks.