Comparison of strategy learning methods in Farmer–Pest problem for various complexity environments without delays

Abstract

In this paper effectiveness of several agent strategy learning algorithms is compared in a new multi-agent Farmer–Pest learning environment. Learning is often utilized by multi-agent systems which can deal with complex problems by means of their decentralized approach. With a number of learning methods available, a need for their comparison arises. This is why we designed and implemented new multi-dimensional Farmer–Pest problem domain, which is suitable for benchmarking learning algorithms. This paper presents comparison results for reinforcement learning (SARSA) and supervised learning (Naïve Bayes, C4.5 and Ripper). These algorithms are tested on configurations with various complexity with not delayed rewards. The results show that algorithm performances depend highly on the environment configuration and various conditions favor different learning algorithms.

Introduction

The goal of this paper is to compare efficaciousness of several agent strategy learning algorithms in a new learning environment designed especially for such a comparison.

The problem of learning naturally appears in multi-agent systems which are efficient architectures for decentralized problem solving. In complex or changing environments it is very difficult, sometimes even impossible, to design all system details a priori. To overcome this problem one can apply a learning algorithm which allows to adapt the system to the environment. To apply learning in a multi-agent system, one should choose a method of learning, which fits well to the problem. There are many algorithms developed so far. However, in multi-agent systems most applications use reinforcement learning or evolutionary computations.

To help the system developer to choose appropriate learning algorithm, these algorithms should be compared in a controlled environment to choose the best method. This paper presents a new domain called Farmer–Pest problem, which is designed to compare various agent learning methods. It is an optimization with feedback problem. It has multiple dimensions and therefore can be easily configured to better adapt to specific needs. It may be simple or very complex, with reward delay or without it. Changes are introduced by tuning environment parameters. It makes the environment more flexible than other environments used by researchers to test agent learning (see Section 2). This flexibility is necessary to compare learning methods in various conditions and settings, without violating the core ideas and rules of the problem.

This paper is an extended version of [1]. We present more experiments and they are described in a more detailed form. The aim of this paper is to investigate selected learning algorithms by comparing their performance in environments of configurable complexity in terms of the number of possible factors and the distribution of their values. We present comparison of reinforcement learning algorithm (SARSA) and three supervised learning algorithms (Naïve Bayes, C4.5 and Ripper).

In our research, we make the following contributions to the state of the art: we propose the multi-dimensional domain allowing comparison leaning algorithms; we show that methods other than reinforcement learning can be used for strategy generation; we compare learning algorithms in configurations with various complexity and show that in specific conditions, supervised learning can improve performance of agents much faster that reinforcement learning.

In the following sections we overview existing environments for learning agents and describe the proposed problem domain. Next, the learning agent architecture is described followed by the presentation of methods applied and experimental results. Finally, conclusions and the further work are outlined.