A novel genetic reinforcement learning for nonlinear fuzzy control problems

Abstract

Unlike a supervise learning, a reinforcement learning problem has only very simple “evaluative” or “critic” information available for learning, rather than “instructive” information. A novel genetic reinforcement learning, called reinforcement sequential-search-based genetic algorithm (R-SSGA), is proposed for solving the nonlinear fuzzy control problems in this paper. Unlike the traditional reinforcement genetic algorithm, the proposed R-SSGA method adopts the sequential-search-based genetic algorithms (SSGA) to tune the fuzzy controller. Therefore, the better chromosomes will be initially generated while the better mutation points will be determined for performing efficient mutation. The adjustable parameters of fuzzy controller are coded as real number components. We formulate a number of time steps before failure occurs as a fitness function. Simulation results have shown that the proposed R-SSGA method converges quickly and minimizes the population size.

Introduction

In recent years, the concept of the fuzzy logic or artificial neural networks for control problems has grown into a popular research area [17], [24], [27]. The reason is that classical control theory usually requires a mathematical model for designing controllers. The inaccuracy of mathematical modeling of plants usually degrades the performance of the controllers, especially for nonlinear and complex control problems [2], [11]. Fuzzy logic has the ability to express the ambiguity of human thinking and translate expert knowledge into computable numerical data.

A fuzzy system consists of a set of fuzzy IF–THEN rules that describe the input–output mapping relationship of the networks. Obviously, it is difficult for human experts to examine all the input–output data from a complex system to find proper rules for a fuzzy system. To cope with this difficulty, several approaches that are used to generate the fuzzy IF–THEN rules from numerical data have been proposed [1], [3], [15], [21]. These methods were developed for supervised learning; i.e., the correct “target” output values are given for each input pattern to guide the learning of the network. However, most of the supervised learning algorithms for neural fuzzy networks require precise training data to tune the networks for various applications. For some real world applications, precise training data are usually difficult and expensive, if not impossible, to obtain. For this reason, there has been a growing interest in reinforcement learning algorithms for use in fuzzy [22], [30] or neural controller [10], [29] design.

In the design of a fuzzy controller, adjusting the required parameters is important. To do this, back-propagation (BP) training was widely used in [10], [23], [29]. It is a powerful training technique that can be applied to networks with a forward structure. Since the steepest descent technique is used in BP training to minimize the error function, the algorithms may reach the local minima very fast and never find the global solution.

The development of genetic algorithms (GAs) has provided another approach for adjusting parameters in the design of controllers. GA is a parallel and global technique [16], [22]. Because it simultaneously evaluates many points in a search space, it is more likely to converge toward the global solution. Some researchers have developed methods to design and implement fuzzy controllers by using GAs. Karr [17] used a GA to generate membership functions for a fuzzy system. In Karr's work, a user needs to declare an exhaustive rule set and then use a GA to design only the membership functions. In [12], a fuzzy controller design method that used a GA to find the membership functions and the rule sets simultaneously was proposed. Lin [19] proposed a hybrid learning method which combines the GA and the least-squares estimate (LSE) method to construct a neural fuzzy controller. In [12], [19], the input space was partitioned into a grid. The number of fuzzy rules (i.e., the length of each chromosome in the GA) increased exponentially as the dimension of the input space increased. To overcome this problem, Juang [14] adopted a flexible partition approach in the precondition part. The method has the admirable property of small network size and high learning accuracy.

Recently, some researchers [9], [16], [18], [22] applied GA methods to implement reinforcement learning in the design of fuzzy controllers. Lin and Jou [22] proposed GA-based fuzzy reinforcement learning to control magnetic bearing systems. In [16], Juang and his colleagues proposed genetic reinforcement learning in designing fuzzy controllers. The GA adopted in [16] was based upon traditional symbiotic evolution which, when applied to fuzzy controller design, complements the local mapping property of a fuzzy rule. In [9], Er and Deng proposed dynamic Q-Learning for on-line tuning the fuzzy inference systems. Kaya and Alhajj [18] proposed a novel multiagent reinforcement learning approach based on fuzzy OLAP association rules mining. However, these approaches encountered one or more of the following major problems: (1) the initial values of the populations were generated randomly; (2) the mutational value was generated by the constant range while the mutation point is also generated randomly; (3) the population sizes always depend on the problem which is to be solved.

In this paper, we propose a reinforcement sequential-search-based genetic algorithm (R-SSGA) method to solve above-mentioned problems. Unlike the traditional reinforcement learning, in this paper, we formulate a number of time steps before failure occurs as the fitness function. The new sequential-search-based genetic algorithm (SSGA) is also proposed to perform parameter learning. Moreover, the SSGA method is different from traditional GA, which the better chromosomes will be initially generated while the better mutation points will be determined for performing efficient mutation. Compared with traditional GA, the SSGA method generates initialize population efficiently and decides efficient mutation points to perform mutation. The advantages of the proposed R-SSGA method are summarized as follows: (1) The R-SSGA method can reduce the population sizes to a minimum size (4); (2) The chromosome which has the best performance will be chosen to perform the mutation operator in each generation; (3) The R-SSGA method converges more quickly than existing traditional genetic methods.

This paper is organized as follows. Section 2 introduces the SSGA. A R-SSGA is presented in Section 3. In Section 4, the proposed R-SSGA method is evaluated using two different control problems, and its performances are benchmarked against other structures. Finally, conclusions on the proposed algorithm are summarized in the last section.