Abstract
In this paper, we propose a new algorithm of an adaptive actor-critic method with multi-step simulated experiences, as a kind of temporal difference (TD) method. In our approach, the TD-error is composed of two value- functions and m utility functions, where m denotes the number of multi-steps in which the experience should be simulated. The value-function is constructed from the critic formulated by a radial basis function neural network (RBFNN), which has a simulated experience as an input, generated from a predictive model based on a kinematic model. Thus, since our approach assumes that the model is available to simulate the m-step experiences and to design a controller, such a kinematic model is also applied to construct the actor and the resultant model based actor (MBA) is also regarded as a network, i.e., it is just viewed as a resolved velocity control network. We implement this approach to control nonholonomic mobile robot, especially in a trajectory tracking control problem for the position coordinates and azimuth. Some simulations show the effectiveness of the proposed method for controlling a mobile robot with two-independent driving wheels.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
van Buijtenen WM, Schram G, Babuška R, Verbruggen HB (1998) Adaptive fuzzy control of satellite attitude by reinforcement learning. IEEE Trans Fuzzy Syst 6(2):185–194
Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3:9–44
Konda VR, Tsitsiklis JN (2003) On actor-critic algorithms. SIAM J Contr Optim 42(4):1143–1166
Watanabe K, Syam R, Izumi K, Kiguchi K (2001) Adaptive actor-critic with current estimated or predicted value-function. In: Proceeding of the international Conference on KES2001. Osaka, Japan, pp 1308–1318
Syam R, Watanabe K, Izumi K, Kiguchi K (2001) Adaptive actor-critic learning of mobile robots using actual and simulated experiences. In: Proceedings of the international Conference on ICCAS2001. Cheju, Korea, pp 312–316
Watanabe K (1992) Adaptive estimation and control. Prentice Hall, London
Syam R, Watanabe K, Izumi K, Kiguchi K (2002) Adaptive actor-critic learning of mobile robot using simulated experience through predictive model. In: Proceedings of the international conference on AROB2002. Beppu, Japan, pp 421–424
Syam R, Watanabe K, Izumi K, Kiguchi K (2002) Control of nonholonomic mobile robot by an adaptive actor-critic method with simulated experience based value-functions. In: Proceedings of 2002 IEEE international conference on Robotics and Automation (ICRA2002). Washington D.C., pp 3960–3965
Prokhorov DV, Wunch DC (1997) Adaptive critic designs. IEEE Trans Neural Network 8(5):997–1007
Barto AG, Sutton RS, Anderson W (1983) Neuron-like adaptive elements can solve difficult learning control problems. IEEE Trans Syst Man, Cybernet 13(5):834–846
Sutton RS, Barto AG (1999) Reinforcement learning, an introduction. MIT Press, Cambridge
Watanabe K, Tang J, Nakamura M, Koga S, Fukuda T (1996) A fuzzy-Gaussian neural network and its application to mobile robot. IEEE Trans Contr Syst Technol 4(2):193–199
Fierro R, Lewis FL (1998) Control of nonholonomic mobile robot using neural networks. IEEE Trans Neural Network 9(4):589–600
Fierro R, Lewis FL (1997) Control of a nonholonomic mobile robot: backstepping kinematics into dynamics. J Robot Syst 14(3):149–163
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Syam, R., Watanabe, K. & Izumi, K. An Adaptive Actor-critic Algorithm with Multi-step Simulated Experiences for Controlling Nonholonomic Mobile Robots. Soft Comput 11, 81–89 (2007). https://doi.org/10.1007/s00500-006-0054-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-006-0054-x