Abstract
Reinforcement Learning (RL) methods have been proposed as an alternative approach to feedback control problems. These algorithms require little input from the system designer and can adapt their behavior to the dynamics of the system. Nevertheless, one of the issues when tackling with a feedback control task with continuous state and action spaces from scratch is the enormous amount of interaction with the system required for the agent to learn an acceptable policy. In this paper, we measure empirically the performance gain achieved from performing a conditioning training phase with the agents using randomly set PID controllers in two feedback control problems: the speed control of an underwater vehicle, and the pitch control of an airplane.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Alam, M.S., Tokhi, M.O.: Hybrid fuzzy logic control with genetic optimisation for a single-link flexible manipulator. Engineering Applications of Artificial Intelligence 21(6), 858–873 (2008)
Bellomo, D., Naso, D., Babuska, R.: Adaptive fuzzy control of a non-linear servo-drive: Theory and experimental results. Engineering Applications of Artificial Intelligence 21(6), 846–857 (2008)
Busoniu, L., Babuska, R., De Schutter, B., Ernst, D.: Reinforcement Learning and Dynamic Programming using Function Approximation. CRC Press (2010)
Douligeris, C., Singh, B.K.: Analysis of neural-network-based congestion control algorithms for atm networks. Engineering Applications of Artificial Intelligence 12(4), 453–470 (1999)
Du, H., Lam, J., Sze, K.Y.: Non-fragile output feedback h vehicle suspension control using genetic algorithm. Engineering Applications of Artificial Intelligence 16, 667–680 (2003)
Fernandez-Gauna, B., Ansoategui, I., Etxeberria-Agiriano, I., Graña, M.: An empirical study of actor-critic methods for feedback controllers of ball-screw drivers. In: Ferrández Vicente, J.M., Álvarez Sánchez, J.R., de la Paz López, F., Toledo Moreo, F. J. (eds.) IWINAC 2013, Part II. LNCS, vol. 7931, pp. 441–450. Springer, Heidelberg (2013)
Fernandez-Gauna, B., Ansoategui, I., Etxeberria-Agiriano, I., Graña, M.: Reinforcement learning of ball screw feed drive controllers. In: Engineering Applications of Artificial Intelligence (2014) (first online)
Hafner, R., Riedmiller, M.: Reinforcement learning in feedback control: Challenges and benchmarks from technical process control. Machine Learning 84(1-2), 137–169 (2011)
Hasselt, H.: Reinforcement Learning in Continuous State and Action Spaces. In: Reinforcement Learning: State of the art. Adaptation, Learning, and Optimization, pp. 207–251. Springer (2012)
Hui, Y., Yan, Z., Long, W.Z., Chao, Y.: Adaptive double-loop pid control method of dc motor based on the ga-fnc algorithm. In: 2012 8th IEEE International Symposium on Instrumentation and Control Technology (ISICT), pp. 324–329 (July 2012)
Koren, Y., Lo, C.C.: Advanced controllers for feed drives. In: Annals of the CIRP, vol. 41 (1992)
Kundu, S., Kawata, S.: Genetic algorithms for optimal feedback control design. Engineering Applications of Artificial Intelligence 9(4), 403–411 (1996)
Lewis, F.L., Liu, D., et al.: Reinforcement Learning and Approximate Dynamic Programming for Feedback Control. Wiley (2013)
Lewis, F.L., Vrabie, D., Vamvoudakis, K.G.: Reinforcement learning and feedback control. IEEE Control Systems Magazine 9, 32–50 (2012)
Madar, J., Abonyi, J., Szeifert, F.: Feedback linearizing control using hybrid neural networks identified by sensitivity approach. Engineering Applications of Artificial Intelligence 18(3), 343–351 (2005)
Moore, M.L., Musacchio, J.T., Passino, K.M.: Genetic adaptive control for an inverted wedge: experiments and comparative analyses. Engineering Applications of Artificial Intelligence 14(1), 1–14 (2001)
Neumann, G.: The Reinforcement Learning Toolbox, Reinforcement Learning for Optimal Control Tasks. PhD thesis, Technischen Universitaet, Graz (2005)
Bucak, I.O., Zohdy, M.A.: Reinforcement learning control of nonlinear multi-link system. Engineering Applications of Artificial Intelligence 14, 563–575 (2001)
Senthilkumar, K.S., Bharadwaj, K.K.: Hybrid genetic-fuzzy approach to autonomous mobile robot. In: IEEE International Conference on Technologies for Practical Robot Applications, TePRA 2009, pp. 29–34, 9–10 (2009)
Srinivasan, K., Tsao, T.C.: Machine feed drives and their control - a survey of the state of the art. Journal of Manufacturing Science and Engineering 119, 743–748 (1997)
Sutton, R., Precup, D., Singh, S.: Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence 112, 181–211 (1999)
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)
van Hasselt, H., Wiering, M.A.: Reinforcement learning in continuous action spaces. In: Proceedings of the 2007 IEEE Symposium on Approximate Dynamic Programming and Reinforcement Learning (2007)
Ziegler, J.G., Nichols, N.B.: Optimum settings for automatic controllers. Journal of Dynamic Systems, Measurements and Control 115, 220–222 (1993)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Fernandez-Gauna, B., Osa, J.L., Graña, M. (2014). Effect of Initial Conditioning of Reinforcement Learning Agents on Feedback Control Tasks over Continuous State and Action Spaces. In: de la Puerta, J., et al. International Joint Conference SOCO’14-CISIS’14-ICEUTE’14. Advances in Intelligent Systems and Computing, vol 299. Springer, Cham. https://doi.org/10.1007/978-3-319-07995-0_13
Download citation
DOI: https://doi.org/10.1007/978-3-319-07995-0_13
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07994-3
Online ISBN: 978-3-319-07995-0
eBook Packages: EngineeringEngineering (R0)