Abstract
Qualitative modeling can be applied to the control of dynamic systems by following these steps: (1) learning a qualitative model of the controlled dynamic system from the system’s behaviors in time, (2) using the learned model to derive a qualitative plan for the control task, and (3) executing the qualitative plan on an actual dynamic system. This approach has been demonstrated in the usual cart-pole control domain as significantly more sample efficient than the usual variants of reinforcement learning, by at least two orders of magnitude. The qualitative approach also enables better explanation of the learned control strategy through symbolic planning. In this paper, we generalize the cart-pole problem to uneven terrains, such as driving over a crater or a hill. We study whether the learned flat-surface qualitative controller can be successfully transferred to the tasks of negotiating uneven terrain. Experiments show that the flat-surface qualitative controller is remarkably robust on new, more difficult tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Anderson, C.W.: Learning to control an inverted pendulum using neural networks. IEEE Control Syst. Mag. 9(3), 31–37 (1989)
Barto, A.G., Sutton, R.S., Anderson, C.W.: Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. SMC 13(5), 834–846 (1983)
Bratko, I., Šuc, D.: Learning qualitative models. AI Mag. 24(4), 107–119 (2003)
Hein, D., Limmer, S., Runkler, T.A.: Interpretable control by reinforcement learning. IFAC-PapersOnLine 53(2), 8082–8089 (2020). https://doi.org/10.1016/j.ifacol.2020.12.2277
Hosokawa, S., Kato, J., Nakano, K.: A reward allocation method for reinforcement learning in stabilizing control tasks. Artif. Life Rob. 19(2), 109–114 (2014). https://doi.org/10.1007/s10015-014-0146-0
Kuipers, B.: Qualitative simulation. Artif. Intell. 29(3), 289–338 (1986)
Kuipers, B.: Qualitative Reasoning: Modeling and Simulation with Incomplete Knowledge. MIT Press, Cambridge, MA, USA (1994)
Linglin, W., Yongxin, L., Xiaoke, Z.: Design of reinforce learning control algorithm and verified in inverted pendulum. In: 2015 34th Chinese Control Conference (CCC), pp. 3164–3168 (2015)
Michie, D., Chambers, R.A.: BOXES: an experiment in adaptive control. Mach. Intell. 2, 125–133 (1968)
Puriel-Gil, G., Yu, W., Sossa, H.: Reinforcement learning compensation based PD control for inverted pendulum. In: 15th International Conference on Electrical Engineering, Computing Science and Automatic Control (CCE), pp. 1–6 (2018)
Ramamoorthy, S., Kuipers, B.: Qualitative heterogeneous control of higher order systems. In: Maler, O., Pnueli, A. (eds.) HSCC 2003. LNCS, vol. 2623, pp. 417–434. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-36580-X_31
Riedmiller, M., Peters, J., Schaal, S.: Evaluation of policy gradient methods and variants on the cart-pole benchmark. In: 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pp. 254–261 (2007)
Šoberl, D.: Automated planning with induced qualitative models in dynamic robotic domains. Ph.D. thesis, University of Ljubljana (2021). https://repozitorij.uni-lj.si/IzpisGradiva.php?id=126285
Šoberl, D., Bratko, I.: Reactive motion planning with qualitative constraints. In: Benferhat, S., Tabia, K., Ali, M. (eds.) IEA/AIE 2017. LNCS (LNAI), vol. 10350, pp. 41–50. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-60042-0_5
Šoberl, D., Bratko, I.: Learning explainable control strategies demonstrated on the pole-and-cart system. In: Wotawa, F., Friedrich, G., Pill, I., Koitz-Hristov, R., Ali, M. (eds.) IEA/AIE 2019. LNCS (LNAI), vol. 11606, pp. 483–494. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22999-3_42
Šoberl, D., Bratko, I.: Learning to control a quadcopter qualitatively. J. Intell. Rob. Syst. (2020). https://doi.org/10.1007/s10846-020-01228-7
Šoberl, D., Žabkar, J., Bratko, I.: Qualitative planning of object pushing by a robot. In: Esposito, F., Pivert, O., Hacid, M.-S., Raś, Z.W., Ferilli, S. (eds.) ISMIS 2015. LNCS (LNAI), vol. 9384, pp. 410–419. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-25252-0_44
Šuc, D.: Machine Reconstruction of Human Control Strategies. Frontiers in Artificial Intelligence and Applications, IOS Press, Inc. (2003)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Šoberl, D., Bratko, I. (2023). Transferring a Learned Qualitative Cart-Pole Control Model to Uneven Terrains. In: Bifet, A., Lorena, A.C., Ribeiro, R.P., Gama, J., Abreu, P.H. (eds) Discovery Science. DS 2023. Lecture Notes in Computer Science(), vol 14276. Springer, Cham. https://doi.org/10.1007/978-3-031-45275-8_30
Download citation
DOI: https://doi.org/10.1007/978-3-031-45275-8_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-45274-1
Online ISBN: 978-3-031-45275-8
eBook Packages: Computer ScienceComputer Science (R0)