Abstract
As a genetics-based machine learning technique, zeroth-level classifier system (ZCS) based on average reward reinforcement learning (ZCSAR) evolves solutions to optimize average reward per time step. However, initial experimental results have shown that, in some cases, the performance of ZCSAR oscillates heavily during the learning period, or cannot reach the optimum during the testing period. In this paper, we modify the selection strategies in ZCSAR to improve its performance, under conditions of minimal changes of ZCSAR. The proposed selection strategies take tournament selection method to choose parents in Genetic Algorithm (GA), and take roulette wheel selection method to choose actions in match set and to choose classifiers for deletion in both GA and covering. Experimental results show that ZCSAR with the new selection strategies can evolve more promising solutions with enough parameter independence, and also with slighter oscillation during the learning period.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Bull L (2015) A brief history of learning classifier systems: From CS-1 to XCS and its variants. Evol Intell 8(2):55–70. https://doi.org/10.1007/s12065-015-0125-y
Bull L, Hurst J (2002) ZCS redux. Evol Comput 10(2):185–205
Bull L (2002) Lookahead and latent learning in ZCS. In: Langdon WB, Paz ECU, Mathias K et al (eds) GECCO 2002: proceedings of the genetic and evolutionary computation conference. Morgan Kaufmann, New York, pp 897–904
Butz MV, Sastry K, Goldberg DE (2005) Strong, stable, and reliable fitness pressure in XCS due to tournament selection. Genet Program Evolvable Mach 6(1):53–77. https://doi.org/10.1007/s10710-005-7619-9
Cádrik T, Mach M (2014) Control of agents in a multi-agent system using ZCS evolutionary classifier systems. In: 2014 IEEE 12th International Symposium on Applied Machine Intelligence and Informatics (SAMI). IEEE, Herl’any, Slovakia, pp 163–166
Cádrik T, Mach M (2015) Usage of ZCS evolutionary classifier system as a rule maker for cleaning robot task. In: Sinčák P, Hartono P, Virčíková M, Vaščák J, Jakša R (eds) Emergent trends in robotics and intelligent systems, vol 316. Springer International Publishing, pp 113–119
Debie E, Shafi K, Merrick K, Lokan C (2017) On taxonomy and evaluation of feature selection-based learning classifier system ensemble approaches for data mining problems. Comput Intell 33(3):554–578
Kovacs T (1999) Deletion schemes for classifier systems. In: Proceedings of the 1st annual conference on genetic and evolutionary computation, vol 1. GECCO’99. Morgan Kaufmann Publishers Inc., Orlando, pp 329–336
Lanzi PL (2014) Learning classifier systems: a gentle introduction. In: Proceedings of the 2014 conference companion on genetic and evolutionary computation companion. GECCO Comp ‘14. ACM, Vancouver, BC, Canada, pp 407–430
Roozegar M, Mahjoob MJ, Esfandyari MJ, Panahi MS (2016) XCS-based reinforcement learning algorithm for motion planning of a spherical mobile robot. Appl Intell 45(3):736–746. https://doi.org/10.1007/s10489-016-0788-9
Schwartz A (1993) A reinforcement learning method for maximizing undiscounted rewards. In: Utgoff P (ed) Proceedings of the tenth international conference on machine learning, vol 298. Morgan Kaufmann, San Francisco (CA), pp 298–305
Siddique A, Iqbal M, Browne WN (2016) A comprehensive strategy for mammogram image classification using learning classifier systems. In: 2016 IEEE congress on evolutionary computation (CEC). IEEE, Vancouver, BC, pp 2201–2208
Singh SP (1994) Reinforcement learning algorithms for average-payoff Markovian decision processes. In: Proceedings of the twelfth national conference on artificial intelligence, vol 1. AAAI ‘94. American Association for artificial Intelligence, Menlo Park, CA, USA, pp 700–705
Stone C, Bull L (2009) Configuring ZCS for continuous-valued single-step Boolean problems. Analysis 2(3):19
Tzima FA, Mitkas PA (2008) ZCS revisited: zeroth-level classifier systems for data mining. In: Proceedings of the 2008 IEEE international conference on data mining workshops. ICDMW ‘08. IEEE Computer Society, Washington, DC, USA, pp 700–709
Tzima FA, Mitkas PA (2013) Strength-based learning classifier systems revisited: effective rule evolution in supervised classification tasks. Eng Appl Artif Intell 26(2):818–832. https://doi.org/10.1016/j.engappai.2012.09.022
Urbanowicz RJ, Browne WN (2017) Introduction to learning classifier systems. Springer, Berlin
Wen X, Shao L, Xue Y, Fang W (2015) A rapid learning algorithm for vehicle classification. Inf Sci 295(1):395–406
Wilson SW (1994) ZCS: a zeroth level classifier system. Evol Comput 2(1):1–18
Wilson SW (1995) Classifier fitness based on accuracy. Evol Comput 3(2):149–175
Zang Z, Li D, Wang J, Xia D (2013) Learning classifier system with average reward reinforcement learning. Knowl Based Syst 40(0):58–71. https://doi.org/10.1016/j.knosys.2012.11.011
Zang Z, Li Z, Wang J, Dan Z (2016) Tournament selection in zeroth-level classifier systems based on average reward reinforcement learning. Revista Tecnica de la Facultad de Ingenieria Universidad del Zulia 39(4):95–104. https://doi.org/10.21311/001.39.4.13
Zatuchna Z, Bagnall A (2009) A learning classifier system for mazes with aliasing clones. Nat Comput 8(1):57–99
Acknowledgements
This work was partially supported by National Natural Science Foundation of China (Grant no. 61502274), Natural Science Foundation of Hubei (Grant nos. 2015CFB336, 2014CFC1144, 2015CFA025), Natural Science Foundation of Hubei Provincial Department of Education (Grant no. Q20151208), the Doctoral Startup Foundation of China Three Gorges University (Grant nos. KJ2013B064, KJ2013B063), Open Fund of Hubei Key Laboratory of Intelligent Vision Based Monitoring for Hydroelectric Engineering (CTGU) (Grant nos. 2015KLA08, 2015KLA03), and the Scientific and Technological Project of Yichang City of Hubei Province (Grant no. A15-302-a13).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zang, Z., Li, Z., Dan, Z. et al. Improving selection strategies in zeroth-level classifier systems based on average reward reinforcement learning. J Ambient Intell Human Comput 15, 1201–1211 (2024). https://doi.org/10.1007/s12652-018-0682-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12652-018-0682-x