Ryoki Hamano
ABSTRACT
The Population-Based Incremental Learning (PBIL) is a Bernoulli distribution-based evolutionary algorithm for binary black-box optimization. The PBIL updates the distribution parameter according to the samples generated from the current distribution and their rankings. In PBIL, when some distribution parameters are continuously updated randomly, undesirable convergence without sufficient exploration is observed. This behavior is called genetic drift and induces an increasing number of function evaluations and convergence to local optima. In particular, large update strength leads to genetic drift while faster search. The ways to deal with genetic drift are limited, such as decreasing the update strength, and there is a trade-off between search efficiency and stability. This paper proposes a method to reduce genetic drift in PBIL based on entropy regularization widely used in reinforcement learning. We introduce entropy regularization into PBIL as a penalty term or constraint. The experimental results on well-known benchmark problems show that the proposed entropy regularization can efficiently suppress genetic drift, decrease the number of function evaluations, and improve stability.
- Abbas Abdolmaleki, Bob Price, Nuno Lau, Luis Paulo Reis, and Gerhard Neumann. 2017. Deriving and Improving CMA-ES with Information Geometric Trust Regions. In Genetic and Evolutionary Computation Conference (GECCO). 657--664.Google Scholar
- Riad Akrour, Gerhard Neumann, Hany Abdulsamad, and Abbas Abdolmaleki. 2016. Model-Free Trajectory Optimization for Reinforcement Learning. In International Conference on Machine Learning (ICML). 2961--2970.Google Scholar
- Shumeet Baluja. 1994. Population-Based Incremental Learning: A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning. Technical Report Tech Rep CMU-CS-94-163. Carnegie Mellon University.Google Scholar
- Shumeet Baluja and Rich Caruana. 1995. Removing the Genetics from the Standard Genetic Algorithm. In International Conference on Machine Learning (ICML). 38--46.Google ScholarCross Ref
- Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous Methods for Deep Reinforcement Learning. In International Conference on Machine Learning (ICML). 1928--1937.Google Scholar
- Yann Ollivier, Ludovic Arnold, Anne Auger, and Nikolaus Hansen. 2017. Information-Geometric Optimization Algorithms: A Unifying Picture via Invariance Principles. Journal of Machine Learning Research 18, 1 (2017), 564--628.Google ScholarDigital Library
- John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust Region Policy Optimization. In International Conference on Machine Learning (ICML). 1889--1897.Google Scholar
- Dirk Sudholt and Carsten Witt. 2016. Update Strength in EDAs and ACO: How to Avoid Genetic Drift. In Genetic and Evolutionary Computation Conference (GECCO). 61--68.Google ScholarDigital Library
- Philip Thomas, Bruno Castro Silva, Christoph Dann, and Emma Brunskill. 2016. Energetic Natural Gradient Descent. In International Conference on Machine Learning (ICML). 2887--2895.Google Scholar
- Ronald Williams and Jing Peng. 1991. Function Optimization Using Connectionist Reinforcement Learning Algorithms. Connection Science 3, 3 (1991), 241--268.Google ScholarCross Ref
- Ronald J. Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. Mach. Learn. 8, 3--4 (1992), 229--256.Google ScholarDigital Library
Index Terms
- Reduction of genetic drift in population-based incremental learning via entropy regularization
Recommendations
Hybrid population-based incremental learning using real codes
LION'05: Proceedings of the 5th international conference on Learning and Intelligent OptimizationThis paper proposes a hybrid evolutionary algorithm (EA) dealing with population-based incremental learning (PBIL) and some efficient local search strategies. A simple PBIL using real codes is developed. The evolutionary direction and approximate ...
Introducing the cumulation to the population based incremental learning and the compact GA to relax genetic drift
GECCO '17: Proceedings of the Genetic and Evolutionary Computation Conference CompanionThe population based incremental learning (PBIL) and the compact genetic algorithm (cGA) are both Bernoulli distribution based search algorithm for optimization of binary variables. The probability parameter that controls the probability of each bit ...
A diversity maintaining population-based incremental learning algorithm
In this paper we propose a new probability update rule and sampling procedure for population-based incremental learning. These proposed methods are based on the concept of opposition as a means for controlling the amount of diversity within a given ...
Comments