Out-of-the-box parameter control for evolutionary and swarm-based algorithms with distributed reinforcement learning

Abstract

Parameter control methods for metaheuristics with reinforcement learning put forward so far usually present the following shortcomings: (1) Their training processes are usually highly time-consuming and they are not able to benefit from parallel or distributed platforms; (2) they are usually sensitive to their hyperparameters, which means that the quality of the final results is heavily dependent on their values; (3) and limited benchmarks have been used to assess their generality. This paper addresses these issues by proposing a methodology for training out-of-the-box parameter control policies for mono-objective non-niching evolutionary and swarm-based algorithms using distributed reinforcement learning with population-based training. The proposed methodology is suitable to be used in any mono-objective optimization problem and for any mono-objective and non-niching Evolutionary and swarm-based algorithm. The results in this paper achieved through extensive experiments show that the proposed method satisfactorily improves all the aforementioned issues, overcoming constant, random and human-designed policies in several different scenarios.

Appendix

1.1 TD3’s hyperparameters

• Update delay between policy and Q-Function parameters: 2 (i.e., for each policy update, the Q-Function is updated twice);

• Target noise (i.e., variance of the gaussian noise $\epsilon$): 0.2;

• Target noise clip (i.e., c): 0.5;

• Standard deviation of the zero-mean gaussian noise added to the actions: 0.1;

• $\gamma$: 0.99;

• Initial random steps (i.e., number of steps with random decisions executed before the algorithm starts learning): 45,000;

• Adam $\beta _1$: 0.9;

• Adam $\beta _2$: 0.999;

• Adam $\epsilon$: $10^{-7}$;

1.2 PBT’s hyperparameters

• Perturbation interval: 4;

• Quantile fraction: 0.125;

• Resample probability: 0.5.

1.3 I/F-race’s hyperparameters

• Number of parameter configurations evaluated for each new F-Race process: 48;

• Minimum F-Race iterations before it starts removing bad setups: 20;

• Parameter setup generator standard deviation: 0.3 * range of values of the parameter;

• Minimum number of configurations in the F-Race pool: 10;

• Maximum number of F-Race iterations: 30.

1.4 Human-designed parameter control policies

• HCLPSO (Lynn & Suganthan, 2015b):

  • w: Linear decrease from 0.99 to 0.2;

  • c: Linear decrease from 3 to 1.5;

  • candidate solution step: Linear decrease from 0.1 to 0.000001;

  • c1: Linear decrease from 2.5 to 0.5;

  • c2: Linear increase from 0.5 to 2.5;

  • m: 5.

• FSS (Filho et al., 2009):

  • candidate solution step: Linear decrease from 0.1 to 0.000001;

  • Volitive step: twice the candidate solution step;

  • Maximum weight: 5000.

• DE (Das et al., 2016):

  • F: 2;

  • Crossover probability: 0.5;

• ACO (Das et al., 2016):

  • $\alpha$: 1;

  • $\beta$: 2;

  • $\rho$: 0.98;

  • Probability of using the best ant ever to update the pheromone trail instead of the best in the iteration: linear increase from 0 to 1.

• GA (Michalewicz & Arabas, 1994; Hristakeva, 2004):

  • Mutation probability: 0.1;

  • Crossover probability: 0.75;

  • Elitism size: 2.

1.5 Sampling interval for the random parameter control policy

• HCLPSO:

  • w: [0.2, 0.99];

  • c: [1.5, 3];

  • c1: [0.5, 2.5];

  • c2: [0.5, 2.5];

  • m: 5.

• FSS:

  • candidate solution step: [0, 0.1];

  • Volitive step [$-$0.2, 0.2] (the RL algorithm decides whether to contract or not);

• DE:

  • F: [0.01, 4];

  • Crossover probability: [0.01, 1].

• ACO:

  • $\alpha$: [0, 4];

  • $\beta$: [0, 4];

  • $\rho$: [0, 1];

  • Probability of using the best ant ever to update the pheromone trail instead of the best in the iteration: [0, 1].

• GA:

  • Mutation probability: [0.001, 1];

  • Crossover probability: [0.001, 1];

  • Elitism size: [1, 5].

1.6 Fully detailed experimental results

See Tables 6, 7, 8, 9, 10, 11, 12, 13, 14.

Table 6 Mean and standard deviation (between parenthesis) of the best fitnesses found by HCLPSO with the best policy found with 4, 8, and 16 PBT workers

Table 7 Mean and standard deviation (between parenthesis) of the best fitnesses found by HCLPSO with the best policy found with 2, 4 and 8 iterations of perturbation interval

Table 8 Mean and standard deviation (between parenthesis) of the best fitnesses found by HCLPSO with the best policy trained with different quantile fractions: 0.125, 0.25, and 0.375

Table 9 Mean and standard deviation (between parenthesis) of the best fitnesses found by HCLPSO with the best policy trained with different resample probabilities: 0.25, 0.5, and 0.75

Table 10 Mean of the best fitnesses found by HCLPSO with its parameters controlled by the selected policies, the best policies, a human-designed policy, a random policy, and the same algorithm with static parameters defined by I/F-Race

Table 11 Mean of the best fitnesses found by DE with its parameters controlled by the selected policies, the best policies, a human-designed policy, a random policy, and the same algorithm with static parameters defined by I/F-Race

Table 12 Mean of the best fitnesses found by FSS with its parameters controlled by the selected policies, the best policies, a human-designed policy, a random policy, and the same algorithm with static parameters defined by I/F-Race

Table 13 Mean of the best fitnesses found by binary GA with its parameters controlled by the selected policies, the best policies, a human-designed policy, a random policy, and the same algorithm with static parameters defined by I/F-Race

Table 14 Mean of the best fitnesses found by ACO with its parameters controlled by the selected policies, the best policies, a human-designed policy, a random policy, and the same algorithm with static parameters defined by I/F-Race