Abstract
F-Race is an offline parameter tuning method which efficiently allocates samples in order to identify the parameter setting with the best expected performance, out of a given set of parameter settings. Using non parametric statistical tests, F-Race discards parameter settings which perform significantly worse than the current best, allowing the surviving parameter settings to be tested on more instances and hence obtaining better estimates for their performance. The statistical tests require setting significance levels which directly affect the algorithm’s ability of detecting the best parameter setting, and the total runtime. In this paper, we show that it is not straightforward to set the significance level and propose a simple modification to automatically adapt the significance level such that the failure rate is minimized. This is tested empirically using data drawn from probability distributions with pre-defined characteristics. Results indicate that, under a strict computational budget, F-Race with online adaptation performs significantly better than its counterpart with even the best fixed value.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Maron, O., Moore, A.: Hoeffding races: accelerating model selection search for classification and function approximation. Adv. Neural Inf. Process. Syst. 6, 59–66 (1994)
Schaffer, J.D., Caruana, R.A., Eshelman, L.J., Das, R.: A study of control parameters affecting online performance of genetic algorithms for function optimization. In: International Conference on Genetic Algorithms, pp. 51–60 (1989)
Conover, W.J.: Practical Nonparametric Statistics. Wiley, New York (1999)
Sheskin, D.: Handbook of Parametric and Nonparametric Statistical Procedures, 5th edn. Chapman and Hall/CRC, New York (2011)
Birattari, M., Stutzle, T., Paquete, L., Varrentrapp, K.: A Racing algorithm for configuring metaheuristics. In: Genetic and Evolutionary Computation Conference, pp. 11–18 (2002)
Rossi-Doria, O., et al.: A comparison of the performance of different metaheuristics on the timetabling problem. In: Burke, E.K., De Causmaecker, P. (eds.) PATAT 2003. LNCS, vol. 2740, pp. 329–351. Springer, Heidelberg (2003)
Paquete, L., Stutzle, T.: A study of stochastic local search algorithms for the biobjective quadratic assignment problem with correlated flow matrices. Eur. J. Oper. Res. 169(3), 943–959 (2006)
Balaprakash, P., Birattari, M., Stützle, T., Dorigo, M.: Adaptive sample size and importance sampling in estimation-based local search for the probabilistic traveling salesman problem. Eur. J. Oper. Res. 199(1), 98–110 (2009)
Chiarandini, M., Stutzle, T.: Stochastic local search algorithms for graph set T-colouring and frequency assignment. Constraints 12(3), 371–403 (2007)
Di Gaspero, L., Roli, A.: Stochastic local search for large-scale instances of the haplotype inference problem by pure parsimony. J. Algorithms 63(3), 55–69 (2008)
Lenne, R., Solnon, C., Stutzle, T., Tannier, E., Birattari, M.: Reactive stochastic local search algorithms for the genomic median problem. In: European Conference on Evolutionary Computation in Combinatorial Optimization, pp. 266–276 (2008)
Di Gaspero, L., di Tollo, G., Roli, A., Schaerf, A.: Hybrid local search for constrained financial portfolio selection problems. In: Van Hentenryck, P., Wolsey, L.A. (eds.) CPAIOR 2007. LNCS, vol. 4510, pp. 44–58. Springer, Heidelberg (2007)
Blum, C., Socha, K.: Training feed-forward neural networks with ant colony optimization: an application to pattern classification, p. 6
Yuan, B., Gallagher, M.: Combining meta-EAs and Racing for difficult EA parameter tuning tasks. In: Lobo, F., Lima, C., Michalewicz, Z. (eds.) Parameter Setting in Evolutionary Algorithms, pp. 121–142. Studies in Computational IntelligenceSpringer, Berlin (2007)
Birattari, M., Balaprakash, P., Dorigo, M.: The ACO/F-Race algorithm for combinatorial optimization under uncertainty. In: Doerner, K., Gendreau, M., Greistorfer, P., Gutjahr, W., Hartl, R., Reimann, M. (eds.) Metaheuristics, Operations Research/Computer Science Interfaces Series, pp. 189–203. Springer, Heidelberg (2007)
Yuan, Z., Stützle, T., Birattari, M.: MADS/F-Race: mesh adaptive direct search meets F-Race. In: GarcÃa-Pedrajas, N., Herrera, F., Fyfe, C., BenÃtez, J., Ali, M. (eds.) IEA/AIE 2010, Part I. LNCS, vol. 6096, pp. 41–50. Springer, Heidelberg (2010)
Yuan, Z., Montes de Oca, M., Birattari, M., Stützle, T.: Continuous optimization algorithms for tuning real and integer parameters of swarm intelligence algorithms. Swarm Intell. 6(1), 49–75 (2012)
Chiarandini, M., Birattari, M., Socha, K., Rossi-Doria, O.: An effective hybrid algorithm for university course timetabling. J. Sched. 9(5), 403–432 (2006)
Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T.: F-Race and iterated F-Race: an overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 311–336. Springer, Berlin (2010)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Branke, J., Elomari, J. (2013). Racing with a Fixed Budget and a Self-Adaptive Significance Level. In: Nicosia, G., Pardalos, P. (eds) Learning and Intelligent Optimization. LION 2013. Lecture Notes in Computer Science(), vol 7997. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-44973-4_29
Download citation
DOI: https://doi.org/10.1007/978-3-642-44973-4_29
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-44972-7
Online ISBN: 978-3-642-44973-4
eBook Packages: Computer ScienceComputer Science (R0)