Abstract
In recent years, general-purpose automated algorithm configuration procedures have enabled impressive improvements in the state of the art in solving a wide range of challenging problems from AI, operations research and other areas. To search vast combinatorial spaces of parameter settings for a given algorithm as efficiently as possible, the most successful configurators combine techniques such as racing, estimation of distribution algorithms, Bayesian optimisation and model-free stochastic search. Two of the most widely used general-purpose algorithm configurators, SMAC and irace, can be seen as combinations of Bayesian optimisation and racing, and of racing and an estimation of distribution algorithm, respectively. Here, we propose a first approach that combines all three of these techniques into one single configurator, while exploiting prior knowledge contained in expert-chosen default parameter values. We demonstrate significant performance improvements over irace and SMAC on a broad range of running time optimisation scenarios from AClib.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Ahmadizadeh, K., Dilkina, B., Gomes, C.P., Sabharwal, A.: An empirical study of optimization for maximizing diffusion in networks. In: Cohen, D. (ed.) CP 2010. LNCS, vol. 6308, pp. 514–521. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-15396-9_41
Anastacio, M., Luo, C., Hoos, H.: Exploitation of default parameter values in automated algorithm configuration. In: Workshop Data Science Meets Optimisation (DSO), IJCAI 2019, August 2019
Ansótegui, C., Malitsky, Y., Samulowitz, H., Sellmann, M., Tierney, K.: Model-based genetic algorithms for algorithm configuration. In: Proceedings of the IJCAI 2015, pp. 733–739 (2015)
Ansótegui, C., Sellmann, M., Tierney, K.: A gender-based genetic algorithm for the automatic configuration of algorithms. In: Gent, I.P. (ed.) CP 2009. LNCS, vol. 5732, pp. 142–157. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-04244-7_14
Atamtürk, A., Muñoz, J.C.: A study of the lot-sizing polytope. Math. Program. 99(3), 443–465 (2004). https://doi.org/10.1007/s10107-003-0465-8
Balint, A., Manthey, N.: SparrowToRiss. In: Proceedings of the SAT Competition 2014, pp. 77–78 (2014)
Biere, A.: Yet another local search solver and lingeling and friends entering the SAT competition 2014 (2014)
Birattari, M., Yuan, Z., Balaprakash, P., Stützle, T.: F-Race and iterated F-Race: an overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds.) Experimental Methods for the Analysis of Optimization Algorithms, pp. 311–336. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-02538-9_13
Brummayer, R., Lonsing, F., Biere, A.: Automated testing and debugging of SAT and QBF solvers. In: Strichman, O., Szeider, S. (eds.) SAT 2010. LNCS, vol. 6175, pp. 44–57. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14186-7_6
Cáceres, L.P., López-Ibáñez, M., Hoos, H., Stützle, T.: An experimental study of adaptive capping in irace. In: Battiti, R., Kvasov, D.E., Sergeyev, Y.D. (eds.) LION 2017. LNCS, vol. 10556, pp. 235–250. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-69404-7_17
Domhan, T., Springenberg, J.T., Hutter, F.: Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves. In: Yang, Q., Wooldridge, M.J. (eds.) IJCAI 2015, pp. 3460–3468. AAAI Press (2015)
Fawcett, C., Helmert, M., Hoos, H., Karpas, E., Röger, G., Seipp, J.: FD-Autotune: domain-specific configuration using fast downward. In: Proceedings of the ICAPS Workshop, PAL 2011, pp. 13–20 (2011)
Fawcett, C., Hoos, H.H.: Analysing differences between algorithm configurations through ablation. J. Heuristics 22(4), 431–458 (2015). https://doi.org/10.1007/s10732-014-9275-9
Gebser, M., Kaufmann, B., Schaub, T.: Conflict-driven answer set solving: from theory to practice. Artif. Intell. 187–188, 52–89 (2012)
Gerevini, A., Saetti, A., Serina, I.: Planning through stochastic local search and temporal action graphs in LPG. J. Artif. Intell. Res. 20, 239–290 (2003)
Gerevini, A., Saetti, A., Serina, I.: An approach to efficient planning with numerical fluents and multi-criteria plan quality. Artif. Intell. 172, 899–944 (2008)
Gerevini, A., Saetti, A., Serina, I.: An empirical analysis of some heuristic features for planning through local search and action graphs. Fundamenta Informaticae 107(2–3), 167–197 (2011)
Gomes, C.P., van Hoeve, W.-J., Sabharwal, A.: Connections in networks: a hybrid approach. In: Perron, L., Trick, M.A. (eds.) CPAIOR 2008. LNCS, vol. 5015, pp. 303–307. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-68155-7_27
Hauschild, M., Pelikan, M.: An introduction and survey of estimation of distribution algorithms. Swarm Evol. Comput. 1(3), 111–128 (2011)
Hoos, H.H.: Automated algorithm configuration and parameter tuning. In: Hamadi, Y., Monfroy, E., Saubion, F. (eds.) Autonomous Search, pp. 37–71. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21434-9_3
Hoos, H.H.: Programming by optimization. Commun. ACM 55(2), 70–80 (2012)
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Automated configuration of mixed integer programming solvers. In: Lodi, A., Milano, M., Toth, P. (eds.) CPAIOR 2010. LNCS, vol. 6140, pp. 186–202. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-13520-0_23
Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Coello, C.A.C. (ed.) LION 2011. LNCS, vol. 6683, pp. 507–523. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25566-3_40
Hutter, F., Hoos, H.H., Leyton-Brown, K., Stützle, T.: ParamILS: an automatic algorithm configuration framework. J. Artif. Intell. Res. 36, 267–306 (2009)
Hutter, F., Lindauer, M., Balint, A., Bayless, S., Hoos, H.H., Leyton-Brown, K.: The configurable SAT solver challenge (CSSC). Artif. Intell. 243, 1–25 (2017)
Hutter, F., et al.: AClib: a benchmark library for algorithm configuration. In: Pardalos, P.M., Resende, M.G.C., Vogiatzis, C., Walteros, J.L. (eds.) LION 2014. LNCS, vol. 8426, pp. 36–40. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-09584-4_4
Kotthoff, L., Thornton, C., Hoos, H.H., Hutter, F., Leyton-Brown, K.: Auto-WEKA 2.0: automatic model selection and hyperparameter optimization in WEKA. J. Mach. Learn. Res. 18, 25:1–25:5 (2017)
Leyton-Brown, K., Pearson, M., Shoham, Y.: Towards a universal test suite for combinatorial auction algorithms. In: Jhingran, A., Mackie-Mason, J., Tygar, D.J. (eds.) Proceedings of the 2nd ACM Conference on Electronic Commerce (EC-00), Minneapolis, MN, USA, 17–20 October 2000, pp. 66–76. ACM (2000)
Lindauer, M., Hutter, F.: Warmstarting of model-based algorithm configuration. In: Proceedings of the AAAI-18, IAAI-18, and EAAI-18, pp. 1355–1362 (2018)
Long, D., Fox, M.: The 3rd international planning competition: results and analysis. J. Artif. Intell. Res. 20, 1–59 (2003)
López-Ibáñez, M., Dubois-Lacoste, J., Pérez Cáceres, L., Stützle, T., Birattari, M.: The irace package: iterated racing for automatic algorithm configuration. Oper. Res. Perspect. 3, 43–58 (2016)
Luo, C., Hoos, H.H., Cai, S., Lin, Q., Zhang, H., Zhang, D.: Local search with efficient automatic configuration for minimum vertex cover. In: IJCAI-19, pp. 1297–1304. International Joint Conferences on Artificial Intelligence Organization (2019)
Mugrauer, F., Balint, A.: Sat encoded low autocorrelation binary sequence (labs) benchmark description. In: Proceedings of the SAT Competition 2013, pp. 117–118 (2013)
Penberthy, J.S., Weld, D.S.: Temporal planning with continuous change. In: Proceedings of the AAAI-94, vol. 2, pp. 1010–1015 (1994)
Pushak, Y., Hoos, H.: Algorithm configuration landscapes: more benign than expected? In: Auger, A., Fonseca, C.M., Lourenço, N., Machado, P., Paquete, L., Whitley, D. (eds.) PPSN 2018. LNCS, vol. 11102, pp. 271–283. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-99259-4_22
Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: The 19th ACM SIGKDD, KDD 2013, pp. 847–855 (2013)
Vallati, M., Fawcett, C., Gerevini, A., Hoos, H.H., Saetti, A.: Automatic generation of efficient domain-optimized planners from generic parametrized planners. In: Proceedings of the RCRA 2011, pp. 111–123 (2011)
Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Hydra-MIP: automated algorithm configuration and selection for mixed integer programming. In: Proceedings of the RCRA 2011, pp. 16–30 (2011)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Appendix: Robustness to Misleading Default Values
Appendix: Robustness to Misleading Default Values
To obtain better insights into the robustness of our approach to misleading default values, we generated random configurations until we found one that performed worse than the default, but produced time-outs on fewer than a third of the training instances from our three SAT benchmarks. Then, we repeated a few configuration experiments from Sect. 5.1, using these new, misleading default configurations (using the same protocol as described earlier). The results of this experiment are shown in Table 5.
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Anastacio, M., Hoos, H. (2020). Model-Based Algorithm Configuration with Default-Guided Probabilistic Sampling. In: Bäck, T., et al. Parallel Problem Solving from Nature – PPSN XVI. PPSN 2020. Lecture Notes in Computer Science(), vol 12269. Springer, Cham. https://doi.org/10.1007/978-3-030-58112-1_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-58112-1_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58111-4
Online ISBN: 978-3-030-58112-1
eBook Packages: Computer ScienceComputer Science (R0)