Abstract
The hyperparameter configuration of machine learning models has a great influence on their performance. These hyperparameters are often set either manually w. r. t. to the experience of an expert or by an Automated Hyperparameter Optimization (HPO) method. However, integrating experience knowledge into HPO methods is challenging. Therefore, we propose the approach HypOCBR (Hyperparameter Optimization with Case-Based Reasoning) that uses Case-Based Reasoning (CBR) to improve the optimization of hyperparameters. HypOCBR is used as an addition to HPO methods and builds up a case base of sampled hyperparameter vectors with their loss values. The case base is then used to retrieve hyperparameter vectors given a query vector and to make decisions whether to proceed trialing with this query or abort and sample another vector. The experimental evaluation investigates the suitability of HypOCBR for two deep learning setups of varying complexity. It shows its potential to improve the optimization results, especially in complex scenarios with limited optimization time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The example is derived from an introduction on convolutional neural networks, accessible at https://www.tensorflow.org/tutorials/images/cnn.
- 2.
References
Aamodt, A., Plaza, E.: Case-based reasoning: Foundational issues, methodological variations, and system approaches. AI Commun. 7(1), 39–59 (1994)
Amin, K., Lancaster, G., Kapetanakis, S., Althoff, K.-D., Dengel, A., Petridis, M.: Advanced similarity measures using word embeddings and siamese networks in CBR. In: Bi, Y., Bhatia, R., Kapoor, S. (eds.) IntelliSys 2019. AISC, vol. 1038, pp. 449–462. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-29513-4_32
Auslander, B., Apker, T., Aha, D.W.: Case-based parameter selection for plans: coordinating autonomous vehicle teams. In: Lamontagne, L., Plaza, E. (eds.) ICCBR 2014. LNCS (LNAI), vol. 8765, pp. 32–47. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11209-1_4
Bergmann, R.: Experience Management: Foundations, Development Methodology, and Internet-Based Applications. LNCS, vol. 2432. Springer, Heidelberg (2002)
Bergmann, R., Grumbach, L., Malburg, L., Zeyen, C.: ProCAKE: a Process-oriented case-based reasoning framework. In: Workshop Proceedings of ICCBR, vol. 2567, pp. 156–161. CEUR-WS.org (2019)
Bergstra, J., Bengio, Y.: Random search for hyper-parameter optimization. J. Mach. Learn. Res. 13(2), 281–305 (2012)
Claesen, M., de Moor, B.: Hyperparameter search in machine learning. CoRR abs/1502.02127 (2015)
Falkner, S., Klein, A., Hutter, F.: Bohb: robust and efficient hyperparameter optimization at scale. In: ICML, pp. 1437–1446 (2018)
Feurer, M., Hutter, F.: Hyperparameter optimization. In: Hutter, F., Kotthoff, L., Vanschoren, J. (eds.) Automated Machine Learning. TSSCML, pp. 3–33. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-05318-5_1
Hoffmann, M., Bergmann, R.: Using graph embedding techniques in process-oriented case-based reasoning. Algorithms 15(2), 27 (2022)
Hoffmann, M., Malburg, L., Klein, P., Bergmann, R.: Using siamese graph neural networks for similarity-based retrieval in process-oriented case-based reasoning. In: Watson, I., Weber, R. (eds.) ICCBR 2020. LNCS (LNAI), vol. 12311, pp. 229–244. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58342-2_15
Kennedy, J., Eberhart, R.: Particle swarm optimization. In: Proceedings of International Conference on Neural Networks (ICNN’95), Perth, WA, Australia, 27 November - 1 December, 1995, pp. 1942–1948. IEEE (1995)
Leake, D., Crandall, D.: On bringing case-based reasoning methodology to deep learning. In: Watson, I., Weber, R. (eds.) ICCBR 2020. LNCS (LNAI), vol. 12311, pp. 343–348. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58342-2_22
Leake, D., Schack, B.: Exploration vs. exploitation in case-base maintenance: leveraging competence-based deletion with ghost cases. In: Cox, M.T., Funk, P., Begum, S. (eds.) ICCBR 2018. LNCS (LNAI), vol. 11156, pp. 202–218. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01081-2_14
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: a novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. 18(1), 6765–6816 (2018)
Luo, G.: A review of automatic selection methods for machine learning algorithms and hyper-parameter values. Netw. Model. Anal. Health Inform. Bioinf. 5(1), 1–16 (2016). https://doi.org/10.1007/s13721-016-0125-6
Malburg, L., Hoffmann, M., Trumm, S., Bergmann, R.: Improving similarity-based retrieval efficiency by using graphic processing units in case-based reasoning. In: Proceedings of the 34th FLAIRS Conference FloridaOJ (2021)
Mathisen, B.M., Bach, K., Aamodt, A.: Using extended siamese networks to provide decision support in aquaculture operations. Appl. Intell. 51(11), 8107–8118 (2021). https://doi.org/10.1007/s10489-021-02251-3
Molina, M.M., Luna, J.M., Romero, C., Ventura, S.: Meta-learning approach for automatic parameter tuning: a case study with educational datasets. In: Proceedings of the 5th International Conference on Educational Data Mining, pp. 180–183. Chania, Greece (2012)
Pavón, R., Díaz, F., Laza, R., Luzón, V.: Automatic parameter tuning with. a bayesian case-based reasoning system a case of study. Expert Syst. Appl. 36(2), 3407–3420 (2009)
Quijano-Sánchez, L., Bridge, D., Díaz-Agudo, B., Recio-García, J.A.: A case-based solution to the cold-start problem in group recommenders. In: Agudo, B.D., Watson, I. (eds.) ICCBR 2012. LNCS (LNAI), vol. 7466, pp. 342–356. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-32986-9_26
Roth-Berghofer, T.R.: Knowledge Maintenance of Case-Based Reasoning Systems: The SIAM Methodology, Dissertationen zur künstlichen Intelligenz. Akad. Verl.-Ges. Aka, Berlin (2003)
Schafer, J.B., Frankowski, D., Herlocker, J., Sen, S.: Collaborative filtering recommender systems. In: Brusilovsky, P., Kobsa, A., Nejdl, W. (eds.) The Adaptive Web. LNCS, vol. 4321, pp. 291–324. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-72079-9_9
Snoek, J., Larochelle, H., Adams, R.P.: Practical bayesian optimization of machine learning algorithms. In: Advances in Neural Information Processing Systems, vol. 25. Curran Associates, Inc. (2012)
Wettschereck, D., Aha, D.W.: Weighting features. In: Veloso, M., Aamodt, A. (eds.) ICCBR 1995. LNCS, vol. 1010, pp. 347–358. Springer, Heidelberg (1995). https://doi.org/10.1007/3-540-60598-3_31
Yang, L., Shami, A.: On hyperparameter optimization of machine learning algorithms: theory and practice. Neurocomputing 415, 295–316 (2020)
Yeguas, E., Luzón, M.V., Pavón, R., Laza, R., Arroyo, G., Díaz, F.: Automatic parameter tuning for evolutionary algorithms using a bayesian case-based reasoning system. Appl. Soft Comput. 18, 185–195 (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Hoffmann, M., Bergmann, R. (2022). Improving Automated Hyperparameter Optimization with Case-Based Reasoning. In: Keane, M.T., Wiratunga, N. (eds) Case-Based Reasoning Research and Development. ICCBR 2022. Lecture Notes in Computer Science(), vol 13405. Springer, Cham. https://doi.org/10.1007/978-3-031-14923-8_18
Download citation
DOI: https://doi.org/10.1007/978-3-031-14923-8_18
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-14922-1
Online ISBN: 978-3-031-14923-8
eBook Packages: Computer ScienceComputer Science (R0)