skip to main content
10.1145/3638530.3654264acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
poster

Extending Instance Space Analysis to Algorithm Configuration Spaces

Published: 01 August 2024 Publication History

Abstract

This paper describes an approach for deriving visual insights about the joint relationship between algorithm performance, algorithm parameters, and problem instance features. This involves the combined analysis and exploration of a 2D instance space, to which instances from some problem space are projected, and a 2D configuration space, to which (algorithm) configurations are projected. Extending on the dimensionality reduction problem solved in Instance Space Analysis, we define an optimisation problem for finding projections to these two spaces, with an interpretable relationship between them. Then, we describe the tools developed for probing those spaces in an investigation of the question: What characterises the algorithm configurations that perform best on a selected group of instances (or vice versa)? We demonstrate the use of these tools on synthetic data with known ground truth.

References

[1]
André Biedenkapp, Joshua Marben, Marius Thomas Lindauer, and Frank Hutter. 2018. CAVE: Configuration assessment, visualization and evaluation. In Learning and Intelligent Optimization, 12th International Conference, LION 12, Roberto Battiti, Mauro Brunato, Ilias Kotsireas, and Panos M. Pardalos (Eds.). Lecture Notes in Computer Science, Vol. 11353. Springer, Cham, Switzerland, 115--130.
[2]
Winston Chang, Joe Cheng, JJ Allaire, Carson Sievert, Barret Schloerke, Yihui Xie, Jeff Allen, Jonathan McPherson, Alan Dipert, and Barbara Borges. 2022. shiny: Web Application Framework for R. https://CRAN.R-project.org/package=shiny R package version 1.7.4.
[3]
François Chollet et al. 2015. Keras. https://keras.io.
[4]
Nguyen Dang and Patrick De Causmaecker. 2019. Analysis of Algorithm Components and Parameters: Some Case Studies. In Learning and Intelligent Optimization, 13th International Conference, LION 13, Nikolaos F. Matsatsinis, Yannis Marinakis, and Panos M. Pardalos (Eds.). Lecture Notes in Computer Science, Vol. 11968. Springer, Cham, Switzerland, 288--303.
[5]
Martín Abadi et al. 2015. TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems. https://www.tensorflow.org/ Software available from tensorflow.org.
[6]
Giles Hooker. 2004. Discovering Additive Structure in Black Box Functions. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, KDD'04, Won Kim, Ronny Kohavi, Johannes Gehrke, and William DuMouchel (Eds.). ACM Press, New York, NY, 575--580.
[7]
Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. 2014. An Efficient Approach for Assessing Hyperparameter Importance. In Proceedings of the 31st International Conference on Machine Learning, ICML 2014, Eric P. Xing and Tony Jebara (Eds.), Vol. 32. PMLR, 754--762. https://proceedings.mlr.press/v32/hutter14.html
[8]
Frank Hutter, Lin Xu, Holger H. Hoos, and Kevin Leyton-Brown. 2014. Algorithm Runtime Prediction: Methods & evaluation. Artificial Intelligence 206 (2014), 79--111.
[9]
Plotly Technologies Inc. 2015. Collaborative data science. Montreal, QC. https://plot.ly
[10]
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7--9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).
[11]
Daniel Lewandowski, Dorota Kurowicka, and Harry Joe. 2009. Generating Random Correlation Matrices Based on Vines and Extended Onion Method. Journal of Multivariate Analysis 100, 9 (2009), 1989--2001.
[12]
Mario A. Muñoz, Laura Villanova, Davaatseren Baatar, and Kate Smith-Miles. 2018. Instance Spaces for Machine Learning Classification. Machine Learning 107, 1 (2018), 109--147.
[13]
Kate Smith-Miles, Davaatseren Baatar, Brendan Wreford, and Rhyd M. R. Lewis. 2014. Towards Objective Measures of Algorithm Performance across Instance Space. Computers & Operations Research 45 (2014), 12--24.
[14]
Kate Smith-Miles and Mario A. Muñoz. 2023. Instance Space Analysis for Algorithm Testing: Methodology and Software Tools. Comput. Surveys 55, 12, Article 255 (March 2023), 31 pages.
[15]
Kate Smith-Miles, Mario A. Muñoz, and Neelofar. 2020. Melbourne Algorithm Test Instance Library with Data Analytics (MATILDA). https://matilda.unimelb.edu.au/
[16]
Jan N. van Rijn and Frank Hutter. 2018. Hyperparameter Importance Across Datasets. In 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Yike Guo and Faisal Farooq (Eds.). ACM Press, New York, NY, 2367--2376.

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
GECCO '24 Companion: Proceedings of the Genetic and Evolutionary Computation Conference Companion
July 2024
2187 pages
ISBN:9798400704956
DOI:10.1145/3638530
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the owner/author(s).

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 August 2024

Check for updates

Author Tags

  1. algorithm configuration
  2. instance space analysis
  3. explainability

Qualifiers

  • Poster

Funding Sources

Conference

GECCO '24 Companion
Sponsor:

Acceptance Rates

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 77
    Total Downloads
  • Downloads (Last 12 months)77
  • Downloads (Last 6 weeks)8
Reflects downloads up to 18 Feb 2025

Other Metrics

Citations

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media