Skip to main content
SpringerLink
Log in

Hybrid Ranking and Regression for Algorithm Selection

  • Conference paper
  • First Online:
KI 2020: Advances in Artificial Intelligence (KI 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12325))

Included in the following conference series:

Abstract

Algorithm selection (AS) is defined as the task of automatically selecting the most suitable algorithm from a set of candidate algorithms for a specific instance of an algorithmic problem class. While suitability may refer to different criteria, runtime is of specific practical relevance. Leveraging empirical runtime information as training data, the AS problem is commonly tackled by fitting a regression function, which can then be used to estimate the candidate algorithms’ runtimes for new problem instances. In this paper, we develop a new approach to algorithm selection that combines regression with ranking, also known as learning to rank, a problem that has recently been studied in the realm of preference learning. Since only the ranking of the algorithms is eventually needed for the purpose of selection, the precise numerical estimation of runtimes appears to be a dispensable and unnecessarily difficult problem. However, discarding the numerical runtime information completely seems to be a bad idea, as we hide potentially useful information about the algorithms’ performance margins from the learner. Extensive experimental studies confirm the potential of our hybrid approach, showing that it often performs better than pure regression and pure ranking methods.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Ties are mainly caused by timeouts in the “ground truth” data but rarely occur in the predicted performances.

  2. 2.

    https://github.com/JonasHanselle/CoRRAS.

References

  1. Bischl, B., et al.: Aslib: A benchmark library for algorithm selection. Artif. Intell. 237, 41–58 (2016)

    Article  MathSciNet  MATH  Google Scholar 

  2. Byrd, R.H., Lu, P., Nocedal, J., Zhu, C.: A limited memory algorithm for bound constrained optimization. SIAM J. Sci. Comput. 16(5), 1190–1208 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  3. Cheng, W., Dembczynski, K., Hüllermeier, E.: Label ranking methods based on the Plackett-Luce model. In: Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21–24, 2010, Haifa, Israel, pp. 215–222. Omnipress (2010)

    Google Scholar 

  4. Collautti, M., Malitsky, Y., Mehta, D., O’Sullivan, B.: SNNAP: Solver-based nearest neighbor for algorithm portfolios. In: Blockeel, H., Kersting, K., Nijssen, S., Železný, F. (eds.) ECML PKDD 2013. LNCS (LNAI), vol. 8190, pp. 435–450. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40994-3_28

    Chapter  Google Scholar 

  5. Friedman, J.H.: Multivariate adaptive regression splines. Ann. Stat., 1–67 (1991)

    Google Scholar 

  6. Haim, S., Walsh, T.: Restart strategy selection using machine learning techniques. In: Kullmann, O. (ed.) SAT 2009. LNCS, vol. 5584, pp. 312–325. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02777-2_30

    Chapter  Google Scholar 

  7. Hutter, F., Xu, L., Hoos, H.H., Leyton-Brown, K.: Algorithm runtime prediction: Methods & evaluation. Artif. Intell. 206, 79–111 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  8. Kanda, J., Soares, C., Hruschka, E., de Carvalho, A.: A meta-learning approach to select meta-heuristics for the traveling salesman problem using MLP-based label ranking. In: Huang, T., Zeng, Z., Li, C., Leung, C.S. (eds.) ICONIP 2012. LNCS, vol. 7665, pp. 488–495. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-34487-9_59

    Chapter  Google Scholar 

  9. Kendall, M.G.: The treatment of ties in ranking problems. Biometrika 33(3), 239–251 (1945)

    Article  MathSciNet  MATH  Google Scholar 

  10. Kerschke, P., Hoos, H.H., Neumann, F., Trautmann, H.: Automated algorithm selection: Survey and perspectives. Evol. Comput. 27(1), 3–45 (2019)

    Article  Google Scholar 

  11. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7–9, Conference Track Proceedings (2015)

    Google Scholar 

  12. Leyton-Brown, K., Nudelman, E., Shoham, Y.: Learning the empirical hardness of optimization problems: The case of combinatorial auctions. In: Van Hentenryck, P. (ed.) CP 2002. LNCS, vol. 2470, pp. 556–572. Springer, Heidelberg (2002). https://doi.org/10.1007/3-540-46135-3_37

    Chapter  Google Scholar 

  13. Luce, R.D.: Individual choice behavior. John Wiley, Oxford, England (1959)

    Google Scholar 

  14. Oentaryo, R.J., Handoko, S.D., Lau, H.C.: Algorithm selection via ranking. In: Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25–30, 2015, Austin, Texas, USA, pp. 1826–1832 (2015)

    Google Scholar 

  15. Pihera, J., Musliu, N.: Application of machine learning to algorithm selection for TSP. In: 26th IEEE International Conference on Tools with Artificial Intelligence, ICTAI 2014, Limassol, Cyprus, November 10–12, pp. 47–54. IEEE Computer Society (2014)

    Google Scholar 

  16. Plackett, R.L.: The analysis of permutations. J. Roy. Stat. Soc. Ser. C (Appl. Stat.) 24(2), 193–202 (1975)

    Google Scholar 

  17. Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)

    Article  Google Scholar 

  18. de Sá, C.R., Soares, C., Knobbe, A.J., Cortez, P.: Label ranking forests. Exp. Syst. 34(1) (2017)

    Google Scholar 

  19. Schäfer, D., Hüllermeier, E.: Dyad ranking using Plackett-Luce models based on joint feature representations. Mach. Learn. 107(5), 903–941 (2018). https://doi.org/10.1007/s10994-017-5694-9

  20. Sculley, D.: Combined regression and ranking. In: Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, July 25–28, pp. 979–988 (2010)

    Google Scholar 

  21. Tornede, A., Wever, M., Hüllermeier, E.: Algorithm selection as recommendation: From collaborative filtering to dyad ranking. In: CI Workshop, Dortmund (2019)

    Google Scholar 

  22. Tornede, A., Wever, M., Hüllermeier, E.: Extreme algorithm selection with dyadic feature representation. CoRR abs/2001.10741 (2020)

    Google Scholar 

  23. Vembu, S., Gärtner, T.: Label ranking algorithms: A survey. In: In: Fürnkranz J., Hüllermeier E. (eds.) Preference Learning, pp. 45–64. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14125-6_3

  24. Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Satzilla: Portfolio-based algorithm selection for SAT. J. Artif. Intell. Res. 32, 565–606 (2008)

    Article  MATH  Google Scholar 

  25. Zhu, C., Byrd, R.H., Lu, P., Nocedal, J.: Algorithm 778: L-BFGS-B: Fortran subroutines for large-scale bound-constrained optimization. ACM Trans. Math. Softw. 23(4), 550–560 (1997)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgements

This work was supported by the German Federal Ministry of Economic Affairs and Energy (BMWi) within the “Innovationswettbewerb Künstliche Intelligenz” and the German Research Foundation (DFG) within the Collaborative Research Center “On-The-Fly Computing” (SFB 901/3 project no. 160364472). The authors also gratefully acknowledge support of this project through computing time provided by the Paderborn Center for Parallel Computing (PC\(^2\)).

Author information

Authors and Affiliations

  1. Heinz Nixdorf Institute and Department of Computer Science, Paderborn University, Paderborn, Germany

    Jonas Hanselle, Alexander Tornede, Marcel Wever & Eyke Hüllermeier

Authors
  1. Jonas Hanselle
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Alexander Tornede
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marcel Wever
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Eyke Hüllermeier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jonas Hanselle .

Editor information

Editors and Affiliations

  1. Universität Bamberg, Bamberg, Germany

    Ute Schmid

  2. Örebro University, Örebro, Sweden

    Franziska Klügl

  3. Universität Bamberg, Bamberg, Germany

    Diedrich Wolter

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Hanselle, J., Tornede, A., Wever, M., Hüllermeier, E. (2020). Hybrid Ranking and Regression for Algorithm Selection. In: Schmid, U., Klügl, F., Wolter, D. (eds) KI 2020: Advances in Artificial Intelligence. KI 2020. Lecture Notes in Computer Science(), vol 12325. Springer, Cham. https://doi.org/10.1007/978-3-030-58285-2_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-58285-2_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-58284-5

  • Online ISBN: 978-3-030-58285-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation