Skip to main content
SpringerLink
Log in

Optimization of heuristic search using recursive algorithm selection and reinforcement learning

  • Published:
Annals of Mathematics and Artificial Intelligence Aims and scope Submit manuscript

Abstract

The traditional approach to computational problem solving is to use one of the available algorithms to obtain solutions for all given instances of a problem. However, typically not all instances are the same, nor a single algorithm performs best on all instances. Our work investigates a more sophisticated approach to problem solving, called Recursive Algorithm Selection, whereby several algorithms for a problem (including some recursive ones) are available to an agent that makes an informed decision on which algorithm to select for handling each sub-instance of a problem at each recursive call made while solving an instance. Reinforcement learning methods are used for learning decision policies that optimize any given performance criterion (time, memory, or a combination thereof) from actual execution and profiling experience. This paper focuses on the well-known problem of state-space heuristic search and combines the A* and RBFS algorithms to yield a hybrid search algorithm, whose decision policy is learned using the Least-Squares Policy Iteration (LSPI) algorithm. Our benchmark problem domain involves shortest path finding problems in a real-world dataset encoding the entire street network of the District of Columbia (DC), USA. The derived hybrid algorithm exhibits better performance results than the individual algorithms in the majority of cases according to a variety of performance criteria balancing time and memory. It is noted that the proposed methodology is generic, can be applied to a variety of other problems, and requires no prior knowledge about the individual algorithms used or the properties of the underlying problem instances being solved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Bast, H., Funke, S., Sanders, P., Schultes, D.: Fast routing in road networks with transit nodes. Science 316(5824), 566 (2007)

    Article  MathSciNet  Google Scholar 

  2. Bellman, R.: Dynamic Programming. Princeton University Press (1957)

  3. Bertsekas, D.P., Tsitsiklis, J.N.: Neuro-Dynamic Programming. Athena Scientific (1996)

  4. Chakrabarti, P.P., Ghose, S., Acharya, A., Sarkar, S.C.D.: Heuristic search in restricted memory. Artif. Intell. 41(2), 197–221 (1989)

    Article  MATH  Google Scholar 

  5. Cherkassky, B.V., Goldberg, A.V., Radzik, T.: Shortest paths algorithms: theory and experimental evaluation. Math. Program. 73, 129–174 (1996)

    MATH  MathSciNet  Google Scholar 

  6. Cormen, T.H., Leiserson, C.E., Rivest, R.L., Stein, C.: Introduction to Algorithms. The MIT Press and McGraw-Hill Book Company (2001)

  7. Delling, D., Sanders, P., Schultes, D., Wagner, D.: Engineering route planning algorithms. In: Lerner, J., Wagner, D., Zweig, K. (eds.) Algorithmics of Large and Complex Networks. Lecture Notes in Computer Science, vol. 5515, pp. 117–139. Springer (2009)

  8. Dijkstra, E.W.: A note on two problems in connexion with graphs. Numer. Math. 1, 269–271 (1959)

    Article  MATH  MathSciNet  Google Scholar 

  9. Gagliolo, M., Schmidhuber, J.: Learning dynamic algorithm portfolios. Ann. Math. Artif. Intell. 47(3–4), 295–328 (2006)

    MATH  MathSciNet  Google Scholar 

  10. Geisberger, R., Sanders, P., Schultes, D., Delling, D.: Contraction hierarchies: faster and simpler hierarchical routing in road networks. In: Proceedings of the 7th International Conference on Experimental Algorithms, pp. 319–333 (2008)

  11. Goldberg, A.V., Harrelson, C.: Computing the shortest path: A* search meets graph theory. In: Proceedings of the 16th Annual ACM-SIAM Symposium on Discrete Algorithms, pp. 156–165 (2005)

  12. Gomes, C.P., Selman, B., Crato, N., Kautz, H.: Heavy-tailed phenomena in satisfiability and constraint satisfaction problems. J. Autom. Reason. 24(1–2), 67–100 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  13. Guo, H.: Algorithm selection for sorting and probabilistic inference: a machine learning-based approach. PhD thesis, Kansas State University, Manhattan, USA (2003)

  14. Hart, P.E., Nilsson, N.J., Raphael, B.: A formal basis for the heuristic determination of minimum cost paths. IEEE Trans. Syst. Sci. Cybern. 4(2), 100–107 (1968)

    Article  Google Scholar 

  15. Howard, R.A.: Dynamic Programming and Markov Processes. The MIT Press (1960)

  16. Huberman, B.A., Lukose, R.M., Hogg, T.: An economics approach to hard computational problems. Science 275(5296), 51–54 (1997)

    Article  Google Scholar 

  17. Hutter, F., Hoos, H.H., Stützle, T.: Automatic algorithm configuration based on local search. In: Proceedings of the 22nd National Conference on Artificial Intelligence, pp. 1152–1157 (2007)

  18. Kaelbling, L.P., Littman, M., Moore, A.: Reinforcement learning: a survey. J. Artif. Intell. Res. 4, 237–285 (1996)

    Google Scholar 

  19. Korf, R.E.: Real-time heuristic search. Artif. Intell. 42(3), 189–212 (1990)

    Article  MATH  Google Scholar 

  20. Lagoudakis, M.G., Littman, M.L.: Algorithm selection using reinforcement learning. In: Proceedings of the 17th International Conference on Machine Learning, pp. 511–518 (2000)

  21. Lagoudakis, M.G., Littman, M.L.: Learning to select branching rules in the DPLL procedure for satisfiability. In: Proceedings of the 2001 Workshop on Theory and Applications of Satisfiability Testing, pp. 344–359 (2001)

  22. Lagoudakis, M.G., Parr, R.: Least-squares policy iteration. J. Mach. Learn. Res. 4, 1107–1149 (2003)

    Article  MathSciNet  Google Scholar 

  23. Lagoudakis, M.G., Littman, M.L., Parr, R.: Selecting the right algorithm. In: Proceedings of the 2001 AAAI Fall Symposium Series: Using Uncertainty within Computation (2001)

  24. Musser, D.R.: Introspective sorting and selection algorithms. Softw. Pract. Exper. 27(8), 983–993 (1997)

    Article  Google Scholar 

  25. Puterman, M.L.: Markov Decision Processes—Discrete Stochastic Dynamic Programming. Wiley, Inc (1994)

  26. Rice, J.R.: The algorithm selection problem. Adv. Comput. 15, 65–118 (1976)

    Article  Google Scholar 

  27. Russell, S.: Efficient memory-bounded search methods. In: Proceedings of the 10th European Conference on Artificial Intelligence, pp. 1–5 (1992)

  28. Russell, S., Norvig, P.: Artificial Intelligence: A Modern Approach, 2nd edn. Prentice Hall (2003)

  29. Sanders, P., Schultes, D.: Engineering highway hierarchies. In: Proceedings of the 14th European Symposium on Algorithms, pp. 804–816 (2006)

  30. Smith-Miles, K.A.: Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput. Surv. 41(1), 1–25 (2008)

    Article  Google Scholar 

  31. Sutton, R., Barto, A.: Reinforcement Learning: An Introduction. The MIT Press (1998)

  32. TIGER/Line: Topologically Integrated Geographic Encoding and Referencing System: US Road Network Data. http://www.census.gov/geo/www/tiger/ (2008)

  33. Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1, 67–82 (1997)

    Article  Google Scholar 

  34. Xu, L., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Satzilla-07: the design and analysis of an algorithm portfolio for SAT. In: Proceedings of the 13th International Conference on Principles and Practice of Constraint Programming, pp. 712–727 (2007)

Download references

Author information

Author notes

  1. Vasileios Vasilikos

    Present address: Faculty of Electrical Engineering, Mathematics and Computer Science, Delft University of Technology, Delft, The Netherlands

Authors and Affiliations

  1. Intelligent Systems Laboratory, Department of Electronic and Computer Engineering, Technical University of Crete, Kounoupidiana University Campus, Chania, Crete, 73100, Greece

    Vasileios Vasilikos & Michail G. Lagoudakis

Authors
  1. Vasileios Vasilikos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Michail G. Lagoudakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michail G. Lagoudakis.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Vasilikos, V., Lagoudakis, M.G. Optimization of heuristic search using recursive algorithm selection and reinforcement learning. Ann Math Artif Intell 60, 119–151 (2010). https://doi.org/10.1007/s10472-010-9217-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10472-010-9217-7

Keywords

Mathematics Subject Classifications (2010)

Search

Navigation