skip to main content
10.1145/1276958.1276979acmconferencesArticle/Chapter ViewAbstractPublication PagesgeccoConference Proceedingsconference-collections
Article

Analyzing heuristic performance with response surface models: prediction, optimization and robustness

Published: 07 July 2007 Publication History

Abstract

This research uses a Design of Experiments (DOE) approach to build a predictive model of the performance of a combinatorial optimization heuristic over a range of heuristic tuning parameter settings and problem instance characteristics. The heuristic is Ant Colony System (ACS) for the Travelling Salesperson Problem. 10 heurstic tuning parameters and 2 problem characteristics are considered. Response Surface Models (RSM) of the solution quality and solution time predicted ACS performance on both new instances from a publicly available problem generator and new real-world instances from the TSPLIB benchmark library. A numerical optimisation of the RSMs is used to find the tuning parameter settings that yield optimal performance in terms of solution quality and solution time. This paper is the first use of desirability functions, a well-established technique in DOE, to simultaneously optimise these conflicting goals. Finally, overlay plots are used to examine the robustness of the performance of the optimised heuristic across a range of problem instance characteristics. These plots give predictions on the range of problem instances for which a given solutionquality can be expected within a given solution time.

References

[1]
B. Adenso-D1az and M. Laguna. Fine-Tuning of Algorithms Using Fractional Experimental Designs and Local Search. Operations Research, 54(1):99--114, 2006.
[2]
D. Applegate, R. Bixby, V. Chvatal, and W. Cook. Implementing the Dantzig-Fulkerson-Johnson algorithm for large traveling salesman problems. Mathematical Programming Series B, 97(1-2):91-153, 2003.
[3]
M. Birattari. The Problem of Tuning Metaheuristics. Phd, Universit Libre de Bruxelles, 2006.
[4]
H. M. Botee and E. Bonabeau. Evolving Ant Colony Optimization. Advances in Complex Systems, 1:149--159, 1998.
[5]
P. Cheeseman, B. Kanefsky, and W. M. Taylor. Where the Really Hard Problems Are. In Proceedings of the Twelfth International Conference on Artificial Intelligence, volume 1, pages 331--337. Morgan Kaufmann Publishers, Inc., 1991.
[6]
S. Coy, B. Golden, G. Runger, and E. Wasil. Using Experimental Design to Find Effective Parameter Settings for Heuristics. Journal of Heuristics, 7(1):77--97, 2001.
[7]
G. Derringer and R. Suich. Simultaneous Optimization of Several Response Variables. Journal of Quality Technology, 12(4):214--219, 1980.
[8]
M. Dorigo and T. Stützle. Ant Colony Optimization. The MIT Press, Massachusetts, USA, 2004.
[9]
J. N. Hooker. Needed: An Empirical Science of Algorithms. Operations Research, 42(2):201--212, 1994.
[10]
D. S. Johnson and L. A. McGeoch. Experimental analysis of heuristics for the STSP. In The Traveling Salesman Problem and Its Variations, pages 369--443. 2002.
[11]
E. L. Lawler, J. K. Lenstra, A. H. G. R. Kan, and D. B. Shmoys, editors. The Traveling Salesman Problem -- A Guided Tour of Combinatorial Optimization. John Wiley and Sons.
[12]
D. C. Montgomery. Design and Analysis of Experiments. John Wiley and Sons Inc, 2005.
[13]
R. H. Myers and D. C. Montgomery. Response Surface Methodology. Process and Product Optimization Using Designed Experiments. John Wiley and Sons Inc., 1995.
[14]
G. Oehlert and P. Whitcomb. Small, Efficient, Equireplicated Resolution V Fractions of 2K designs and their Application to Central Composite Designs. In Proceedings of 46th Fall Technical Conference. American Statistical Association, Section on Physical and Engineering Sciences, 2002.
[15]
B. Ostle. Statistics in Research. Iowa State University Press, 2nd edition, 1963.
[16]
M.-W. Park and Y.-D. Kim. A systematic procedure for setting parameters in simulated annealing algorithms. Computers and Operations Research, 25(3):207--217, 1998.
[17]
R. Parsons and M. Johnson. A Case Study in Experimental Design Applied to Genetic Algorithms with Applications to DNA Sequence Assembly. American Journal of Mathematical and Management Sciences, 17(3):369--396, 1997.
[18]
W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling. Numerical Recipes in Pascal: the art of scientific computing. Cambridge University Press, 1989.
[19]
G. Reinelt. TSPLIB -- A traveling salesman problem library. ORSA Journal of Computing, 3:376--384, 1991.
[20]
E. Ridge and D. Kudenko. An Analysis of Problem Difficulty for a Class of Optimisation Heuristics. In Proceedings of the Seventh European Conference on Evolutionary Computation in Combinatorial Optimisation, volume 4446 of LNCS, pages 198--209. Springer-Verlag, 2007.
[21]
E. Ridge and D. Kudenko. Screening the Parameters Affecting Heuristic Performance. In Proceedings of the Genetic and Evolutionary Computation Conference, volume N/A, page N/A. ACM, 2007.
[22]
T. Stützle and H. H. Hoos. Max-Min Ant System. Future Generation Computer Systems, 16(8):889--914, 2000.

Cited By

View all
  • (2021)Parameter setting of meta-heuristic algorithms: a new hybrid method based on DEA and RSMEnvironmental Science and Pollution Research10.1007/s11356-021-17364-y29:15(22404-22426)Online publication date: 17-Nov-2021
  • (2018)Homogeneous genetic algorithmsInternational Journal of Computer Mathematics10.1080/0020716080196877087:3(476-490)Online publication date: 27-Dec-2018
  • (2014)Implicit memory-based technique in solving dynamic scheduling problems through Response Surface Methodology – Part IIInternational Journal of Intelligent Computing and Cybernetics10.1108/IJICC-12-2013-00547:2(143-174)Online publication date: 3-Jun-2014
  • Show More Cited By

Index Terms

  1. Analyzing heuristic performance with response surface models: prediction, optimization and robustness

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    GECCO '07: Proceedings of the 9th annual conference on Genetic and evolutionary computation
    July 2007
    2313 pages
    ISBN:9781595936974
    DOI:10.1145/1276958
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 July 2007

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. ant colony optimization
    2. design of experiments
    3. minimum run resolution V design
    4. overlay plots
    5. response surface model

    Qualifiers

    • Article

    Conference

    GECCO07
    Sponsor:

    Acceptance Rates

    GECCO '07 Paper Acceptance Rate 266 of 577 submissions, 46%;
    Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)7
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)Parameter setting of meta-heuristic algorithms: a new hybrid method based on DEA and RSMEnvironmental Science and Pollution Research10.1007/s11356-021-17364-y29:15(22404-22426)Online publication date: 17-Nov-2021
    • (2018)Homogeneous genetic algorithmsInternational Journal of Computer Mathematics10.1080/0020716080196877087:3(476-490)Online publication date: 27-Dec-2018
    • (2014)Implicit memory-based technique in solving dynamic scheduling problems through Response Surface Methodology – Part IIInternational Journal of Intelligent Computing and Cybernetics10.1108/IJICC-12-2013-00547:2(143-174)Online publication date: 3-Jun-2014
    • (2014)Implicit memory-based technique in solving dynamic scheduling problems through Response Surface Methodology – Part IInternational Journal of Intelligent Computing and Cybernetics10.1108/IJICC-12-2013-00537:2(114-142)Online publication date: 3-Jun-2014
    • (2014)Design of ExperimentsDesign of Experiments for Reinforcement Learning10.1007/978-3-319-12197-0_3(53-66)Online publication date: 23-Nov-2014
    • (2010)Sequential Parameter Optimization of an Evolution Strategy for the design of Mold Temperature Control SystemsIEEE Congress on Evolutionary Computation10.1109/CEC.2010.5586314(1-8)Online publication date: Jul-2010
    • (2008)MST Ant Colony Optimization with Lin-Kerninghan Local Search for the Traveling Salesman ProblemProceedings of the 2008 International Symposium on Computational Intelligence and Design - Volume 0110.1109/ISCID.2008.166(344-347)Online publication date: 17-Oct-2008
    • (2007)Tuning the performance of the MMAS heuristicProceedings of the 2007 international conference on Engineering stochastic local search algorithms: designing, implementing and analyzing effective heuristics10.5555/1770849.1770854(46-60)Online publication date: 6-Sep-2007
    • (2007)Screening the parameters affecting heuristic performanceProceedings of the 9th annual conference on Genetic and evolutionary computation10.1145/1276958.1276994(180-180)Online publication date: 7-Jul-2007

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media