Skip to main content
SpringerLink
Log in
Book cover

Experimental Methods for the Analysis of Optimization Algorithms pp 51–72Cite as

Design and Analysis of Computational Experiments: Overview

  • Chapter
  • First Online:

Abstract

This chapter presents an overview of the design and analysis of computational experiments with optimization algorithms. It covers classic designs and their corresponding (meta)models; namely, Resolution-III designs including fractional factorial two-level designs for first-order polynomial models, Resolution-IV and Resolution-V designs for two-factor interactions, and designs including central composite designs for second-degree polynomials. It also reviews factor screening in experiments with very many factors, focusing on the sequential bifurcation method. Furthermore, it reviews Kriging models and their designs. Finally, it discusses experiments aimed at the optimization of the parameters of a given optimization algorithm, allowing multiple random experimental outputs. This optimization may use either generalized response surface methodology or Kriging combined with mathematical programming; the discussion also covers Taguchian robust optimization.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   159.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • Adenso-Diaz B, Laguna M (2006) Fine-tuning of algorithms using fractional experimental designs and local search. Operations Research 54(1):99–114

    Article  MATH  Google Scholar 

  • Angün E, Kleijnen J (2009) An asymptotic test of optimality conditions in multiresponse simulation-based optimization, working paper

    Google Scholar 

  • Angün E, Kleijnen J, den Hertog D, Gürkan G (2009) Response surface methodology with stochastic constrains for expensive simulation. Journal of the Operational Research Society 60:735–746

    Article  MATH  Google Scholar 

  • Ankenman B, Nelson B, Staum J (2009) Stochastic kriging for simulation metamodeling. Operations Research (accepted)

    Google Scholar 

  • Bartz-Beielstein T (2006) Experimental research in evolutionary computation—The new experimentalism. Natural Computing Series, Springer

    Google Scholar 

  • Bartz-Beielstein T, Preuss M (2010) The future of experimental research. In: Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (eds) Empirical methods for the analysis of optimization algorithms, Springer, pp 17–46

    Google Scholar 

  • Ben-Tal A, Nemirovski A (2002) Robust optimization: methodology and applications. Mathematical Programming 92(3):353–380

    Article  MathSciNet  Google Scholar 

  • Bettonvil B, Kleijnen J (1996) Searching for important factors in simulation models with many factors: sequential bifurcation. European Journal of Operational Research 96:180–194

    Article  Google Scholar 

  • Bettonvil B, del Castillo E, Kleijnen J (2009) Statistical testing of optimality conditions in multiresponse simulation-based optimization. European Journal of Operational Research 199(2):448–458

    Article  MATH  MathSciNet  Google Scholar 

  • Box G, Wilson K (1951) On the experimental attainment of optimum conditions. Journal Royal Statistical Society, Series B 13(1):1–38

    MathSciNet  Google Scholar 

  • Cressie N (1993) Statistics for spatial data: revised edition. Wiley, New York

    Google Scholar 

  • Dellino G, Kleijnen J, Meloni C (2009) Robust optimization in simulation: Taguchi and response surface methodology. In: Rossini M, Hill R, Johansson B, Dunkin A, Ingalls R (eds) Proceedings of the 2009 Winter Simulation Conference, (accepted)

    Google Scholar 

  • Den Hertog D, Kleijnen J, Siem A (2006) The correct Kriging variance estimated by bootstrapping. Journal of the Operational Research Society 57(4):400–409

    Article  MATH  Google Scholar 

  • Efron B, Tibshirani R (1993) An introduction to the bootstrap. Chapman & Hall, NY

    MATH  Google Scholar 

  • Fu M (2008) What you should know about simulation and derivatives. Naval Research Logistics 55:723–736

    Article  MATH  MathSciNet  Google Scholar 

  • Kleijnen J (2008) Design and analysis of simulation experiments. Springer

    MATH  Google Scholar 

  • Kleijnen J, Sargent R (2000) A methodology for the fitting and validation of metamodels in simulation. European Journal of Operational Research 120(1):14–29

    Article  MATH  Google Scholar 

  • Kleijnen J, Van Beers W (2009) Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations, working paper

    Google Scholar 

  • Kleijnen J,Wan J (2007) Optimization of simulated systems: OptQuest and alternatives. Simulation Modelling Practice and Theory 15:354–362

    Article  Google Scholar 

  • Kleijnen J, Sanchez S, Lucas T, Cioppa T (2005) State-of-the-art review: a user’s guide to the brave new world of designing simulation experiments. INFORMS Journal on Computing 17(3):263–289

    Article  Google Scholar 

  • Kleijnen J, Bettonvil B, Persson F (2006a) Screening for the important factors in large discrete-event simulation: sequential bifurcation and its applications. In: Dean A, Lewis S (eds) Screening: Methods for experimentation in industry, drug discovery, and genetics, Springer, pp 287–307

    Google Scholar 

  • Kleijnen J, den Hertog D, Angün E (2006b) Response surface methodology’s steepest ascent and step size revisited: correction. European Journal of Operational Research 170:664–666

    Article  MATH  Google Scholar 

  • Kleijnen J, Van Beers W, Van Nieuwenhuyse I (2010) Constrained optimization in simulation: a novel approach. European Journal of Operational Research 202:164–174

    Article  MATH  Google Scholar 

  • Law A (2007) Simulation modeling and analysis, 4th edn. McGraw-Hill, Boston

    Google Scholar 

  • Lophaven S, Nielsen H, Søndergaard J (2002) DACE—A Matlab Kriging Toolbox. Tech. Rep. IMM-REP-2002-12, Informatics and Mathematical Modelling, Technical University of Denmark, Copenhagen, Denmark

    Google Scholar 

  • Montgomery D (2009) Design and analysis of experiments, 7th edn. Wiley, Hoboken, New Jersey

    Google Scholar 

  • Myers R, Montgomery D (1995) Response surface methodology: process and product optimization using designed experiments. Wiley, NY

    MATH  Google Scholar 

  • Park S, Fowler J, Mackulak G, Keats J, Carlyle W (2002) D-optimal sequential experiments for generating a simulation-based cycle time-throughput curve. Operations Research 50(6):981–990

    Article  Google Scholar 

  • Rajagopalan HK, Vergara FE, Saydam C, Xiao J (2007) Developing effective meta-heuristics for a probabilistic location model via experimental design. European Journal of Operational Research 177(1):83–101, URL http://dx.doi. org/10.1016/j.ejor.2005.11.007

    Article  MATH  Google Scholar 

  • Ridge E, Kudenko D (2007) Screening the parameters affecting heuristic performance. In: Lipson H (ed) GECCO, ACM, p 180, URL http://doi.acm. org/10.1145/1276958.1276994

  • Ridge E, Kudenko D (2010) Sequential experiment designs for screening and tuning parameters of stochastic heuristics. In: Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (eds) Empirical Methods for the Analysis of Optimization Algorithms, Springer, pp 265–287

    Google Scholar 

  • Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Statistical Science 4(4):409–435

    Article  MATH  MathSciNet  Google Scholar 

  • Santner TJ, Williams BJ, Notz WI (2003) The design and analysis of computer experiments. Springer

    MATH  Google Scholar 

  • Taguchi G (1987) System of experimental designs. Krauss International, NY

    Google Scholar 

  • Van Beers W, Kleijnen J (2003) Kriging for interpolation in random simulation. Journal of the Operational Research Society (54):255–262

    Article  MATH  Google Scholar 

  • Van Beers W, Kleijnen J (2008) Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping. European Journal of Operational Research 186(3):1099–1113

    Article  MATH  Google Scholar 

  • Wu C, Hamada M (2000) Experiments; planning, analysis, and parameter design optimization. Wiley, NY

    MATH  Google Scholar 

  • Xu J, Yang F, Wan H (2007) Controlled sequential bifurcation for software reliability study. In: Henderson S, Biller B, Hsieh MH, Shortle J, Tew J, Barton R (eds) Proceedings of the 2007 Winter Simulation Conference, pp 281–288

    Google Scholar 

  • Yin J, Ng S, Ng K (2008) Kriging model with modified nugget effect. In: Proceedings of the 2008 IEEE International Conference on Industrial Engineering and Engineering Management, pp 1714–1718

    Google Scholar 

  • Yu HF (2007) Designing a screening experiment with a reciprocal Weibull degradation rate. Computers & Industrial Engineering 52(2):175–191

    Article  Google Scholar 

Download references

Acknowledgements

Comments by two anonymous referees led to a much improved version of the present chapter.

Author information

Authors and Affiliations

  1. Department of Information Management, CentER Tilburg University, 90153, 5000, LE, Tilburg, Netherlands

    Jack P.C. Kleijnen

Authors
  1. Jack P.C. Kleijnen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Institute of Computer Science, Faculty of Computer Science, Cologne University of Applied Sciences, Steinmüllerallee 1, Gummersbach, 51643, Germany

    Thomas Bartz-Beielstein

  2. , Department of Mathematics, University of Southern Denmark, Campusvej 55, Odense, 5230, Denmark

    Marco Chiarandini

  3. CISUC, Department of Informatics Engineering, University of Coimbra, Coimbra, 3030-290, Portugal

    Luís Paquete

  4. Algorithm Engineering, TU Dortmund, Otto-Hahn-Str. 14, Dortmund, 44227, Germany

    Mike Preuss

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Kleijnen, J.P. (2010). Design and Analysis of Computational Experiments: Overview. In: Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M. (eds) Experimental Methods for the Analysis of Optimization Algorithms. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02538-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-02538-9_3

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-02537-2

  • Online ISBN: 978-3-642-02538-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation