Skip to main content
SpringerLink
Log in

Analysing and characterising optimization problems using length scale

  • Methodologies and Application
  • Published:
Soft Computing Aims and scope Submit manuscript

Abstract

Analysis of optimization problem landscapes is fundamental in the understanding and characterisation of problems and the subsequent practical performance of algorithms. In this paper, a general framework is developed for characterising black-box optimization problems based on length scale, which measures the change in objective function with respect to the distance between candidate solution pairs. Both discrete and continuous problems can be analysed using the framework, however, in this paper, we focus on continuous optimization. Length scale analysis aims to efficiently and effectively utilise the information available in black-box optimization. Analytical properties regarding length scale are discussed and illustrated using simple example problems. A rigorous sampling methodology is developed and demonstrated to improve upon current practice. The framework is applied to the black-box optimization benchmarking problem set, and shows greater ability to discriminate between the problems in comparison to seven well-known landscape analysis techniques. Dimensionality reduction and clustering techniques are applied comparatively to an ensemble of the seven techniques and the length scale information to gain insight into the relationships between problems. A fundamental summary of length scale information is an estimate of its probability density function, which is shown to capture salient structural characteristics of the benchmark problems.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Aggarwal CC, Hinneburg A, Keim DA (2001) On the surprising behavior of distance metrics in high dimensional spaces. In: Proceedings of the 8th international conference on database theory. Springer, London, pp 420–434

  • Bartz-Beielstein T, Chiarandini M, Paquete L, Preuss M (2010) Experimental methods for the analysis of optimization algorithms. Springer, New York

    Book  MATH  Google Scholar 

  • Beliakov G (2006) Interpolation of lipschitz functions. J Comput Appl Math 196(1):20–44

    Article  MathSciNet  MATH  Google Scholar 

  • Beyer KS, Goldstein J, Ramakrishnan R, Shaft U (1999) When is ”nearest neighbor” meaningful? In: Proceedings of the 7th international conference on database theory. Springer, London, pp 217–235

  • Borenstein Y, Poli R (2006) Kolmogorov complexity, optimization and hardness. In: IEEE congress on evolutionary computation (CEC 2006), pp 112–119

  • Cheeseman P, Kanefsky B, Taylor W (1991) Where the really hard problems are. In: Proceedings of 12th international joint conference on AI, Morgan Kauffman, pp 331–337

  • Collard P, Vérel S, Clergue M (2004) Local search heuristics: fitness cloud versus fitness landscape. In: The 2004 European conference on artificial intelligence, IOS Press, pp 973–974

  • Cover TM, Thomas JA (1991) Elements of information theory. Wiley, New York

    Book  MATH  Google Scholar 

  • Fletcher R (1987) Practical methods of optimization, 2nd edn. Wiley, Hoboken

    MATH  Google Scholar 

  • Forrester A, Sóbester A, Keane A (2008) Engineering design via surrogate modelling: a practical guide. Wiley, Hoboken

    Book  Google Scholar 

  • Gallagher M (2000) Multi-layer perceptron error surfaces: visualization, structure and modelling. PhD thesis, Department of Computer Science and Electrical Engineering, University of Queensland

  • Gallagher M (2001) Fitness distance correlation of neural network error surfaces: a scalable, continuous optimization problem. In: Raedt LD, Flach P (eds) European conference on machine learning, Singapore, Lecture notes in artificial intelligence, vol 2167, pp 157–166

  • Gallagher M, Downs T, Wood I (2002) Empirical evidence for ultrametric structure in multi-layer perceptron error surfaces. Neural Process Lett 16(2):177–186

    Article  MATH  Google Scholar 

  • Gent I, Walsh T (1996) The TSP phase transition. Artif Intell 88(1–2):349–358

    Article  MathSciNet  MATH  Google Scholar 

  • Grinstead CM, Snell JL (2012) Introduction to probability. American Mathematical Society, Providence

    MATH  Google Scholar 

  • Guyon I, Elisseeff A (2003) An introduction to variable and feature selection. J Mach Learn Res 3:1157–1182

    MATH  Google Scholar 

  • Hansen N (2000) Invariance, self-adaptation and correlated mutations in evolution strategies. In: Schoenauer et al M (ed) Parallel problem solving from nature—PPSN VI. Lecture notes in computer science, vol 1917, Springer, pp 355–364

  • Hansen N, Finck S, Ros R, Auger A (2010) Real-parameter black-box optimization benchmarking 2010: noiseless functions definitions. Technical Report, RR-6829, INRIA

  • Horst R, Tuy H (1996) Global optimization: deterministic approaches. Springer, New Yotk

    Book  MATH  Google Scholar 

  • Hutter F, Hamadi Y, Hoos H, Leyton-Brown K (2006) Performance prediction and automated tuning of randomized and parametric algorithms. In: Benhamou F (ed) Principles and practice of constraint programming. Lecture notes in computer science, vol 4204, Springer, pp 213–228

  • Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9(1):3–12

    Article  Google Scholar 

  • Jones T, Forrest S (1995) Fitness distance correlation as a measure of problem difficulty for genetic algorithms. In: Proceedings of the 6th international conference on genetic algorithms. Morgan Kaufmann, San Francisco, CA, pp 184–192

  • Kullback S, Leibler RA (1951) On information and sufficiency. Ann Math Stat 22(1):79–86

    Article  MathSciNet  MATH  Google Scholar 

  • Lee JA, Verleysen M (2007) Nonlinear dimensionality reduction. Springer, New York

    Book  MATH  Google Scholar 

  • Lunacek M, Whitley D (2006) The dispersion metric and the CMA evolution strategy. In: Proceedings of the 8th annual conference on genetic and evolutionary computation. ACM, New York, pp 477–484

  • Macready W, Wolpert D (1996) What makes an optimization problem hard. Complexity 5:40–46

    Article  MathSciNet  Google Scholar 

  • Malan K, Engelbrecht A (2009) Quantifying ruggedness of continuous landscapes using entropy. In: IEEE congress on evolutionary computation, pp 1440–1447

  • Mersmann O, Bischl B, Trautmann H, Preuss M, Weihs C, Rudolph G (2011) Exploratory landscape analysis. In: Proceedings of the 13th annual conference on genetic and evolutionary computation. ACM, New York, pp 829–836

  • Morgan R, Gallagher M (2012) Length scale for characterising continuous optimization problems. In: Coello et al CAC (ed) Parallel problem solving from nature—PPSN XII. Lecture notes in computer science, vol 7491, Springer, pp 407–416

  • Morgan R, Gallagher M (2014) Sampling techniques and distance metrics in high dimensional continuous landscape analysis: limitations and improvements. IEEE Trans Evol Comput 18(3):456–461

    Article  Google Scholar 

  • Muñoz MA, Kirley M, Halgamuge S (2012a) A meta-learning prediction model of algorithm performance for continuous optimization problems. In: Coello et al CAC (ed) Parallel problem solving from nature—PPSN XII. Lecture notes in computer science, vol 7491, Springer, pp 226–235

  • Muñoz MA, Kirley M, Halgamuge SK (2012b) Landscape characterization of numerical optimization problems using biased scattered data. In: IEEE congress on evolutionary computation, pp 1180–1187

  • Müller C, Baumgartner B, Sbalzarini I (2009) Particle swarm CMA evolution strategy for the optimization of multi-funnel landscapes. In: IEEE congress on evolutionary computation, pp 2685–2692

  • Müller CL, Sbalzarini IF (2011) Global characterization of the CEC 2005 fitness landscapes using fitness-distance analysis. In: Proceedings of the 2011 international conference on applications of evolutionary computation. vol Part I. Springer, Berlin, Heidelberg, pp 294–303

  • Overton M (2001) Numerical computing with IEEE floating point arithmetic. Cambridge University Press, Cambridge

    Book  MATH  Google Scholar 

  • Pitzer E, Affenzeller M (2012) A comprehensive survey on fitness landscape analysis. In: Fodor J, Klempous R, Suárez Araujo C (eds) Recent advances in intelligent engineering systems, studies in computational intelligence. Springer, New York, pp 161–191

    Google Scholar 

  • Pitzer E, Affenzeller M, Beham A, Wagner S (2012) Comprehensive and automatic fitness landscape analysis using heuristiclab. In: Moreno-Díaz R, Pichler F, Quesada-Arencibia A (eds) Computer aided systems theory–EUROCAST 2011. Lecture notes in computer science, vol 6927, Springer, pp 424–431

  • Reidys C, Stadler P (2002) Combinatorial landscapes. SIAM Rev 44(1):3–54

    Article  MathSciNet  MATH  Google Scholar 

  • Rice JR (1976) The algorithm selection problem. Adv Comput 15:65–118

    Article  Google Scholar 

  • Ridge E, Kudenko D (2007) An analysis of problem difficulty for a class of optimisation heuristics. In: Proceedings of the 7th european conference on evolutionary computation in combinatorial optimization, Springer, pp 198–209

  • Rosé H, Ebeling W, Asselmeyer T (1996) The density of states—a measure of the difficulty of optimisation problems. In: Voigt et al HM (ed) Parallel problem solving from nature PPSN IV. Lecture notes in computer science, vol 1141, Springer, pp 208–217

  • Rosen K (1999) Handbook of discrete and combinatorial mathematics, 2nd edn., Discrete mathematics and its applicationsTaylor & Francis, Routledge

    Book  Google Scholar 

  • Sergeyev YD, Kvasov DE (2010) Lipschitz global optimization. Wiley Encycl Oper Res Manag Sci 4:2812–2828

    Google Scholar 

  • Sheather S, Jones M (1991) A reliable data-based bandwidth selection method for kernel density estimation. J R Stat Soc Ser B (Methodological) 53:683–690

    MathSciNet  MATH  Google Scholar 

  • Shlesinger MF, West BJ, Klafter J (1987) Lévy dynamics of enhanced diffusion: application to turbulence. Phys Rev Lett 58:1100–1103

    Article  MathSciNet  Google Scholar 

  • Smith T, Husbands P, O’Shea M (2002) Fitness landscapes and evolvability. Evol Comput 10(1):1–34

    Article  Google Scholar 

  • Smith-Miles K (2008) Cross-disciplinary perspectives on meta-learning for algorithm selection. ACM Comput Surv 41(1):1–25

    Article  Google Scholar 

  • Smith-Miles K, Lopes L (2011) Measuring instance difficulty for combinatorial optimization problems. Comput Oper Res 39(5):875–889

    Article  MathSciNet  MATH  Google Scholar 

  • Smith-Miles K, Tan TT (2012) Measuring algorithm footprints in instance space. In: IEEE congress on evolutionary computation (CEC), pp 1–8

  • Solla SA, Sorkin GB, White SR (1986) Configuration space analysis for optimization problems. In: et al EB (ed) Disordered systems and biological organization, NATO ASI Series, vol F20, Springer, Berlin, New York, pp 283–293

  • Stadler PF (1996) Landscapes and their correlation functions. J Math Chem 20:1–45

    Article  MathSciNet  MATH  Google Scholar 

  • Stadler PF (2002) Fitness landscapes. In: Lässig M, Valleriani A (eds) Biological evolution and statistical physics. Lecture notes in physics, vol 585, Springer, pp 183–204

  • Stadler PF, Schnabl W (1992) The landscape of the traveling salesman problem. Phys Lett A 161(4):337–344

    Article  MathSciNet  MATH  Google Scholar 

  • Steer K, Wirth A, Halgamuge S (2008) Information theoretic classification of problems for metaheuristics. In: Li et al X (ed) Simulated evolution and learning, Lecture notes in computer science, vol 5361, Springer, pp 319–328

  • Strongin R (1973) On the convergence of an algorithm for finding a global extremum. Eng Cybern 11:549–555

    MathSciNet  Google Scholar 

  • Talbi E (2009) Metaheuristics: from design to implementation., Wiley series on parallel and distributed computingWiley, Hoboken

    Book  MATH  Google Scholar 

  • Van der Maaten L, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(11):2579–2605

    MATH  Google Scholar 

  • van Hemert J (2005) Property analysis of symmetric travelling salesman problem instances acquired through evolution. Evol Comput Comb Optim 3448:122–131

    MATH  Google Scholar 

  • Vassilev VK, Fogarty TC, Miller JF (2000) Information characteristics and the structure of landscapes. Evol Comput 8:31–60

    Article  Google Scholar 

  • Wang Y, Li B (2008) Understand behavior and performance of real coded optimization algorithms via nk-linkage model. In: IEEE world congress on computational intelligence, pp 801–808

  • Weinberger E (1990) Correlated and uncorrelated fitness landscapes and how to tell the difference. Biol Cybern 63:325–336

    Article  MATH  Google Scholar 

  • Whitley D, Watson JP (2005) Complexity theory and the no free lunch theorem. In: Search methodologies, Springer, pp 317–339

  • Whitley D, Lunacek M, Sokolov A (2006) Comparing the niches of CMA-ES, CHC and pattern search using diverse benchmarks. In: Runarsson et al TP (ed) Parallel problem solving from nature—PPSN IX. Lecture notes in computer science, vol 4193, Springer, pp 988–997

  • Wood GR, Zhang BP (1996) Estimation of the Lipschitz constant of a function. J Glob Optim 8:91–103

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang W (2004) Phase transitions and backbones of the asymmetric traveling salesman problem. J Artif Intell Res 21(1):471–497

    MathSciNet  MATH  Google Scholar 

  • Zhang W, Korf RE (1996) A study of complexity transitions on the asymmetric traveling salesman problem. Artif Intell 81(1–2):223–239

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Information Technology and Electrical Engineering, University of Queensland, Brisbane, 4072, Australia

    Rachael Morgan & Marcus Gallagher

Authors
  1. Rachael Morgan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcus Gallagher
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rachael Morgan.

Ethics declarations

Conflicts of interest

None.

Additional information

Communicated by V. Loia.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Morgan, R., Gallagher, M. Analysing and characterising optimization problems using length scale. Soft Comput 21, 1735–1752 (2017). https://doi.org/10.1007/s00500-015-1878-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00500-015-1878-z

Keywords

Search

Navigation