Skip to main content
Springer Nature Link
Log in

Performance assessment of the metaheuristic optimization algorithms: an exhaustive review

  • Published:
Artificial Intelligence Review Aims and scope Submit manuscript

Abstract

The simulation-driven metaheuristic algorithms have been successful in solving numerous problems compared to their deterministic counterparts. Despite this advantage, the stochastic nature of such algorithms resulted in a spectrum of solutions by a certain number of trials that may lead to the uncertainty of quality solutions. Therefore, it is of utmost importance to use a correct tool for measuring the performance of the diverse set of metaheuristic algorithms to derive an appropriate judgment on the superiority of the algorithms and also to validate the claims raised by researchers for their specific objectives. The performance of a randomized metaheuristic algorithm can be divided into efficiency and effectiveness measures. The efficiency relates to the algorithm’s speed of finding accurate solutions, convergence, and computation. On the other hand, effectiveness relates to the algorithm’s capability of finding quality solutions. Both scopes are crucial for continuous and discrete problems either in single- or multi-objectives. Each problem type has different formulation and methods of measurement within the scope of efficiency and effectiveness performance. One of the most decisive verdicts for the effectiveness measure is the statistical analysis that depends on the data distribution and appropriate tool for correct judgments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

(Full forms of the algorithms used should be mentioned in the legend)

Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Abouhawwash M, Jameel MA (2019) In: Proceeding on 10th international conference EMO 2019. Springer, East Lansing, Mi, USA, pp 27–38

  • Adekanmbi O, Green P (2015) Conceptual comparison of population-based metaheuristics for engineering problems. Sci World J 936106:1–9

    Google Scholar 

  • Agrawal AP, Kaur A (2016) An empirical evaluation of three popular meta-heuristics for solving travelling salesman problem. In: 6th International conference—cloud system and big data engineering (Confluence). Noida, India, pp 16–21

  • Akhand MAH, Ayon SI, Shahriyar SA, Siddique N, Adeli H (2020) Discrete spider monkey optimization for travelling salesman problem. Appl Soft Comput J 86:105887

    Google Scholar 

  • Ali IM, Essam D, Kasmarik K (2020) A novel design of differential evolution for solving discrete travelling salesman problems. Swarm Evol Comput 52:100607

    Google Scholar 

  • Ambati BK, Ambati J, Mokhtar MM (1991) Heuristic combinatorial optimization by simulated Darwinian evolution: a polynomial-time algorithm for the traveling salesman problem. Biol Cybern 65(1):31–35

    MATH  Google Scholar 

  • Araya-Sassi C, Miranda PA, Paredes-Belmar G (2018) Lagrangian relaxation for an inventory location problem with periodic inventory control and stochastic capacity constraints. Math Probl Eng 8237925:1–27

    MathSciNet  MATH  Google Scholar 

  • Asafuddoula M, Ray T, Singh H (2015) Characterizing Pareto front approximations in many-objective optimization. In: Proceedings of the 2015 annual conference on genetic and evolutionary computation. ACM, pp 607–614

  • Askarzadeh A (2016) A novel metaheuristic method for solving constrained engineering optimization problems: crow search algorithm. Comput Struct 169:1–12

    Google Scholar 

  • Aspnes J (2017) Notes on computational complexity theory CPSC 468/568 ch. 9. pp 58–59

  • Attiya I, Elaziz MA, Xiong S (2020) Job scheduling in cloud computing using a modified harris hawks optimization and simulated annealing algorithm. Comput Intell Neurosci 3504642:1–17

    Google Scholar 

  • Audet C, Bigeon J, Cartier D, Le Digabel S, Salomon L (2018) Performance indicators in multiobjective optimization. Optimization Online. Retrieved from http://www.optimization-online.org/DB_HTML/2018/10/6887.html

  • Augusto OB, Rabeau S, Dépince Ph, Bennis F (2006) Multi-objective genetic algorithms: a way to improve the convergence rate. Eng Appl Artif Intell 19:501–510

    Google Scholar 

  • Awad NH, Ali MZ, Suganthan PN, Liang JJ, Qu BY (2016) Problem definitions and evaluation criteria for the CEC 2017 special session and competition on single objective real-parameter numerical optimization. Technical report

  • Aziz NAA, Mubin M, Ibrahim Z, Nawawi SW (2014) Performance and diversity of gravitational search algorithm. Adv Appl Converg Lett 3(1):232–235

    Google Scholar 

  • Back T (1996) Evolutionary algorithms in theory and practice: evolution strategies, evolutionary programming, genetic algorithms. Oxford University Press, Oxford

    MATH  Google Scholar 

  • Bader J, Zitzler E (2011) HypE: an algorithm for fast hypervolume-based many-objective optimization. Evol Comput 19(1):45–76

    Google Scholar 

  • Bahameish HA (2014) Finding a cost effective LNG annual delivery program (ADP) using genetic algorithms. Master’s thesis, Qatar University

  • Balogh J, Békési J, Dósa G, Sgall J, Stee R (2015) The optimal absolute ratio for online bin packing. In: SODA’15 Proceedings of the twenty-sixth annual ACM-SIAM symposium on Discrete algorithms. Philadelphia, USA, pp 1425–1438

  • Barr RS, Golden BL, Kelly JP, Resende MG, Steward WR (1995) Designing and reporting on computational experiments with heuristic methods. J Heuristics 1:9–32

    MATH  Google Scholar 

  • Bartz-Beielstein T (2005) New experimentalism applied to evolutionary computation. Ph.D. dissertation, Universitaet Dortmund, Fachbereich Informatik

  • Bassi M, Cursi ES, Pagnacco E, Ellaia R (2018) Statistics of the Pareto front in multi-objective optimization under uncertainties. Latin Am J Solids Struct 15(11):1–18

    Google Scholar 

  • Beiranvand V, Hare W, Lucet Y (2017) Best practices for comparing optimization algorithms. Optim Eng 18(4):815–848

    MathSciNet  MATH  Google Scholar 

  • Benavoli A, Mangili F, Corani G, Zaffalon M, Ruggeri F (2014) A Bayesian Wilcoxon signed-rank test based on the Dirichlet process. In: Proceedings of the 30th international conference on machine learning (ICML 2014). pp 1–9

  • Benson HY, Shanno DF, Vanberei RJ (2000) Interior-point methods for nonconvex nonlinear programming: jamming and comparative numerical testing, technical report ORFE-00-02. Princeton University, Princeton

    Google Scholar 

  • Bezerra LC, Ibánez ML, Stuetzel T (2017) An empirical assesment of the properties of inverted generational distance on multi- and many-objective optimization. In: EMO international conferecne on evolutionary multicriterion optimization. Springer, Muenster, Germany, pp 31–45

  • Bhattacharjee KS, Singh HK, Ray T (2016) A study on performance metrics to identify of interest from a trade-off set. In: Proceedings of the second Australasian conference on artificial life and computational intelligence, vol 9592. Springer, pp 66–77

  • Billups SC, Dirkse SP, Ferris MC (1997) A comparison of large scale mixed complementary problem solvers. Comput Optim Appl 7(1):3–25

    MathSciNet  MATH  Google Scholar 

  • Blanca MJ, Alarcón R, Arnau J, Bono R, Bendayan R (2017) Non-normal data: Is ANOVA still a valid option? Psicothema 29(4):552–557

    Google Scholar 

  • Bogoya JM, Vargas A, Cuate O, Schütze O (2018) A (p, q)-averaged Hausdorf distance for arbitrary measurable sets. Math Comput Appl 23:51

    Google Scholar 

  • Bolufé AR, González SF, Chen S (2015) A minimum population search hybrid for large scale global optimization. In: IEEE congress on evolutionary computation (CEC). Sendai, pp 1958–1965

  • Bongartz I, Conn AR, Gould NIM, Saunders MA, Toint PL (1997) A numerical comparison between the LANCELOT and MINOS packages for large-scale numerical optimization. Technical report 97/13, Namur University

  • Bonyandi MR, Michalewicz Z, Barone L (2013) The travelling thief problem: the first step in the transition from theoretical problems to realistic problems. In: IEEE congress on evolutionary computation. Cancun, pp 1037–1044

  • Bonyandi MR, Michalewicz Z, Przybylek MR, Wierbicki A (2014) Socially inspired algorithms for the travelling thief problem. In: GECCO’14 proceedings of the 2014 annual conference on genetic and evolutionary computation. pp 421–428

  • Boryczka U, Szwarc K (2019) The harmony search algorithm with additional improvement of harmony memory for asymmetric travelling salesman problem. Expert Syst Appl 122:43–53

    Google Scholar 

  • Bozkurt B, Fowler JW, Gel ES, Kim B, Köksalan M, Wallenius J (2010) Quantitative comparison of approximate solution sets for multicriteria optimization problems with weighted Tchebycheff preference function. Oper Res 58(3):650–659

    MathSciNet  MATH  Google Scholar 

  • Branke J (2002) Evolutionary optimization in dynamic environments. Springer, New York, pp 13–29

    MATH  Google Scholar 

  • Bringmann K, Friedrich T (2010) Approximating the volume of unions and intersections of high-dimensional geometric objects. Comput Geom 43(6–7):601–610

    MathSciNet  MATH  Google Scholar 

  • Brockhoff D, Auger A, Hansen N, Tušar T (2017) Quantitative performance assesment of multiobjective optimizers: the average runtime attainment function. In: Trautman H et al (eds) Evolutionary multi-criterion optimization. EMO 2017 Lecture notes in computer science, 10173. Springer, Cham

    Google Scholar 

  • Brownlee J (2011) Clever algorithms: nature-inspired programming recipes. Lulu, pp 404–405

  • Cai X, Sun H, Fan Z (2018) A diversity indicator based on reference vectors for many-objective optimization. Inf Sci 430:467–486

    MathSciNet  Google Scholar 

  • Calvo B, Shir OM, Ceberio J, Doerr C, Wang H, Bäck T, Lozano JA (2019) Bayesian performance analysis for black-box optimization benchmarking. In: proceedings of the genetic and evolutionary computation conference companion, GECCO 19, ACM, New York, NY, USA pp 1789–1797

  • Campuzano G, Obreque C, Aguayo MM (2020) Accelerating the Miller–Tucker–Zemlin model for the asymmetric travelling salesman problem. Expert Syst Appl 148:113229

    Google Scholar 

  • Cardona JGF, Coello CAC (2020) Indicator-based multi-objective evolutionary algorithms: a comprehensive survey. ACM Comput Surv 53(2):1–35

    Google Scholar 

  • Carlyle WM, Fowler JW, Gel ES, Kim B (2003) Quantitative comparison of approximation solution sets for bi-criteria optimization problems. Decis Sci 34(1):63–82

    Google Scholar 

  • Carrasco J, García S, del Mar Rueda M, Herrera F (2017) rNPBST: an R package covering non-parametric and Bayesian statistical tests. In: International conferecne hybrid artificial intelligent systems. pp 281–292

  • Carrasco J, García S, Rueda M, Das S, Herrera F (2020) Recent trends in the use of statistical tests for comparing swarm and evolutionary computing algorithms: practical guidelines and a critical review. Swarm Evol Comput 54:100665

    Google Scholar 

  • Chang KH (2015) Design theory and methods using CAD/CAE. The computer aided engineering design series, Ch. 5—multiobjective optimization and advanced topics. pp 325–406

  • Chen WN, Zhang J, Chung HSH, Zhong WL, Wu WG, Shi YH (2010) A novel set-based particle swarm optimization method for discrete optimization problems. IEEE Trans Evol Comput 14(2):278–300

    Google Scholar 

  • Cheng S, Shi Y, Qin Q, Zhang Q, Bai R (2014) Population diversity maintenance in brain storm optimization algorithm. J Artif Intell Soft Comput Res 4(2):83–97

    Google Scholar 

  • Chi Y, Sun F, Jiang L, Yu C (2012) An efficient population diversity measure for improved particle swarm optimization algorithm. In: 6th IEEE international conference intelligent systems. Sofia, Bulgaria, pp 361–367

  • Chiarandini M, Paquete L, Preuss M, Ridge E (2007) Experiments on metaheuristics: methodological overview and open issues. Technical report DMF-2007-03-003, the Danish mathematical society, Denmark

  • Chicco G, Mazza A (2019) Heuristic optimization of electrical energy systems: refined metrics to compare the solutions. Sustain Energy Grids Netw 17:10097

    Google Scholar 

  • Chih M, Lin CJ, Chern MS, Ou TY (2014) Particle swarm optimization with time-varying acceleration coefficients for the multimodal knapsack problem. Appl Math Mod. 38:1338–1350

    MATH  Google Scholar 

  • Chopard B, Tomassini M (2018) Performance and limitations of metaheuristics. In: An introduction to metaheuristics for optimization. Natural computing series. Springer, Cham. https://doi.org/10.1007/978-3-319-93073-2_11

  • Coello CA, Sierra MR (2004) A study of the parallelization of a coevolutionary multi-objective evolutionary algorithm. In: MICAI 2004: advances in artificial intelligence (LNCS 2972). Springer, Mexico City, pp 688–697

  • Coello CA, Lamont GB, Veldhuizen DAV (2008) Evolutionary algorithms for solving multi-objective problems, 2nd edn. Springer, New York

    MATH  Google Scholar 

  • Coello CA, Dhaenens C, Jourdan L (2010) Multi-objective combinatorial optimization: problematic and context. In: Advances in multi-objective nature inspired computing, SCI272. Springer, pp 1–21

  • Collette Y, Siarry P (2005) Three new metrics to measure the convergence of metaheuristics towards the Pareto frontier and the aesthetic of a set of solutions in biobjective optimization. Comput Oper Res 32(4):773–792

    MATH  Google Scholar 

  • Conn AR, Gould NIM, Toint PL (1996) Numerical experiments with the LANCELOT package (release a) for large-scale nonlinear optimization. Math Program 73:73–110

    MathSciNet  MATH  Google Scholar 

  • Conover J (1980) Practical nonparametric statistics. Wiley, Hoboken

    Google Scholar 

  • Corani G, Benavoli A (2015) A Bayesian approach for comparing cross-validated algorithms on multiple data sets. Mach Learn 100(2–3):285–304

    MathSciNet  MATH  Google Scholar 

  • Cormen TH, Leiserson CE, Rivest RL, Stein C (2001) Introduction to algorithms. MIT Press, Cambridge, p 47

    MATH  Google Scholar 

  • Crainic TG, Mancini S, Perboli G, Tadei R (2011) Multi-start heuristics for the two-echelon vehicle routing problem. In: Merz P, Hao JK (eds) Evolutionary computation in combinatorial optimization. EvoCOP 2011. Lecture notes in computer science, vol 6622. Springer, Berlin, Heidelberg

  • Cubukcuoglu C, Tasgetiren MF, Sariyildiz IS, Gao L, Kucukvar M (2019) A memetic algorithm for the bi-objective quadratic assignment problem. Procedia Manuf 39:1215–1222

    Google Scholar 

  • Czyzak P, Jaskiewicz A (1998) Pareto simulated annealing—a metaheuristic technique for multiple-objective combinatorial optimization. J Multi-Criteria Decis Anal 7:34–47

    MATH  Google Scholar 

  • Dahmani I, Hifi M, Saadi T, Yousef L (2020) A swarm optimization-based search algorithm for the quadratic knapsack problem with conflict graphs. Expert Syst Appl 148:113224

    Google Scholar 

  • Das S (2018) Evaluating the swarm intelligence algorithms for continuous optimization—a (non-parametric) statistical perspective. In: ICSI, Shanghai, China

  • Das S, Abraham A, Chakraborty UK, Konar A (2009) Differential evolution using a neighborhood-based mutation operator. IEEE Trans Evol Comput 13(3):526–553

    Google Scholar 

  • Dawoud SD, Peplow R (2010) Digital system design-use of microcontroller. River Publishers series in signal, image and speech processing. River Publishers, Gistrup

    Google Scholar 

  • De Sousa Santos L, Secchi AR, Prata DM, Biscaica EC Jr (2019) An adaptive sequential wavelet-based algorithm developed for dynamic optimization problems. Comput Chem Eng 121:465–482

    Google Scholar 

  • Deb K, Gupta S (2011) Understanding knee points in bicriteria problems and their implications as preferred solutions principles. Eng Optim 43(11):1175–1204

    MathSciNet  Google Scholar 

  • Deb K, Jain S (2002) Running performance metrics for evolutionary multi-objective optimizations. In: Proceedings of the fourth Asia-pacific conference on simulated evolution and learning (SEAL’02). Singapore, pp 13–20

  • Deb K, Pratap A, Agrawal S, Meyarivan T (2002) A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans Evol Comput 6(2):182–197

    Google Scholar 

  • Deepa N, Vincent DR, Kumar SM, Srinivasan K, Chang CY, Bashir AK (2019) An efficient ensemble VTOPES multi-criteria decision-making model for sustainable sugarcane farms. Sustainability 11:4288

    Google Scholar 

  • Degertekin SO, Lamberti L, Hayalioglu MS (2016) Heat transfer search algorithm for sizing optimization of truss structures. Latin Am J Solids Struct 14(3):373–397

    Google Scholar 

  • Derrac J, García S, Hui S, Suganthan PN, Herrera F (2014) Analysing convergence performance of evolutionary algorithms: a statistical approach. Inf Sci 289:41–58

    Google Scholar 

  • Dhivyaprabha TT, Subashini P, Krishnaveni M (2018) Synergistic fibroblast optimization: a novel nature-inspired computing algorithm. Front Inform Technol Electron Eng 19(7):825–833

    Google Scholar 

  • Dilettoso E, Rizzo SA, Salerno N (2017) A weakly pareto compliant quality indicator. Math Comput Appl 22(1):25

    MathSciNet  Google Scholar 

  • Dixon WJ, Mood AM (1946) The statistical sign test. J Am Stat Assoc 41(236):557–566

    Google Scholar 

  • Dolan E, Moré JJ (2002) Benchmarking optimization software with performance profiles. Math Program 91:201–213

    MathSciNet  MATH  Google Scholar 

  • Durillo JJ, Nebro AJ, Luna F, Coello CSC, Alba E (2010) Convergence speed in multi-objective metaheuristics: efficiency criteria and empirical study. Int J Numer Methods Eng 84:1344–1375

    MATH  Google Scholar 

  • Eftimov T, Kovec D (2019) Performance measures fusion for experimental comparison of methods for multi-label classification. In: AAAI 2019 Spring symposium on combining machine learning with knowledge engineering. Palo Alto, California, USA

  • Eftimov T, Petelin G, Korošec P (2020) DSCTool: a web-service-based framework for statistical comparison of stochastic optimization algorithms. Appl Soft Comput J 87:105977

    Google Scholar 

  • Ehrgott M (2005) Multicriteria optimization. Springer, Berlin, Heidelberg, pp 7–8

    MATH  Google Scholar 

  • El-Ghandour HA, Elbeltagi E (2018) Comparison of five evolutionary algorithms for optimization of water distribution networks. J Comput Civ Eng 32(1):04017066. https://doi.org/10.1007/s00366-019-00718-z

    Article  Google Scholar 

  • Ellison SLR, Barwick VJ, Farrant TJD (2009) Practical statistics for the analytical scientist a bench guide, 2nd edn. Chapter 5 outliers in analytical data. RSC Publishing, Cambridge, pp 48–58

    Google Scholar 

  • Embedded Staff (2011) CoreMark: A realistic way to benchmark CPU performance. Embedded. https://www.embedded.com/coremark-a-realistic-way-to-benchmark-cpu-performance/

  • Emmerich and Deutz (2018) A tutorial on multiobjective optimization: fundamentals and evolutionary methods. Nat Comput 17:585–609

    MathSciNet  Google Scholar 

  • Emmerich M, Deutz A, Kruisselbrink J, Shukla PK (2013) Cone-based hypervolume indicators: construction, properties, and efficient computation. In: Purshouse RC, Flemming PJ, Fonseca CM, Grego S, Shaw J (eds) Evolutionary multi-criterion optimization EMO 2013, lecture notes in computer science, vol 7811. Springer, Berlin, Heidelberg

  • Epitropakis MG, Tasoulis DK, Pavlidis NG, Plagionakos VP, Vrahatis MN (2011) Enhancing differential evolution utilizing proximity-based mutation operators. IEEE Trans Evol Comput 15(1):99–119

    Google Scholar 

  • Fahoome GF (2002) Twenty nonparametric statistics and their large sample approximations. J Mod Appl Stat Methods 1(2):248–268

    Google Scholar 

  • Farhang-Mehr A, Azarm S (2002) Diversity assesment of pareto optimal solution sets: an entropy approach. In: Proceedings of IEEE congress on evolutionary computation CEC 2002. pp 723–728

  • Fenet S, Solnon C (2003) Searching for maximum cliques with ant colony optimization. In: Cagnoni S et al (eds) Applications of evolutionary computing. EvoWorkshops 2003. Lecture notes in computer science, vol 2611. Springer, Berlin, Heidelberg

    Google Scholar 

  • Ferrer J, Chicano F, Alba E (2012) Evolutionary algorithms for the multi-objective test data generation problem. Softw Pract Exp 42(11):1–31

    Google Scholar 

  • Filipič B, Tušar T (2018) A taxonomy of methods for visualizing pareto front approximations. In: Proceedings of the genetic and evolutionary computation conference on—GECCO’18. p 649

  • Fomeni FD, Kaparis K, Letchford AN (2020) A cut-and-branch algorithm for the quadratic knapsack problem. Discrete Optim. https://doi.org/10.1016/j.disopt.2020.100579

    Article  Google Scholar 

  • Fonseca CM, Fleming PJ (1996) On the performance assessment and comparison of stochastic multiobjective optimizers. In: Proceeding of the 4th international conference on parallel problem solving from nature PPSN IV. Springer, London, pp 584–593

  • Friedrich T, Bringmann K, Voβ T (2011) The logarithmic hypervolume indicator. In: Proceedings of the 11th workshop on foundations of genetic algorithms. pp 81–92

  • García S, Herrera F (2008) Design of experiments in computational intelligence: on the use of statistical inference. In: Corchado E, Abraham A, Pedrycz W (eds) Hybrid artificial intelligence systems HAIS 2008. Lecture notes in computer science, vol 5271. Springer, Berlin, Heidelberg

    Google Scholar 

  • García S, Molina D, Lozano M, Herrera F (2009) A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behavior: a case study on the CEC’2005 special session on real parameter optimization. J Heuristics 15:617–644

    MATH  Google Scholar 

  • Garcia S, Fernández A, Luengo J, Herrera F (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. Inf Sci 180(10):2044–2064

    Google Scholar 

  • García J, Altimiras F, Pena A, Astorga G, Peredo O (2018) A binary cuckoo search big data algorithm applied to large-scale crew scheduling problems. Complexity 8395193:1–15

    Google Scholar 

  • Goh CK, Tan KC (2009) A competitive-cooperative coevolutionary paradigm for dynamic multiobjective optimization. IEEE Trans Evol Comput 13(1):103–127

    Google Scholar 

  • Gong M, Wang Z, Zhu Z, Jiao L (2017) A Similarity-based multiobjective evolutionary algorithm for deployment optimization of near space communication system. IEEE Trans Evol Comput 21(6):878–897

    Google Scholar 

  • Grange A, Kacem I, Martin S (2018) Algorithms for the bin packing problem with overlapping items. Comput Ind Eng 115:331–341

    Google Scholar 

  • Grishagin VA (1978) Operation characteristics of some global optimization algorithms. Prob Stoch Search 7:198–206 (in Russian)

    MATH  Google Scholar 

  • Gunantara N (2018) A review of multi-objective optimization: methods and its applications. Congent Eng 5(1):1502242

    Google Scholar 

  • Gunantara N, Hendrantoro G (2013) Multi-objective cross layer optimization for selection of cooperative path pairs in multihop wireless ad hoc networks. J Commun Softw Syst 9(3):170

    Google Scholar 

  • Hale JG, Zhu H, Zhou E (2020) Domination measure: a new metric for solving multiobjective optimization. INFORMS J Comput 32(3):1–17

    MathSciNet  Google Scholar 

  • Halim AH, Ismail I (2019) Combinatorial optimization: comparison of heuristic algorithms in travelling salesman problem. Arch Comput Methods Eng 26(2):367–380

    MathSciNet  Google Scholar 

  • Hamacher K (2007) Adaptive extremal optimization by detrended fluctuation analysis. J Comput Phys 227(2):1500–1509

    MATH  Google Scholar 

  • Hamacher K (2014) Online performance measures for metaheuristic optimization, hybrid metaheuristics, vol 8457. Lecture notes in computer science. Springer, Cham

    Google Scholar 

  • Hampel FR (1974) The influence curve and its role in robust estimation. J Am Stat Assoc 69:383–393

    MathSciNet  MATH  Google Scholar 

  • Hansen MP, Jaskiewicz A (1998) Evaluating the quality of approximations to the non-dominated set. Technical report IMM-REP-1998-7. Institute of Mathematical Modelling, Technical University of Denmark, Lyngby

  • Hansen N, Auger A, Finck S, Ros R (2012) Real-parameter blackbox optimization benchmarking: experimental setup. Technical report INRIA Futurs, Équipe TAO, Univ. Paris Sud, Orsay, France

  • Hansen N, Auger A, Brockhoff D, Tusar D, Tusar T (2016) COCO: performance assessment. arXiv:1605.03560 Retrieved from https://arxiv.org/abs/1605.03560

  • Hare WL, Sagastizábal C (2006) Benchmark of some nonsmooth optimization solvers for computing nonconvex proximal points. Pac J Optim 2(3):545–573

    MathSciNet  MATH  Google Scholar 

  • He J (2016) An analytic expression of relative approximation error for a class of evolutionary algorithms. In: Proceedings of IEEE congress on evolutionary computation. pp 4366–4373

  • He J, Lin G (2016) Average convergence rate of evolutionary algorithms. IEEE Trans Evol Comput 20(2):316–321

    Google Scholar 

  • Heinonen J (2001) Lectures on Analysis on Metric Spaces. Springer, New York, USA

    MATH  Google Scholar 

  • Heliodore F, Nakib A, Ismail B, Ouchraa S, Schmitt L (2017) Metaheuristic for intelligent electrical networks, ch: performance evaluation of metaheuristics. Wiley, Hoboken, pp 43–58

    Google Scholar 

  • Hellwig M, Beyer HG (2019) Benchmarking evolutionary algorithms for single-objective real-valued constrained optimization–a critical review. Swarm Evol Comput 44:927–944

    Google Scholar 

  • Hendtlass T (2004) An introduction to collective intelligence. In: Fulcher J (ed) Applied intelligent systems: new direction, studies in fuzziness and soft computing, vol 153. Springer, Berlin, Heidelberg

    Google Scholar 

  • Hendtlass T, Randall M (2001) A survey of ant colony and particle swarm meta-heuristics and their applications to discrete optimization problems. In: Proceedings of the inaugural workshop on artificial life. pp 15–25

  • Herring D, Kirley M, Yao X (2020) Dynamic multi-objective optimization of the travelling thief problem. arXiv preprint: https://arxiv.org/abs/2002.02636

  • Hoos HH (1998) Stochastic local search—methods, models, applications. Ph.D. dissertation. Technical University Darmstadt, Germany

  • Hwang C, Yoon K (1981) Multiple attribute decision making: methods and applications, a state of the art survey. Springer, New York

    MATH  Google Scholar 

  • Ibánez ML, Paquete L, Stuetzle T (2010) Exploratory analysis of stochastic local search algorithms in biobjective optimization. In: Bielstein TB, Chiarandini M, Paquete L, Preus M (eds) Experimental methods for the analysis of optimization algorithms. Springer, Berlin, pp 209–222

    Google Scholar 

  • Ibrahim D. 2019. Arm-based microcontroller projects using Mbed, Chapter 3—the arm microcontrollers. Newnes, pp 25–41

  • Ishibuchi H, Masuda H, Nojima Y (2015) A study on performance evaluation ability of a modified inverted generational distance indicator. In: Proceedings of the 2015 annual conference on genetic and evolutionary computation. ACM, Madrid, Spain, pp 695–702

  • Ivkovic N, Jakobovic D, Golub M (2016) Measuring performance of optimization algorithms in evolutionary computation. Int J Mach Learn Comput 6(3):167–171

    Google Scholar 

  • Jackson RHF, Boggs PT, Nash SG, Powell S (1991) Guidelines and reporting results of computational experiments: report of the ad hoc committee. Math Program 49:413–425

    MathSciNet  Google Scholar 

  • Jalili S, Husseinzadeh Kashan A, Hosseinzadeh Y (2016) League championship algorithms for optimum design of pin-jointed structures. J Comput Civ Eng. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000617

    Article  Google Scholar 

  • Janssens GK, Pangilinan JM (2010) Multiple criteria performance analysis of non-dominated sets obtained by multi-objective evolutionary algorithm for optimisation. In: 6th IFIP WG 12.5 international conference on artificial intelligence applications and innovations (AIAI). Larnaca, Cyprus, pp 94–103

  • Jena UK, Das PK, Kabat MR (2020) Hybridization of meta-heuristic algorithm for load balancing in cloud computing environment. J King Saud Univ Comput Inf Sci. https://doi.org/10.1016/j.jksuci.2020.01.012

    Article  Google Scholar 

  • Jiang S, Ong YS, Zhang J, Feng L (2014) Consistencies and contradictions of performance metrics in multiobjective optimization. IEEE Trans Cybern 44(12):2391–2404

    Google Scholar 

  • Jiang S, Yang S, Li M (2016) On the use of hypervolume for diversity measurement of Pareto front approximations. In: 2016 IEEE symposium series on computational intelligence (SSCI). Athens, pp 1–8

  • Jonckheere AR (1954) A distribution-free k-sample test against ordered alternatives. Biometrika 41:133–145

    MathSciNet  MATH  Google Scholar 

  • Kaur D, Murugappan M (2008) Performance enhancement in solving travelling salesman problem using hybrid genetic algorithm. In: Annual meeting of the North American fuzzy information processing society. New York City, NY, pp 1–6

  • Khorramshahgol R, Moustakis VS (1988) Delphi hierarchy process (DHP): a methodology for priority setting derived from the Delphi method and analytical hierarchy process. Eur J Oper Res 37:347–354

    MATH  Google Scholar 

  • Kiliç H, Yüzgeç U (2019) Tournament selection based antlion optimization algorithm for solving quadratic assignment problem. Eng Sci Technol Int J 22:673–691

    Google Scholar 

  • Kim JH, Lee HM, Jung D, Sadollah A (2016) Performance measures of metaheuristic algorithms. In: Kim J, Geem Z (eds) Harmony search algorithm. Advances in intelligent systems and computing, vol 382. Springer, Berlin, Heidelberg

    Google Scholar 

  • Kiran MS (2017) Particle swarm optimization with a new update mechanism. Appl Soft Comput 60:670–678

    Google Scholar 

  • Knowles JD, Corne D (2002) On metrics for comparing nondominated sets. In: Proceeding of the 2002 congress on evolutionary computation (CEC 2002). pp 711–716

  • Krink T, Vesterstrøm JS, Riget J (2002) Particle swarm optimisation with spatial particle extension. In: Proceedings of the congress on evolutionary computation, CEC’02. Honolulu, USA, pp 1474–1479

  • Laumanns M, Zitzler E, Thiele L (2000) A unified model for multi-objective evolutionary algorithms with elitism. In: Evolutionary computation proceedings of the 2000 congress on, vol 1. pp 46–53

  • Lee HM, Jung D, Sadollah A, Yoo DG, Kim JH (2019a) Generation of benchmark problems for optimal design of water distribution systems. Water 11:1637

    Google Scholar 

  • Lee HM, Jung D, Sadollah A, Lee EH, Kim JH (2019b) Performance comparison of metaheuristic optimization algorithms using water distribution system design benchmarks. In: Yadav N, Yadav A, Bansal J, Deep K, Kim J (eds) Harmony search and nature inspired optimization algorithms. Advances in intelligent systems and computing, vol 741. Springer, Singapore

    Google Scholar 

  • Lehre PK, Witt C (2011) Finite first hitting time versus stochastic convergence in particle swarm optimisation. arXiv:1105.5540v1. Retrieved from https://arxiv.org/pdf/1105.5540v1.pdf

  • Leung Y, Wang Y (2003) U-measure: a quality measure for multiobjective programming. IEEE Trans Syst Man Cybern Part A Syst Hum 33(3):337–343

    Google Scholar 

  • Lewis A, Mostaghim S, Scriven I (2009) Asynchronous multi-objective optimisation in unreliable distributed environments. In: Lewis A, Mostaghim S, Randall M (eds) Biologigally-Inspired optimisation methods, vol 210. Springer, Berlin, Heidelberg, pp 51–78

    Google Scholar 

  • Li Z, Hu X (2011) The orienteering problem with compulsory nodes and time window. ICSSSM11, Tianjin, pp 1–4

  • Li M, Yao X (2017) Dominance move: a measure of comparing solution sets in multiobjective optimization. arXiv preprint arXiv:1702.00477

  • Li M, Yao X (2019) Quality evaluation of solution sets in multiobjective optimisation: a survey. ACM Comput Surv 52(2):26

    Google Scholar 

  • Li X, Tang K, Omidvar MN, Yang Z and Qin K (2013) Benchmark functions for the CEC’2013 special session and competition on large-scale global optimization. Technical report

  • Li M, Yang S, Liu X (2014) Diversity comparison of Pareto front approximations in many-objective optimization. IEEE Trans Cybern 44(12):2568–2584

    Google Scholar 

  • Liang J, Runarsson TP, Mezura-Montes E, Clerc M, Suganthan P, Coello CC, Deb K (2006) Problem definitions and evaluation criteria for the CEC 2006 special session on constrained real-parameter optimization. J Appl Mech 41(8):1–24

    Google Scholar 

  • Liang JJ, Qu BY, Suganthan PN (2013) Problem definitions and evaluation criteria for the CEC 2014 special session and competition on single objective real-parameter numerical optimization. Technical report 201311, Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou, China and Technical Report, Nanyang Technological University, Singapore

  • Liu X, Zhang D (2019) An improved SPEA2 algorithm with local search for multi-objective investment decision-making. Appl Sci 9(8):1675. https://doi.org/10.3390/app9081675

    Article  Google Scholar 

  • Liu S, Qiu Z, Xie L (2017a) Convergence rate analysis of distributed optimization with projected subgradient problem. Automatica 83:162–169

    MATH  Google Scholar 

  • Liu Q, Chen WN, Deng JD, Gu T, Zhang H, Yu Z, Zhang J (2017b) Benchmarking stochastic algorithms for global optimization problems by visualizing confidence intervals. IEEE Trans Cybern 47(9):2924–2937

    Google Scholar 

  • Lo M, Liang Y, Hsieh J (2010) A modified variable neighborhood search algorithm for orienteering problems. In: 40th International conferecne on computers and industrial engineering. Awaji, pp 1–6

  • MacFarland TW, Yates JM (2016) Mann–Whitney U test. In: Introduction to nonparametric statistics for the biological sciences using R. Springer, Cham

  • Martí L, García J, Berlanga A, Molina JM (2016) A stopping criterion for multi-objective optimization evolutionary algorithms. Inf Sci 367:700–718

    MATH  Google Scholar 

  • Mavrovouniotis M, Li C, Yang S (2017) A survey of swarm intelligence for dynamic optimization: algorithms and applications. Swarm Evol Comput 33:1–17

    Google Scholar 

  • McGeoh CC (1996) Toward an experimental method for algorithm simulation INFORMS. J Comput 8(1):1–11

    Google Scholar 

  • Memari A, Ahmad R, Rahim Abdul AR (2017) Metaheuristic algorithms: guidelines for implementation. J Soft Comput Decis Support Syst 4(6):1–6

    Google Scholar 

  • Meng H, Zhang X, Liu S (2005) New quality measures for multiobjective programming. In: Wang L, Chen K, Ong YS (eds) Advances in natural computation. ICNC 2005, Lecture notes in computer science, vol 3611. Springer, Berlin, Heidelberg

    Google Scholar 

  • Merz P, Freisleben B (2000) Fitness landscape analysis and memetic algorithms for the quadratic assignment problem. IEEE Trans Evol Comput 4(4):337–352

    Google Scholar 

  • Messac A, Mattson CA (2004) Normal constraint method with guarantee of even representation of complete pareto frontier. AIAA J 42(10):2101–2111

    Google Scholar 

  • Mezura-Montes E, Miranda-Varela ME, Gómez-Ramón C (2010) Differential evolution in constrained numerical optimization: an empirical study. Inf Sci 180:4223–4262

    MathSciNet  MATH  Google Scholar 

  • Minella G, Ruiz R, Ciavotta M (2011) Restarted Iterated Pareto greedy algorithm for multi-objective flowshop scheduling problems. Comput Oper Res 38:1521–1533

    MathSciNet  Google Scholar 

  • Ming L, Wang Y, Cheung YM (2006) On convergence rate of class of genetic algorithms. In: Proceedings of 2006 world automation congress IEEE. Budapest, Hungary, pp 1–6

  • Mirjalili S, Lewis A (2015) Novel performance metrics for robust multi-objective optimization algorithms. Swarm Evol Comput 21:1–23

    Google Scholar 

  • Monteiro RDC, Ortiz C, Svaiter BF (2016) An adaptive accelerated first-order method for convex optimization. Comput Optim Appl 64:31–73. https://doi.org/10.1007/s10589-015-9802-0

    MathSciNet  MATH  Google Scholar 

  • Mora-Melia D, Iglesias-Rey PL, Martinez-Solano FJ, Ballesteros-Pérez P (2015) Efficiency of evolutionary algorithms in water network pipe sizing. Water Resour Manag 29:4817–4831

    Google Scholar 

  • Moré JJ, Wild SM (2009) Benchmarking derivative-free optimization algorithms. SIAM J Optim 20(1):172–191

    MathSciNet  MATH  Google Scholar 

  • Mortazavi A, Toğan V, Moloodpoor M (2019) Solution of structural and mathematical optimization problems using a new hybrid swarm intelligence optimization algorithm. Adv Eng Softw 127:106–123

    Google Scholar 

  • Murata T, Ishibuchi H, Tanaka H (1996) Multi-objective genetic algorithm and its application to flow-shop scheduling. Int J Comput Eng 30(4):957–968

    Google Scholar 

  • Nagata Y (2006) Fast EAX algorithm considering population diversity for traveling salesman problems. In: Gottlieb J, Raidl GR (eds) Evolutionary computation in combinatorial optimization. EvoCOP 2006. Lecture notes in computer science, vol 3906. Springer, Berlin, Heidelberg

    Google Scholar 

  • Nakib A, Siarry P (2013) Performance analysis of dynamic optimization algorithms. In: Alba E, Nakib A, Siarry P (eds) Metaheuristic for dynamic optimization, recent research on metaheuristics for dynamic optimization, studies in computational intelligence. Springer, Berlin

    Google Scholar 

  • Neave HR, Worthington PL (1988) Distribution free tests. Unwin Hyman Inc, Boston

    Google Scholar 

  • Nebro AJ, Durillo JJ, Nietp JG, Coello CA, Luna F, Alba E (2009) SMPSO: A new PSO-based metaheuristic for multi-objective optimization

  • Nesmachnow S (2014) An overview of metaheuristics: accurate and efficient methods for optimization. Int J Metaheuristics 3(4):320–347

    Google Scholar 

  • Ngamtawee R, Wardkein P (2014) Simplified genetic algorithm: simplify and improve RGA for parameter optimizations. Adv Electr Comput Eng 14(4):55–64

    Google Scholar 

  • Nguyen T, Yao X (2012) Continuous dynamic constrained optimization—the challenges. IEEE Trans Evol Comput 16(6):769–786

    Google Scholar 

  • Nguyen T, Yang S, Branke J (2012) Evolutionary dynamic optimization: a survey of the state of the art. Swarm Evol Comput 6:1–24

    Google Scholar 

  • Nishida K, Aguirre HE, Saito S, Shirakawa S, Akimoto Y (2018) Parameterless stochastic natural gradient method for discrete optimization and its application to hyper-parameter optimization for neural network. arXiv:1809.06517. Retrieved from https://arxiv.org/pdf/1809.06517.pdf

  • Noergaard T (2013) Chapter 4—embedded processors, embedded systems architecture, 2nd edn. Elsevier, Waltham, pp 137–229

    Google Scholar 

  • Obagbuwa IC, Adewumi AO (2014) An improved cockroach swarm optimization. Sci World J 2014:375358. https://doi.org/10.1155/2014/375358

    Google Scholar 

  • Okabe T, Jin Y, Sendhoff B (2003) A critical survey of performance indices for multi-objective optimisation. In: 2003 Congress on evolutionary computation, CEC’03. Canberra, ACT, Australia

  • Olewuezi NP (2011) Note on the comparison of some outlier labeling techniques. J Math Stat 7(4):353–355

    Google Scholar 

  • Olorunda O, Engelbrecht AP (2008) Measuring exploration/exploitation in particle swarm using swarm diversity. In: IEEE congress on evolutionary computationm. Hong Kong, pp 1128–1134

  • Opricovic S (1998) Multicriteria optimization of civil engineering systems. Fac Civ Eng Belgrade 2:5–21

    Google Scholar 

  • Page EB (1963) Ordered hypotheses for multiple treatments: a significance test for linear ranks. J Am Stat Assoc 58:216–230

    MathSciNet  MATH  Google Scholar 

  • Pan QK, Gao L, Wang L, Liang J, Li XY (2019) Effective heuristics and metaheuristics to minimize total flowtime for the distributed permutation flowshop problem. Expert Syst Appl 124:309–324

    Google Scholar 

  • Paul PV, Moganarangan N, Sampath KS, Raju R, Vengattaraman T, Dhavachelvan P (2015) Performance analyses over population seeding techniques of the permutation-coded genetic algorithm: an empirical study based on travelling salesman problems. Appl Soft Comput 32:383–402

    Google Scholar 

  • Peitz S, Dellnitz M (2018) A survey of recent trends in multiobjective optimal control—surrogate models, feedback control and objective reduction. Math Comput Appl 23:30

    MathSciNet  Google Scholar 

  • Pierezan J, Maidl G, Yamao EM, Coelho LDS, Mariani VC (2019) Cultural coyote optimization algorithm applied to a heavy duty gas turbine operation. Energy Convers Manag 199:111932

    Google Scholar 

  • Polap D, Kesik K, Woźniak M, Damaševičius R (2018) Parallel technique for metaheuristic algorithms using devoted local search and manipulating the solutions space. Appl Sci. 8(2):1–25

    Google Scholar 

  • Price KV, Awad NH, Ali MZ, Suganthan PN (2018) The 100-digit challenge: problem definitions and evaluation criteria for the 100-digit shallenge special session and competition on single objective numerical optimization. Tech. rep. Nanyang Technological University, Singapore

  • Pyzdek T, Keller P (2010) The six sigma handbook, 3rd edn. Tata Mcgraw Hill, New York, pp 389–391

    Google Scholar 

  • Rakesh K, Suganthan PN (2017) An ensemble of kernel ridge regression for multi-class classification. Procedia Comput Sci 108:375–383

    Google Scholar 

  • Rardin RL, Uzsoy R (2018) Experimental evaluation of heuristic optimization algorithm: a tutorial. J Heuristics 7(3):261–304

    MATH  Google Scholar 

  • Rashedi E, Nezamabadi-pour H, Saryazdi S (2009) GSA: a gravitational search algorithm. Inf Sci 179:2232–2248

    MATH  Google Scholar 

  • Ray SS, Bandyopadhyay S, Pal SK (2007) Genetic operators for combinatorial optimization in TSP and microarray gene ordering. J Appl Intell 26(3):183–195

    MATH  Google Scholar 

  • Rey D, Neuhäuser M (2011) Wilcoxon-signed-rank test. In: Lovric M (ed) International encyclopedia of statistical science. Springer, Berlin, Heidelberg

    Google Scholar 

  • Ribeiro CC, Rosseti I, Vallejos R (2009) On the use of run time distributions to evaluate and compare stochastic local search algorithms. In: SLS ‘09 Proceedings of the second international workshop on engineering stochastic local search algorithms. Designing, implementing and analyzing effective heuristics. Springer, Brussels, Belgium, pp 16–30

  • Riget J, Vesterstrøm JS (2002) A diversity-guided particle swarm optimizer—the arpso. EVALife Project Group, technical report 2002-02, Department of Computer Science, Aarhus Universitet

  • Riquelme N, Lücken CV, Barán B (2015) Performance metrics in multi-objective optimization. In: 2015 XLI Latin American computing conference

  • Rouky N, Abourraja MN, Boukachour J, Boudebous D, Alaoui AH, Khoukhi FE (2019) Simulation optimization based ant colony algorithm for the uncertain quay crane scheduling problem. Int J Ind Eng Comput 10:111–132

    Google Scholar 

  • Roy B (1991) The outranking approach and the foundations of electre methods. Theor Decis 31:49–73. https://doi.org/10.1007/BF00134132

  • Saad AEH, Dong Z, Karimi M (2017) A comparative study on recently-introduced nature-based global optimization methods in complex mechanical system design. Algorithms 10(4):1–30

    MathSciNet  MATH  Google Scholar 

  • Saaty TL (2004) Decision making-The analytic hierarchy and network processes (AHP, ANP). J Syst Sci Syst Eng 13:1–35

    Google Scholar 

  • Sadollah A, Sayyandi H, Yoo DG, Lee HM, Kim JH (2018) Mine blast harmony search: a new hybrid method for improving exploration and exploitaqtion capabilities. Appl Soft Comput 68:548–564

    Google Scholar 

  • Salomon R (1998) Evolutionary algorithms and gradient search: similarities and differences. IEEE Trans Evol Comput 2(2):45–55

    Google Scholar 

  • Santos G, Chagas JBC (2018) The thief orienteering problem: formulation and heuriatic approaches. In: IEEE congress on evolutionary computation (CEC). Rio de Jeneiro, pp 1–9

  • Santos T, Xavier S (2018) A convergence indiator for multi-objective optimisation algorithms. Tend Math Apli Comput 19(3):437–448

    Google Scholar 

  • Santos LFOM, Iwayama RS, Cavalcanti LB, Turi LM, Morais FEdS, Mormilho G, Cunha CB (2019) A variable neighborhood search algorithm for the bin packing problem with compatible categories. Expert Syst Appl 124:209–225

    Google Scholar 

  • Schott JR (1995) Fault tolerant design using single and multicretiria genetic algorithm optimization. Master’s thesis, Massachusetts Institute of Technology

  • Schutze O, Esquivel X, Lara A, Coello CAC (2012) Using the averaged hausdorff distance as a performance measure in evolutionary multiobjective optimization. IEEE Trans Evol Comput 16(4):504–522

    Google Scholar 

  • Senning JR (2015) Computing and estimating the rate of convergence. Dept. Math. Comput. Sci, Gordon Colledge, Wenham

    Google Scholar 

  • Sergeyev YD, Kvasov DE, Mukhametzhanov MS (2017) Operational zones for comparing metaheuristic and deterministic one-dimensional global optimization algorithms. Math Comput Simul 141:96–109

    MathSciNet  Google Scholar 

  • Sergeyev YD, Kvasov DE, Mukhametzhanov MS (2018) On the efficiency of nature-inspired metaheuristics in expensive global optimization with limited budget. Sci Rep 8(453):1–9

    Google Scholar 

  • Sheskin D (2006) Handbook of parametric and nonparametric statistical procedures. Chapman & Hall/CRC, Boca Raton

    MATH  Google Scholar 

  • Shukor SA, Shaheed IM, Abdullah S (2018) Population initialisation methods for fuzzy job-shop scheduling problems: issues and future trends. Int J Adv Sci Eng Inf Tech 8(4–2):1820–1828

    Google Scholar 

  • Silva VL, Wanner EF, Cerqueira SA, Takahashi RH (2007) A new performance metric for multiobjective optimization: the integrated sphere counting. In: Proceedings of the 2007 IEEE congress on evolutionary computation. IEEE Press, pp 3625–3630

  • Silva BCH, Fernandes IFC, Goldbarg MC, Goldbarg EFG (2020) Quota travelling salesman problem with passengers, incomplete ride and collection time optimization by ant-based algorithms. Comput Oper Res. https://doi.org/10.1016/j.cor.2020.104950

    Article  MathSciNet  Google Scholar 

  • Skidmore GS (2006) Metaheuristics and combinatorial optimization problems. Master’s thesis. Rochester Institute of Technology

  • Smith-Miles, Van Hemert J (2011) Discovering the suitability of optimisation algorithms by learning from evolved instances. Ann Math Artif Intell 61:87–104

    MathSciNet  MATH  Google Scholar 

  • Smith-Miles K, Van Hemert J, Lim XY (2010) Understanding TSP Difficulty by Learning from Evolved Instances. In: Blum C, Battiti R (eds) Learning and intelligent optimization. LION 2010. Lecture notes in computer science, vol 6073. Springer, Berlin¸Heidelberg

    Google Scholar 

  • Snedecor GW, Cochran WG (1989) Statistical methods, 8th edn. Iowa State University Press, Iowa

    MATH  Google Scholar 

  • Suganthan PN, Hansen N, Liang JJ, Deb K, Chen YP, Auger A, Tiwari S (2005) Problem definitions and evaluation criteria for the CEC 2005 special session on real-parameter optimization. Technical report

  • Sun Y, Yen GG, Yi Z (2018) IGD indicator-based evolutionary algorithm for many-objective optimization problems. IEEE Trans Evol Comput 23(2):173–187

    Google Scholar 

  • Suzuki J (1995) A Markov chain analysis on simple genetic algorithm. IEEE Trans Syst Man Cybern 25(4):655–659

    Google Scholar 

  • Tang L, Li Z, Luo L, Liu B (2015) Multi-strategy adaptive particle swarm optimization for numerical optimization. Eng Appl Artif Intell 37:9–19

    Google Scholar 

  • Terpstra TJ (1952) The asymptotic normality and consistency of Kendall’s test against trend, when ties are present in one ranking. Indag Math 14:327–333

    MathSciNet  MATH  Google Scholar 

  • Trefethen N (2002) A hundred-dollar, hundred-digit challenge. SIAM News 35(1):65

    Google Scholar 

  • Trojanowski K, Michalewicz Z (1999) Searching for optima in non-stationary environments, In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 3. pp 1843–1850

  • Tsai HK, Yang JM, Tsai YF, Kao CY (2003) Heterogeneous selection genetic algorithms for travelling salesman problems. Eng Optim 35(3):297–311

    Google Scholar 

  • Tsai HK, Yang JM, Tsai YF, Kao CY (2004) An evolutionary algorithm for large travelling salesman problems. IEEE Trans Syst Man Cybern Part B Cybern 34(4):1718–1729

    Google Scholar 

  • Tsai CW, Tseng SP, Chiang MC, Yang CS, Hong TP (2014) A high-performance genetic algorithms: using traveling salesman problem as a case. Sci World J 2014:1–14

    Google Scholar 

  • Tseng LY, Yang SB (2001) A genetic approach to the automatic clustering problem. Pattern Recognit 34(2):415–424

    MATH  Google Scholar 

  • Tukey JW (1977) Exploratory data analysis. Addison-Wesley, Philippines

    MATH  Google Scholar 

  • Tušar T, Filipič B (2014) Visualizing exact and approximated 3d empirical attainment functions. Math Probl Eng 569346:1–18

    Google Scholar 

  • Upton G, Cook I (2014) Dictionary of statistics. Oxford University Press, Oxford

    MATH  Google Scholar 

  • Van Hemert J (2005) Property analysis of symmetric travelling salesman problem instances acquired through evolution. In: Raidl GR, Gottlieb J (eds) Evolutionary computation in combinatorial optimization. EvoCOP 2005. Lecture notes in computer science, vol 3448. Springer, Berlin, Heidelberg

    Google Scholar 

  • Van Veldhuizen DA, Lamont GB (1999) Multiobjective evolutionary algorithm test suites. In Proceedings of the ACM symposium on applied computing. pp 351–357

  • Van Veldhuizen DA, Lamont GB (2000) On measuring multiobjective evolutionary algorithm performance. In: Proceedings of the IEEE congress on evolutionary computation. pp 204–211

  • Vargas A, Bogoya J (2018) A generalization of the averaged hausdorff distance. Comput Sist 22(2):331–345

    Google Scholar 

  • Vaz AIF, Vicente LN (2009) PSwarm: a hybrid solver for linearly constrained global derivative-free optimization. Optim Methods Softw 24(4–5):669–685

    MathSciNet  MATH  Google Scholar 

  • Wagner M, Lindauer M, Misir M, Nallaperuma S, Hutter F (2018) A case study of algorithm selection for the travelling thief problem. J Heuristics 24(3):295–320

    Google Scholar 

  • Wang WL, Li WK, Wang Z, Li L (2019) Opposition-based multi-objective whale optimization algorithm with global grid ranking. Neurocomputing 341:41–59

    Google Scholar 

  • Wanner EF, Guimarães FG, Takahashi RH, Fleming PJ (2006) A quadratic approximation-based local search procedure for multiobjective genetic algorithms. In: Procedings of the IEEE congress on evolutionary computation. pp 938–945

  • Weck OLD (2004) Multiobjective optimization: history and promise. In: Proceedings of 3rd China–Japan–Korea joint symposium on optimization of structural and mechanical systems. Kanazawa, Japan

  • Whitley D, Rana S, Dzubera J, Mathias KE (1996) Evaluating evolutionary algorithms. Artif Intell 85:245–276

    Google Scholar 

  • Woolway M, Majozi T (2019) On the application of a metaheuristic suite with parallel implementations for the scheduling of multipurpose batch plants. Comput Chem Eng 126:371–390

    Google Scholar 

  • Wu J, Azarm S (2001) Metrics for quality assessment of a multiobjective optimization solution set, transactions of the ASME. J Mech Des 123:18–25

    Google Scholar 

  • Wu W, Yagiura M, Ibaraki T (2018) Generalized assignment problem. In: Gonzalez TF (ed) Handbook of approximation algorithms and metaheuristics, methodologies and traditional applications, vol 1, 2nd edn. Chapman and Hall/CRC, New York

    Google Scholar 

  • Wu Z, Jiang B, Karimi HR (2020) A logarithmic descent direction algorithm for the quadratic knapsack problem. Appl Math Comput 369:124854

    MathSciNet  MATH  Google Scholar 

  • Yaghini M, Momeni M, Sarmadi M (2011) DIMMA-implemented metaheuristics for finding shortest hamiltonian path between Iranian cities using sequential DOE approach for parameters tuning. Int J Appl Metaheuristic Comput 2(2):74–92

    Google Scholar 

  • Yang XS (2010) Nature-inspired metaheuristic algorithms, 2nd edn. Luniver Press, Bristol

    Google Scholar 

  • Yang XS (2011) Metaheuristics optimization: algorithm analysis and open problems. Lect Notes Comput Sci 6630:21–32

    Google Scholar 

  • Yang S, Li C (2010) A Clustering particle swarm optimizer for locating and tracking multiple optima in dynamic environments. IEEE Trans Evol Comput 14(6):959–974

    Google Scholar 

  • Yang S, Yao X (2013) Evolutionary computation for dynamic optimization problems, ch. 8. Springer, Berlin, pp 203–205

    Google Scholar 

  • Yang XS, Deb S, Fong S (2011) Accelerated particle swarm optimization and support vector machine for business optimization and applications. In: Networked digital technologies (NDT2011), communications in computer and information science, vol 136. Springer, pp 53–66

  • Yen GG, He Z (2014) Performance metric ensemble for multiobjective evolutionary algorithms. IEEE Trans Evol Comput 18(1):131–144

    Google Scholar 

  • Yu X, Zhang X (2014) Unit commitment using Lagrangian relaxation and particle swarm optimization. Electr Power Energy Syst 61:510–522

    Google Scholar 

  • Yu HJ, Zhang LP, Chen DZ, Hu SX (2005) Adaptive particle swarm optimization algorithm based on feedback mechanism. J Zhejiang Univ (Eng Sci Ed) 39(9):1286–1291

    Google Scholar 

  • Yu X, Lu Y, Yu X (2018) Evaluating multiobjective evolutionary algorithms using MCDM methods. Math Probl Eng 9751783:1–13

    Google Scholar 

  • Yu G, Jin Y, Olhofer M (2019) Benchmark problems and performance indicators for search of knee points in multiobjective optimization. IEEE Trans Cybern 99:1–14

    Google Scholar 

  • Zehmakan AN (2015) Bin packing problem: two approximation algorithms. arXiv:1508.01376. Retrieved from https://arxiv.org/pdf/1508.01376v1.pdf

  • Zhan B, Haslbeck MPL (2018) Verifying asymptotic time complexity of imperative programs in Isabelle. In: Galmiche D, Schulz S, Sebastiani R (eds) Automatic reasoning. IJCAR. Lecture notes in computer science, vol 10900. Springer, Cham

    Google Scholar 

  • Zhang X, Hu H, Wang L, Sun Z, Zhang Y, Han K, Xu Y (2018) A novel bin design problem and high performance algorithm for e-commerce logistics system. arXiv:1812.02565v1. Retrieved from: https://arxiv.org/pdf/1812.02565v1.pdf

  • Zhang HG, Liang ZH, Liu HJ, Wang R, Liu YA (2020) Ensemble framework by using nature inspired algorithms for the early-stage forest fire rescue—a case study of dynamic optimization problems. Eng Appl Artif Intell 90:103517

    Google Scholar 

  • Zheng K, Yang R, Xu H, Hu J (2017) A new distribution metric for comparing pareto optimal solutions. Struct Multidiscip Optim 55(1):53–62

    MathSciNet  Google Scholar 

  • Zhong Y, Wang L, Lin M, Zhang H (2019) Discrete pigeon-inspired optimization algorithm with metropolis acceptance criterion for large-scale travelling salesman problem. Swarm Evol Comput 48:134–144

    Google Scholar 

  • Zhou A, Jin Y, Zhang Q, Sendhoff B, Tsang E (2006) Combining model-based and genetics-based offspring generation for multi-objective optimization using a convergence criterion. In: Evolutionary computation CEC 2006. IEEE congress on evolutionary computation. pp 892–899

  • Zhou AH, Zhu LP, Hu B, Deng S, Song Y, Qiu H, Pan S (2019) Traveling-salesman-problem algorithm based on simulated annealing and gene-expression programming. Information 10(7):1–15

    Google Scholar 

  • Zitzler E (1999) Evolutionary algorithms for multiobjective optimization: methods and appliations. Doctor of technical sciences dissertation, Swiss Federal Institute of Technology Zurich

  • Zitzler E, Thiele L (1998) Multiobjective optimization using evolutionary algorithms—a comparative case study. In: Proceedings of the international conference on parallel problem solving from nature. Armsterdam, Netherlands, pp 292–301

  • Zitzler E, Thiele L (1999) Multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. IEEE Trans Evol Comput 3(4):257–271

    Google Scholar 

  • Zitzler E, Deb K, Thiele L (2000) Comparison of multiobjective evolutionary algorithms: empirical results. Evol Comput 8(2):173–195

    Google Scholar 

  • Zitzler E, Thiele L, Laumanns M, Foncesca CM, Foncesca VG (2003) Performance assessment of multiobjective optimizers: an analysis and review. IEEE Trans Evol Comput 7(2):117–132

    Google Scholar 

  • Zitzler E, Brockhoff D, Thiele L (2007) The hypervolume indicator revisited: on the design of pareto-compliant indicators via weighted integration. In: Proceedings of the 4th international conference on evolutionary multi-criterion optimization (EMO’07). pp 862–876

Download references

Author information

Authors and Affiliations

  1. Electrical and Electronic Engineering, Universiti Teknologi PETRONAS, Tronoh, Perak, Malaysia

    A. Hanif Halim & I. Ismail

  2. Electronics and Communication Science Unit, Indian Statistical Institute, 203 B. T. Road, Kolkata, India

    Swagatam Das

Authors
  1. A. Hanif Halim
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. I. Ismail
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Swagatam Das
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Swagatam Das.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Halim, A.H., Ismail, I. & Das, S. Performance assessment of the metaheuristic optimization algorithms: an exhaustive review. Artif Intell Rev 54, 2323–2409 (2021). https://doi.org/10.1007/s10462-020-09906-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10462-020-09906-6

Keywords