Skip to main content
SpringerLink
Log in

A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization

  • Published:
Journal of Heuristics Aims and scope Submit manuscript

Abstract

In recent years, there has been a growing interest for the experimental analysis in the field of evolutionary algorithms. It is noticeable due to the existence of numerous papers which analyze and propose different types of problems, such as the basis for experimental comparisons of algorithms, proposals of different methodologies in comparison or proposals of use of different statistical techniques in algorithms’ comparison.

In this paper, we focus our study on the use of statistical techniques in the analysis of evolutionary algorithms’ behaviour over optimization problems. A study about the required conditions for statistical analysis of the results is presented by using some models of evolutionary algorithms for real-coding optimization. This study is conducted in two ways: single-problem analysis and multiple-problem analysis. The results obtained state that a parametric statistical analysis could not be appropriate specially when we deal with multiple-problem results. In multiple-problem analysis, we propose the use of non-parametric statistical tests given that they are less restrictive than parametric ones and they can be used over small size samples of results. As a case study, we analyze the published results for the algorithms presented in the CEC’2005 Special Session on Real Parameter Optimization by using non-parametric test procedures.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  • Abramowitz, M.: Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Tables. Dover, New York (1974)

    Google Scholar 

  • Auger, A., Hansen, N.: A restart CMA evolution strategy with increasing population size. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1769–1776 (2005a)

  • Auger, A., Hansen, N.: Performance evaluation of an advanced local search evolutionary algorithm. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1777–1784 (2005b)

  • Ballester, P.J., Stephenson, J., Carter, J.N., Gallagher, K.: Real-parameter optimization performance study on the CEC-2005 benchmark with SPC-PNX. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 498–505 (2005)

  • Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation: The New Experimentalism. Springer, New York (2006)

    MATH  Google Scholar 

  • Czarn, A., MacNish, C., Vijayan, K., Turlach, R., Gupta, R.: Statistical exploratory analysis of genetic algorithms. IEEE Trans. Evol. Comput. 8(4), 405–421 (2004)

    Article  Google Scholar 

  • Demšar, J.: Statistical comparisons of classifiers over multiple data sets. J. Mach. Learn. Res. 7, 1–30 (2006)

    MathSciNet  Google Scholar 

  • Gallagher, M., Yuan, B.: A general-purpose tunable landscape generator. IEEE Trans. Evol. Comput. 10(5), 590–603 (2006)

    Article  Google Scholar 

  • García, S., Molina, D., Lozano, M., Herrera, F.: An experimental study on the use of non-parametric tests for analyzing the behaviour of evolutionary algorithms in optimization problems. In: Proceedings of the Spanish Congress on Metaheuristics, Evolutionary and Bioinspired Algorithms (MAEB’2007), pp. 275–285 (2007) (in Spanish)

  • García-Martínez, C., Lozano, M.: Hybrid real-coded genetic algorithms with female and male differentiation. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 896–903 (2005)

  • Hansen, N.: (2005). Compilation of results on the CEC benchmark function set. Tech. Report, Institute of Computational Science, ETH Zurich, Switzerland. Available as http://www.ntu.edu.sg/home/epnsugan/index_files/CEC-05/compareresults.pdf

  • Hochberg, Y.: A sharper Bonferroni procedure for multiple tests of significance. Biometrika 75, 800–803 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  • Holm, S.: A simple sequentially rejective multiple test procedure. Scandinavian J. Statist. 6, 65–70 (1979)

    MATH  MathSciNet  Google Scholar 

  • Hooker, J.: Testing heuristics: we have it all wrong. J. Heuristics 1(1), 33–42 (1997)

    Article  MathSciNet  Google Scholar 

  • Iman, R.L., Davenport, J.M.: Approximations of the critical region of the Friedman statistic. Commun. Stat. 18, 571–595 (1980)

    Article  Google Scholar 

  • Kramer, O.: An experimental analysis of evolution strategies and particle swarm optimisers using design of experiments. In: Proceedings of the Genetic and Evolutionary Computation Conference 2007 (GECCO’2007), pp. 674–681 (2007)

  • Liang, J.J., Suganthan, P.N.: Dynamic multi-swarm particle swarm optimizer with local search. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 522–528 (2005)

  • Molina, D., Herrera, F., Lozano, M.: Adaptive local search parameters for real-coded memetic algorithms. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 888–895 (2005)

  • Moreno-Pérez, J.A., Campos-Rodríguez, C., Laguna, M.: On the comparison of metaheuristics through non-parametric statistical techniques. In: Proceedings of the Spanish Congress on Metaheuristics, Evolutionary and Bioinspired Algorithms (MAEB’2007), pp. 286–293 (2007) (in Spanish)

  • Morse, D.T.: Minsize2: a computer program for determining effect size and minimum sample size for statistical significance for univariate, multivariate, and nonparametric tests. Educ. Psychol. Meas. 59(3), 518–531 (1999)

    Article  Google Scholar 

  • Noether, G.E.: Sample size determination for some common nonparametric tests. J. Am. Stat. Assoc. 82(398), 645–647 (1987)

    Article  MATH  MathSciNet  Google Scholar 

  • Ortiz-Boyer, D., Hervás-Martínez, C., García-Pedrajas, N.: Improving crossover operators for real-coded genetic algorithms using virtual parents. J. Heuristics 13, 265–314 (2007)

    Article  Google Scholar 

  • Ozcelik, B., Erzurumlu, T.: Comparison of the warpage optimization in the plastic injection molding using ANOVA, neural network model and genetic algorithm. J. Mater. Process. Technol. 171(3), 437–445 (2006)

    Article  Google Scholar 

  • Patel, J.K., Read, C.B.: Handbook of the Normal Distribution. Dekker, New York (1982)

    MATH  Google Scholar 

  • Pošík, P.: Real-parameter optimization using the mutation step co-evolution. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 872–879 (2005)

  • Qin, A.K., Suganthan, P.N.: Self-adaptive differential evolution algorithm for numerical optimization. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1785–1791 (2005)

  • Rojas, I., González, J., Pomares, H., Merelo, J.J., Castillo, P.A., Romero, G.: Statistical analysis of the main parameters involved in the design of a genetic algorithm. IEEE Trans. Syst. Man Cybern. Part C 32(1), 31–37 (2002)

    Article  Google Scholar 

  • Rônkkônen, J., Kukkonen, S., Price, K.V.: Real-parameter optimization using the mutation step co-evolution. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 506–513 (2005)

  • Shaffer, J.P.: Multiple hypothesis testing. Annu. Rev. Psychol. 46, 561–584 (1995)

    Article  Google Scholar 

  • Sinha, A., Tiwari, S., Deb, K.: A population-based, steady-state procedure for real-parameter optimization. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 514–521 (2005)

  • Suganthan, P.N., Hansen, N., Liang, J.J., Deb, K., Chen, Y.P., Auger, A., Tiwari, S.: (2005). Problem definitions and evaluation criteria for the CEC 2005 Special Session on Real Parameter Optimization. Tech. Report, Nanyang Technological University. Available as http://www.ntu.edu.sg/home/epnsugan/index_files/CEC-05/Tech-Report-May-30-05.pdf

  • Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. CRC Press, Boca Raton (2003)

    Google Scholar 

  • Whitley, D.L., Beveridge, R., Graves, C., Mathias, K.E.: Test driving three 1995 genetic algorithms: new test functions and geometric matching. J. Heuristics 1(1), 77–104 (1995)

    Article  MATH  Google Scholar 

  • Whitley, D.L., Rana, S., Dzubera, J., Mathias, K.E.: Evaluating evolutionary algorithms. Artif. Intell. 85(1–2), 245–276 (1996)

    Article  Google Scholar 

  • Wolpert, D.H., Macready, W.G.: No free lunch theorems for optimization. IEEE Trans. Evol. Comput. 1(1), 67–82 (1997)

    Article  Google Scholar 

  • Wright, S.P.: Adjusted p-values for simultaneous inference. Biometrics 48, 1005–1013 (1992)

    Article  Google Scholar 

  • Yuan, B., Gallagher, M.: On building a principled framework for evaluating and testing evolutionary algorithms: a continuous landscape generator. In: Proceedings of the 2003 Congress on Evolutionary Computation (CEC2003), pp. 451–458 (2003)

  • Yuan, B., Gallagher, M.: Experimental results for the Special Session on Real-Parameter Optimization at CEC 2005: a simple, continuous EDA. In: Proceedings of the 2005 IEEE Congress on Evolutionary Computation (CEC’2005), pp. 1792–1799 (2005)

  • Zar, J.H.: Biostatistical Analysis. Prentice Hall, Englewood Cliffs (1999)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Artificial Intelligence, University of Granada, Granada, 18071, Spain

    Salvador García, Manuel Lozano & Francisco Herrera

  2. Department of Computer Engineering, University of Cádiz, Cádiz, Spain

    Daniel Molina

Authors
  1. Salvador García
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Daniel Molina
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Manuel Lozano
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francisco Herrera
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Salvador García.

Additional information

This work was supported by Project TIN2005-08386-C05-01.

S. García holds a FPU scholarship from Spanish Ministry of Education and Science.

Rights and permissions

Reprints and permissions

About this article

Cite this article

García, S., Molina, D., Lozano, M. et al. A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special Session on Real Parameter Optimization. J Heuristics 15, 617–644 (2009). https://doi.org/10.1007/s10732-008-9080-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10732-008-9080-4

Keywords

Search

Navigation