Abstract
Choosing a proper approximation model should be the first and the most fundamental problem to be solved when dealing with surrogate-assisted evolutionary algorithms. Till now, most of the model selection methods emphasize on obtaining the best surrogate model basing on model accuracy assessments. As the population ranking is of the most important part in evolutionary optimization, the target function of surrogate model should focus on the right ranking of candidate solutions. Therefore, in this paper, we make a comparison study on several model quality measures which basically dedicated to measuring the capability of surrogate model in selecting and ranking the candidate solutions. In order to investigate the compatibility between accuracy assessments and ranking correlation methods, four algorithms with different model selection strategies based on different quality measures are designed and comparative study is made by contrasting them to three specific surrogate-assisted evolutionary algorithms as well as the standard particle swarm optimization. Simulation results on ten commonly used benchmark problems and one engineering case demonstrate the efficacy of the designed model selection strategies and meanwhile provide further insight into the three model quality measures studied in this paper in model selection.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Buche D, Schraudolph NN, Koumoutsakos P (2005) Accelerating evolutionary algorithms with Gaussian process fitness function models. IEEE Trans Syst Man Cybern C 35:183–194. https://doi.org/10.1109/TSMCC.2004.841917
Clerc M (1999) The swarm and the queen: towards a deterministic and adaptive particle swarm optimization. In: Proceedings of the 1999 congress on evolutionary computation-CEC99 (Cat. No. 99TH8406), vol 1953, pp 1–1957. https://doi.org/10.1109/cec.1999.785513
Derrac J, García S, Molina D, Herrera F (2011) A practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm Evol Comput 1:3–18. https://doi.org/10.1016/j.swevo.2011.02.002
Díaz-Manríquez A, Toscano-Pulido G, Gómez-Flores W (2011) On the selection of surrogate models in evolutionary optimization algorithms. In: 2011 IEEE congress of evolutionary computation (CEC), pp 2155–2162. https://doi.org/10.1109/cec.2011.5949881
Díaz-Manríquez A, Toscano G, Coello Coello CA (2016) Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput. https://doi.org/10.1007/s00500-016-2140-z
Díaz-Manríquez A, Toscano G, Coello Coello CA (2017) Comparison of metamodeling techniques in evolutionary algorithms. Soft Comput 21:5647–5663. https://doi.org/10.1007/s00500-016-2140-z
Efron B, Tibshirani RJ (1994) An introduction to the bootstrap. CRC Press, Boca Raton
Emmerich MTM, Giannakoglou KC, Naujoks B (2006) Single- and multiobjective evolutionary optimization assisted by Gaussian random field metamodels. IEEE Trans Evol Comput 10:421–439. https://doi.org/10.1109/TEVC.2005.859463
Fisher NI, Hall P (1991) Bootstrap algorithms for small samples. J Stat Plan Inference 27:157–169. https://doi.org/10.1016/0378-3758(91)90013-5
Forrester AIJ, Keane AJ (2009) Recent advances in surrogate-based optimization. Prog Aerosp Sci 45:50–79. https://doi.org/10.1016/j.paerosci.2008.11.001
Hüsken M, Jin Y, Sendhoff B (2005) Structure optimization of neural networks for evolutionary design optimization. Soft Comput 9:21–28. https://doi.org/10.1007/s00500-003-0330-y
Ingu T, Takagi H (1999) Accelerating a GA convergence by fitting a single-peak function. In: Fuzzy systems conference proceedings, FUZZ-IEEE ‘99. 1999 IEEE International, vol 1413, pp 1415–1420. https://doi.org/10.1109/fuzzy.1999.790111
Jin Y (2005) A comprehensive survey of fitness approximation in evolutionary computation. Soft Comput 9:3–12
Jin Y (2011) Surrogate-assisted evolutionary computation: recent advances and future challenges. Swarm Evol Comput 1:61–70. https://doi.org/10.1016/j.swevo.2011.05.001
Jin Y, Michael H (2003) Quality measures for approximate models in evolutionary computation. In: GECCO, pp 170–173
Jin R, Chen W, Simpson TW (2001) Comparative studies of metamodelling techniques under multiple modelling criteria. Struct Multidiscipl Optim 23:1–13. https://doi.org/10.1007/s00158-001-0160-4
Jones DR, Schonlau M, Welch WJ (1998) Efficient global optimization of expensive black-box functions. J Global Optim 13:455–492. https://doi.org/10.1023/a:1008306431147
Lawrence CT, Tits AL (2001) A computationally efficient feasible sequential quadratic programming algorithm. SIAM J Optim 11:1092–1118. https://doi.org/10.1137/s1052623498344562
Le MN, Ong YS, Menzel S, Jin Y, Sendhoff B (2013) Evolution by adapting surrogates. Evol Comput 21:313–340. https://doi.org/10.1162/EVCO_a_00079
Lendasse A, Wertz V, Verleysen M (2003) Model selection with cross-validations and bootstraps: application to time series prediction with RBFN models. In: Artificial neural networks and neural information processing–ICANN/ICONIP 2003, pp 573–580. Springer, Berlin
Lesh FH (1959) Multi-dimensional least-squares polynomial curve fitting. Commun ACM 2:29–30. https://doi.org/10.1145/368424.368443
Liang K-H, Yao X, Newton C (2000) Evolutionary search of approximated n-dimensional landscapes. Int J Knowl Based Intell Eng Syst 4:172–183
Lim D, Ong Y-S, Jin Y, Sendhoff B (2007) A study on metamodeling techniques, ensembles, and multi-surrogates in evolutionary computation. Paper presented at the Proceedings of the 9th annual conference on Genetic and evolutionary computation, London, England
Lim D, Ong Y-S, Jin Y, Sendhoff B (2008) Evolutionary optimization with dynamic fidelity computational models. In: Huang D-S, Wunsch DC, Levine DS, Jo K-H (eds) Advanced intelligent computing theories and applications. With Aspects of artificial intelligence: 4th international conference on intelligent computing, ICIC 2008 Shanghai, China, September 15-18, 2008. Springer, Berlin, pp 235–242 https://doi.org/10.1007/978-3-540-85984-0_29
Lim D, Jin Y, Ong YS, Sendhoff B (2010) Generalizing surrogate-assisted evolutionary computation. IEEE Trans Evol Comput 14:329–355. https://doi.org/10.1109/TEVC.2009.2027359
Liu H, Hervas J-R, Ong Y-S, Cai J, Wang Y (2018) An adaptive RBF-HDMR modeling approach under limited computational budget. Struct Multidiscipl Optim 57:1233–1250. https://doi.org/10.1007/s00158-017-1807-0
Lophaven SN, Nielsen HB, Søndergaard J (2002) DACE-A Matlab Kriging toolbox, version 2.0
Matheron G (1963) Principles of geostatistics. Econ Geol 58:1246–1266
Mokarram V, Banan MR (2018) A new PSO-based algorithm for multi-objective optimization with continuous and discrete design variables. Struct Multidiscipl Optim 57:509–533. https://doi.org/10.1007/s00158-017-1764-7
Munk DJ, Vio GA, Steven GP (2015) Topology and shape optimization methods using evolutionary algorithms: a review. Struct Multidiscipl Optim 52:613–631. https://doi.org/10.1007/s00158-015-1261-9
Nakayama H, Arakawa M, Sasaki R (2001) A computational intelligence approach to optimization with unknown objective functions. In: Artificial neural networks—ICANN 2001. Springer, Berlin, pp 73–80
Powell M (2001) Radial basis function methods for interpolation to functions of many variables. In: HERCMA. Citeseer, pp 2–24
Praveen C, Duvigneau R (2009) Low cost PSO using metamodels and inexact pre-evaluation: application to aerodynamic shape design. Comput Methods Appl Mech Eng 198:1087–1096. https://doi.org/10.1016/j.cma.2008.11.019
Queipo NV, Haftka RT, Shyy W, Goel T, Vaidyanathan R, Kevin Tucker P (2005) Surrogate-based analysis and optimization. Prog Aerosp Sci 41:1–28. https://doi.org/10.1016/j.paerosci.2005.02.001
Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysis of computer experiments. Stat Sci 4:409–423
Sadoughi M, Hu C, MacKenzie CA, Eshghi AT, Lee S (2018) Sequential exploration-exploitation with dynamic trade-off for efficient reliability analysis of complex engineered systems. Struct Multidiscipl Optim 57:235–250. https://doi.org/10.1007/s00158-017-1748-7
Shi L, Rasheed K (2008) ASAGA: an adaptive surrogate-assisted genetic algorithm. Paper presented at the Proceedings of the 10th annual conference on Genetic and evolutionary computation, Atlanta, GA, USA
Shi L, Rasheed K (2010) A survey of fitness approximation methods applied in evolutionary algorithms. In: Tenne Y, Goh C-K (eds) Computational intelligence in expensive optimization problems. Springer, Berlin, pp 3–28. https://doi.org/10.1007/978-3-642-10701-6_1
Simpson TW, Poplinski JD, Koch PN, Allen JK (2001) Metamodels for computer-based engineering design: survey and recommendations. Eng Comput 17:129–150. https://doi.org/10.1007/pl00007198
Stern RE, Song J, Work DB (2017) Accelerated Monte Carlo system reliability analysis through machine-learning-based surrogate models of network connectivity. Reliab Eng Syst Safe 164:1–9. https://doi.org/10.1016/j.ress.2017.01.021
Tang Y, Chen J, Wei J (2013) A surrogate-based particle swarm optimization algorithm for solving optimization problems with expensive black box functions. Eng Optim 45:557–576. https://doi.org/10.1080/0305215X.2012.690759
Wang GG, Shan S (2006) Review of metamodeling techniques in support of engineering design optimization. J Mech Des 129:370–380. https://doi.org/10.1115/1.2429697
Wang D, Wu Z, Fei Y, Zhang W (2014) Structural design employing a sequential approximation optimization approach. Comput Struct 134:75–87. https://doi.org/10.1016/j.compstruc.2013.12.004
Wang H, Jin Y, Doherty J (2017) Committee-based active learning for surrogate-assisted particle swarm optimization of expensive problems. IEEE Trans Cybern 47:2664–2677. https://doi.org/10.1109/TCYB.2017.2710978
Wang H, Jin Y, Sun C, Doherty J (2018) Offline data-driven evolutionary optimization using selective surrogate ensembles. IEEE Trans Evol Comput. https://doi.org/10.1109/tevc.2018.2834881
Wu Z, Wang D, Okolo NP, Hu F, Zhang W (2016) Global sensitivity analysis using a gaussian radial basis function metamodel. Reliab Eng Syst Safe 154:171–179. https://doi.org/10.1016/j.ress.2016.06.006
Yang D, Flockton SJ (1995) Evolutionary algorithms with a coarse-to-fine function smoothing. In: IEEE international conference on evolutionary computation. IEEE, pp 657–662
Yang Q, Xue D (2015) Comparative study on influencing factors in adaptive metamodeling. Eng Comput 31:561–577. https://doi.org/10.1007/s00366-014-0358-x
Yang Q, Kianimanesh A, Freiheit T, Park SS, Xue D (2011) A semi-empirical model considering the influence of operating parameters on performance for a direct methanol fuel cell. J Power Sources 196:10640–10651. https://doi.org/10.1016/j.jpowsour.2011.08.104
Yew Soon O, Keane AJ (2004) Meta-Lamarckian learning in memetic algorithms. IEEE Trans Evol Comput 8:99–110. https://doi.org/10.1109/TEVC.2003.819944
Younis A, Dong Z (2010) Trends, features, and tests of common and recently introduced global optimization methods. Eng Optim 42:691–718. https://doi.org/10.1080/03052150903386674
Yu H, Tan Y, Sun C, Zeng J, Jin Y (2016) An adaptive model selection strategy for surrogate-assisted particle swarm optimization algorithm. In: 2016 IEEE symposium series on computational intelligence (SSCI), pp 1–8 https://doi.org/10.1109/ssci.2016.7850208
Yu H, Tan Y, Sun C, Zeng J (2018a) A generation-based optimal restart strategy for surrogate-assisted social learning particle swarm optimization. Knowl Based Syst 10:1–12. https://doi.org/10.1016/j.knosys.2018.08.010
Yu H, Tan Y, Zeng J, Sun C, Jin Y (2018b) Surrogate-assisted hierarchical particle swarm optimization. Inf Sci 454–455:59–72. https://doi.org/10.1016/j.ins.2018.04.062
Acknowledgements
This work is supported by National Natural Science Foundation of China (Grant Nos. 61472269 and 61403272) and the State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, China, as well as the Fund Program for the Scientific Activities of Selected Returned Overseas Professionals in Shanxi Province.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Author Haibo Yu declares that he has no conflict of interest. Author Ying Tan declares that she has no conflict of interest. Author Chaoli Sun declares that she has no conflict of interest. Author Jianchao Zeng declares that he has no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Communicated by V. Loia.
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Yu, H., Tan, Y., Sun, C. et al. A comparison of quality measures for model selection in surrogate-assisted evolutionary algorithm. Soft Comput 23, 12417–12436 (2019). https://doi.org/10.1007/s00500-019-03783-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-019-03783-0