Skip to main content

Advertisement

Springer Nature Link
Log in

On the performance of the Bayesian optimization algorithm with combined scenarios of search algorithms and scoring metrics

  • Published:
Genetic Programming and Evolvable Machines Aims and scope Submit manuscript

Abstract

The Bayesian Optimization Algorithm (BOA) is one of the most prominent Estimation of Distribution Algorithms. It can detect the correlation between multiple variables and extract knowledge on regular patterns in solutions. Bayesian Networks (BNs) are used in BOA to represent the probability distributions of the best individuals. The BN’s construction is challenging since there is a trade-off between acuity and computational cost to generate it. This trade-off is determined by combining a search algorithm (SA) and a scoring metric (SM). The SA is responsible for generating a promising BN and the SM assesses the quality of such networks. Some studies have already analyzed how this relationship affects the learning process of a BN. However, such investigation had not yet been performed to determine the bond linking the selection of SA and SM and the BOA’s output quality. Acting on this research gap, a detailed comparative analysis involving two constructive heuristics and four scoring metrics is presented in this work. The classic version of BOA was applied to discrete and continuous optimization problems using binary and floating-point representations. The scenarios were compared through graphical analyses, statistical metrics, and difference detection tests. The results showed that the selection of SA and SM affects the quality of the BOA results since scoring metrics that penalize complex BN models perform better than metrics that do not consider the complexity of the networks. This study contributes to a discussion on this metaheuristic’s practical use, assisting users with implementation decisions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. The normality analysis was performed using the Shapiro–Wilk [51] test with a \(95\%\) confidence interval and indicated the need for the use of non-parametric tests.

  2. The COCO platform provides benchmark function testbeds and has been used for the Black-Box Optimization Benchmarking (BBOB) workshops of Genetic and Evolutionary Computation Conference (GECCO) since 2009 [65, 66].

  3. Asymptotic convergence of the average fitness to a normal distribution was observed in 10, 15, and 20 runs. The histograms can be found in Ref. [50].

  4. Asymptotic convergence of the average fitness to a normal distribution was observed in 10, 15, and 20 runs. The histograms can be found in Ref. [50].

References

  1. D.E. Goldberg, J.H. Holland, Genetic algorithms and machine learning. Mach. Learn. 3(2–3), 95–99 (1988). https://doi.org/10.1007/bf00113892

    Article  Google Scholar 

  2. A. Gaspar Cunha, R. Takahashi, C.H. Antunes, Manual de computação evolutiva e metaheurística (Imprensa da Universidade de Coimbra, Coimbra, 2012). https://doi.org/10.14195/978-989-26-0583-8

    Book  Google Scholar 

  3. Y. Zhang, S. Wang, G. Ji, A comprehensive survey on particle swarm optimization algorithm and its applications. Math. Probl. Eng. 2015, 1 (2015). https://doi.org/10.1155/2015/931256

    Article  MathSciNet  MATH  Google Scholar 

  4. S. Mahdavi, M.E. Shiri, S. Rahnamayan, Metaheuristics in large-scale global continues optimization: a survey. Inf. Sci. 295, 407–428 (2015). https://doi.org/10.1016/j.ins.2014.10.042

    Article  MathSciNet  Google Scholar 

  5. M. Pelikan, D.E. Goldberg, E. Cantu-Paz, Linkage problem, distribution estimation, and bayesian networks. Evol. Comput. 8(3), 311–340 (2000). https://doi.org/10.1162/106365600750078808

    Article  Google Scholar 

  6. I. Tanev, Genetic programming incorporating biased mutation for evolution and adaptation of snakebot. Genet. Programm. Evolvable Mach. 8(1), 39–59 (2007). https://doi.org/10.1007/s10710-006-9008-4

    Article  Google Scholar 

  7. H. Mühlenbein, The equation for response to selection and its use for prediction. Evol. Comput. 5(3), 303–346 (1997). https://doi.org/10.1162/evco.1997.5.3.303

    Article  Google Scholar 

  8. S. Baluja, Population-based incremental learning: a method for integrating genetic search based function optimization and competitive learning. Technical report, Carnegie Mellon University (1994). https://doi.org/10.5555/865123

  9. J. Smith, On appropriate adaptation levels for the learning of gene linkage. Genet. Program Evolvable Mach. 3(2), 129–155 (2002). https://doi.org/10.1023/A:1015579825262

    Article  MATH  Google Scholar 

  10. M.W. Przewozniczek, M.M. Komarnicki, Empirical linkage learning. IEEE Trans. Evol. Comput. 24(6), 1097–1111 (2020). https://doi.org/10.1109/TEVC.2020.2985497

    Article  Google Scholar 

  11. G.R. Harik, F.G. Lobo, K. Sastry, Linkage learning via probabilistic modeling in the extended compact genetic algorithm (ECGA), in Scalable Optimization via Probabilistic Modeling (Springer, Berlin, Heidelberg, 2006), pp. 39–61. https://doi.org/10.1007/978-3-540-34954-9_3

  12. D.E. Goldberg, The Design of Innovation: Lessons from and for Competent Genetic Algorithms, vol. 7 (Springer, Berlin, 2013). https://doi.org/10.1007/978-1-4757-3643-4

    Book  Google Scholar 

  13. M. Pelikan, D.E. Goldberg, E. Cantú-Paz, et al., Boa: the bayesian optimization algorithm, in Proceedings of the Genetic and Evolutionary Computation Conference GECCO-99, vol. 1 (1999), pp. 525–532. https://doi.org/10.5555/2933923.2933973

  14. M.K. Crocomo, Algoritmo de otimizaçao bayesiano com detecçao de comunidades. PhD thesis, Universidade de São Paulo (2012). https://doi.org/10.11606/T.55.2012.tde-23012013-160605

  15. H.M. Torun, M. Swaminathan, A.K. Davis, M.L.F. Bellaredj, A global bayesian optimization algorithm and its application to integrated system design. IEEE Trans. Very Large Scale Integr. VLSI Syst. 26(4), 792–802 (2018). https://doi.org/10.1109/tvlsi.2017.2784783

    Article  Google Scholar 

  16. F. He, J. Zhou, Z. Feng, G. Liu, Y. Yang, A hybrid short-term load forecasting model based on variational mode decomposition and long short-term memory networks considering relevant factors with bayesian optimization algorithm. Appl. Energy 237, 103–116 (2019). https://doi.org/10.1016/j.apenergy.2019.01.055

    Article  Google Scholar 

  17. R. Ding, W. Zhou, H. Cheng, A novel hybrid model of wind speed forecasting based on EWT, BiLSTM, SVR optimized by BOA in inner Mongolia, China, in Lecture Notes in Electrical Engineering (Springer, Singapore, 2019), pp. 183–191. https://doi.org/10.1007/978-981-32-9686-2_23

  18. B. Huang, Q. Ding, G. Sun, H. Li, Stock prediction based on bayesian-lstm, in Proceedings of the 2018 10th International Conference on Machine Learning and Computing (2018), pp. 128–133. https://doi.org/10.1145/3195106.3195170

  19. R. Tanaka, H. Iwata, Bayesian optimization for genomic selection: a method for discovering the best genotype among a large number of candidates. Theor. Appl. Genet. 131(1), 93–105 (2017). https://doi.org/10.1007/s00122-017-2988-z

    Article  Google Scholar 

  20. L. Chan, G.R. Hutchison, G.M. Morris, BOKEI: bayesian optimization using knowledge of correlated torsions and expected improvement for conformer generation. Phys. Chem. Chem. Phys. 22(9), 5211–5219 (2020). https://doi.org/10.1039/c9cp06688h

    Article  Google Scholar 

  21. C.W. Ahn, R.S. Ramakrishna, D.E. Goldberg, Real-coded bayesian optimization algorithm: bringing the strength of boa into the continuous world, in Genetic and Evolutionary Computation Conference (Springer, 2004), pp. 840–851. https://doi.org/10.1007/978-3-540-24854-5_86

  22. M. Pelikan, D.E. Goldberg, Hierarchical bayesian optimization algorithm, in Scalable Optimization via Probabilistic Modeling (Springer, 2006), pp. 63–90. https://doi.org/10.1007/b10910

  23. J. Očenášek, J. Schwarz, The parallel bayesian optimization algorithm, in The State of the Art in Computational Intelligence (Springer, 2000), pp. 61–67. https://doi.org/10.1007/978-3-7908-1844-4_11

  24. N. Khan, D.E. Goldberg, M. Pelikan, Multi-objective bayesian optimization algorithm, in Proceedings of the 4th Annual Conference on Genetic and Evolutionary Computation (Citeseer, 2002), pp. 684–684

  25. M. Scanagatta, A. Salmerón, F. Stella, A survey on bayesian network structure learning from data. Prog. Artif. Intell. 8(4), 425–439 (2019). https://doi.org/10.1007/s13748-019-00194-y

    Article  Google Scholar 

  26. I. Tsamardinos, L.E. Brown, C.F. Aliferis, The max-min hill-climbing bayesian network structure learning algorithm. Mach. Learn. 65(1), 31–78 (2006). https://doi.org/10.1007/s10994-006-6889-7

    Article  MATH  Google Scholar 

  27. M. Scutari, C.E. Graafland, J.M. Gutiérrez, Who learns better bayesian network structures: Accuracy and speed of structure learning algorithms. Int. J. Approx. Reason. 115, 235–253 (2019). https://doi.org/10.1016/j.ijar.2019.10.003

    Article  MathSciNet  MATH  Google Scholar 

  28. M. Scutari, An empirical-Bayes score for discrete Bayesian networks, in Proceedings of the Eighth International Conference on Probabilistic Graphical Models ed. by A. Antonucci, G. Corani, C.P. Campos (2016), pp. 438–448

  29. S. Beretta, M. Castelli, I. Gonçalves, R. Henriques, D. Ramazzotti, Learning the structure of bayesian networks: a quantitative assessment of the effect of different algorithmic schemes. Complexity (2018). https://doi.org/10.1155/2018/1591878

    Article  Google Scholar 

  30. M. Pelikan, D.E. Goldberg, F.G. Lobo, A survey of optimization by building and using probabilistic models. Comput. Optim. Appl. 21(1), 5–20 (2002). https://doi.org/10.1023/a:1013500812258

    Article  MathSciNet  MATH  Google Scholar 

  31. M. Pelikan, D.E Goldberg, K. Sastry, et al., Bayesian optimization algorithm, decision graphs, and occam’s razor, in Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), vol. 519526 (2001). https://doi.org/10.5555/2955239.2955319

  32. H. Karshenas, A. Nikanjam, B.H. Helmi, A.T. Rahmani, Combinatorial effects of local structures and scoring metrics in bayesian optimization algorithm, in Proceedings of the first ACM/SIGEVO Summit on Genetic and Evolutionary Computation - GEC ’09 (ACM Press, 2009). https://doi.org/10.1145/1543834.1543870

  33. S. Gheisari, M.R. Meybodi, BNC-PSO: structure learning of bayesian networks by particle swarm optimization. Inf. Sci. 348, 272–289 (2016). https://doi.org/10.1016/j.ins.2016.01.090

    Article  MathSciNet  MATH  Google Scholar 

  34. A.H.M. Soares, Algoritmos de estimação de distribuição baseados em árvores filogenéticas. PhD thesis, Universidade de São Paulo (2014). https://doi.org/10.11606/T.55.2014.tde-25032015-111952

  35. J. Martins, Analysis of Linkage Learning in Evolutionary Optimization. PhD thesis, Universidade de São Paulo, 05 (2015). https://doi.org/10.13140/RG.2.1.4317.2325/1

  36. S. Russell, Artificial Intelligence : A Modern Approach, 3rd edn. (Prentice Hall, Upper Saddle River, 2010), p. 0136042597

    Google Scholar 

  37. F. Glover, Heuristics for integer programming using surrogate constraints. Decis. Sci. 8(1), 156–166 (1977). https://doi.org/10.1111/j.1540-5915.1977.tb01074.x

    Article  Google Scholar 

  38. K.B. Korb, A.E. Nicholson, Bayesian Artificial Intelligence (CRC Press, Boca Raton, 2010). ISBN 9781439815915

  39. H. Akaike, A new look at the statistical model identification. IEEE Trans. Autom. Control 19(6), 716–723 (1974). https://doi.org/10.1109/tac.1974.1100705

    Article  MathSciNet  MATH  Google Scholar 

  40. David Maxwell Chickering, Learning equivalence classes of bayesian-network structures. J. Mach. Learn. Res. 2, 445–498 (2002)

    MathSciNet  MATH  Google Scholar 

  41. C. Echegoyen, J.A. Lozano, R. Santana, P. Larranaga, Exact bayesian network learning in estimation of distribution algorithms, in 2007 IEEE Congress on Evolutionary Computation (IEEE, 2007). https://doi.org/10.1109/cec.2007.4424586

  42. M. Scutari, Learning bayesian networks with the bnlearn r package. J. Stat. Softw. 35(3), 1 (2010). https://doi.org/10.18637/jss.v035.i03

    Article  Google Scholar 

  43. A. Ankan, A. Panda, pgmpy: probabilistic graphical models using python, in Proceedings of the 14th Python in Science Conference (SCIPY 2015) (Citeseer, 2015). https://doi.org/10.25080/Majora-7b98e3ed-001

  44. M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, I.H. Witten, The WEKA data mining software. ACM SIGKDD Explor. Newsl. 11(1), 10 (2009). https://doi.org/10.1145/1656274.1656278

    Article  Google Scholar 

  45. Y. Lavinas, C. Aranha, T. Sakurai, M. Ladeira, Experimental analysis of the tournament size on genetic algorithms, in 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (2018). pp. 3647–3653. https://doi.org/10.1109/SMC.2018.00617

  46. Y. Wang, W. Chen, C. Tellambura, Genetic algorithm based nearly optimal peak reduction tone set selection for adaptive amplitude clipping papr reduction. IEEE Trans. Broadcast. 58(3), 462–471 (2012). https://doi.org/10.1109/TBC.2012.2191029

    Article  Google Scholar 

  47. H. Zhang, F. Liu, Y. Zhou, Z. Zhang, A hybrid method integrating an elite genetic algorithm with tabu search for the quadratic assignment problem. Inf. Sci. 539, 347–374 (2020). https://doi.org/10.1016/j.ins.2020.06.036

    Article  MathSciNet  MATH  Google Scholar 

  48. G.F. Cooper, E. Herskovits, A bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9(4), 309–347 (1992). https://doi.org/10.1007/bf00994110

    Article  MATH  Google Scholar 

  49. D. Heckerman, D. Geiger, D.M. Chickering, Learning bayesian networks: the combination of knowledge and statistical data. Mach. Learn. 20(3), 197–243 (1995). https://doi.org/10.1016/j.ijar.2019.10.003

    Article  MATH  Google Scholar 

  50. C. Aparecido L. Nametala, W.R. Faria, B.R. Pereira Júnior, On the Performance of the Bayesian Optimization Algorithm with Combined Scenarios of Search Algorithms and Scoring Metrics: R Source Code and Experiment Data (2021). https://doi.org/10.5281/zenodo.4710554

  51. P. Royston, Remark AS r94: a remark on algorithm AS 181: the w-test for normality. Appl. Stat. 44(4), 547 (1995). https://doi.org/10.2307/2986146

    Article  Google Scholar 

  52. D.C. Montgomery, G.C. Runger, Applied Statistics and Probability for Engineers (Wiley, Hoboken, 2010)

    MATH  Google Scholar 

  53. M. Hollander, E. Chicken, D. Wolfe, Nonparametric Statistical Methods (Wiley, Hoboken, 2013), p. 0470387378

    MATH  Google Scholar 

  54. C. Doerr, F. Ye, N. Horesh, H. Wang, O.M. Shir, T. Back, Benchmarking discrete optimization heuristics with iohprofiler. Appl. Soft Comput. 88, 106027 (2020). https://doi.org/10.1016/j.asoc.2019.106027

    Article  Google Scholar 

  55. G.R. Harik, F.G. Lobo, D.E. Goldberg, The compact genetic algorithm. IEEE Trans. Evol. Comput. 3(4), 287–297 (1999). https://doi.org/10.1109/4235.797971

    Article  Google Scholar 

  56. X. Li, K. Deb, Comparing lbest pso niching algorithms using different position update rules, in IEEE Congress on Evolutionary Computation (2010), pp. 1–8. https://doi.org/10.1109/CEC.2010.5586317

  57. M. Kronfeld, A. Zell, Towards scalability in niching methods, in IEEE Congress on Evolutionary Computation (2010), pp. 1–8. https://doi.org/10.1109/CEC.2010.5585916

  58. A. Soares, R. Râbelo, A. Delbem, Optimization based on phylogram analysis. Expert Syst. Appl. 78, 32–50 (2017). https://doi.org/10.1016/j.eswa.2017.02.012

    Article  Google Scholar 

  59. D.E Goldberg, A design approach to problem difficulty, in The Design of Innovation (Springer, 2002), pp. 71–100

  60. C. Qian, C. Bian, W. Jiang, K. Tang, Running time analysis of the (1+1)-ea for onemax and leadingones under bit-wise noise. Algorithmica 81(2), 749–795 (2019). https://doi.org/10.1145/3071178.3071347

    Article  MathSciNet  MATH  Google Scholar 

  61. N. Buskulic, C. Doerr, Maximizing drift is not optimal for solving onemax, in Proceedings of the Genetic and Evolutionary Computation Conference Companion (2019), pp. 425–426. https://doi.org/10.1145/3319619.3321952

  62. S. Strasser, J.W Sheppard, Evaluating factored evolutionary algorithm performance on binary deceptive functions, in 2017 IEEE Symposium Series on Computational Intelligence (SSCI) (IEEE, 2017), pp. 1–8. https://doi.org/10.1109/SSCI.2017.8285227

  63. R. Tinós, S. Yang, A self-organizing random immigrants genetic algorithm for dynamic optimization problems. Genet. Program. Evolvable Mach. 8(3), 255–286 (2007). https://doi.org/10.1007/s10710-007-9024-z

    Article  Google Scholar 

  64. S. Shakya, R. Santana, J.A. Lozano, A markovianity based optimisation algorithm. Genet. Program. Evolvable Mach. 13(2), 159–195 (2012). https://doi.org/10.1007/s10710-011-9149-y

    Article  Google Scholar 

  65. Complex Systems Design Lab (CSDL), Comparing Continuous Optimizers (coco) (CSDL, 2021). https://coco.gforge.inria.fr

  66. GECCO, Gecco: Genetic and Evolutionary Computation Conference (GECCO, 2021). https://dl.acm.org/conference/gecco

  67. M. Hellwig, H.-G. Beyer, Benchmarking evolutionary algorithms for single objective real-valued constrained optimization—a critical review. Swarm Evol. Comput. 44, 927–944 (2019). https://doi.org/10.1016/j.swevo.2018.10.002

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. São Carlos School of Engineering, Universidade de São Paulo (USP), São Paulo, Brazil

    Ciniro A. L. Nametala, Wandry R. Faria & Benvindo R. Pereira Júnior

Authors
  1. Ciniro A. L. Nametala
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Wandry R. Faria
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Benvindo R. Pereira Júnior
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Ciniro A. L. Nametala.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nametala, C.A.L., Faria, W.R. & Pereira Júnior, B.R. On the performance of the Bayesian optimization algorithm with combined scenarios of search algorithms and scoring metrics. Genet Program Evolvable Mach 23, 193–223 (2022). https://doi.org/10.1007/s10710-022-09430-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10710-022-09430-2

Keywords