Skip to main content
SpringerLink
Log in

Analysis of Bayesian Network Learning Techniques for a Hybrid Multi-objective Bayesian Estimation of Distribution Algorithm: a case study on MNK Landscape

  • Published:
Journal of Heuristics Aims and scope Submit manuscript

Abstract

This work investigates different Bayesian network structure learning techniques by thoroughly studying several variants of Hybrid Multi-objective Bayesian Estimation of Distribution Algorithm (HMOBEDA), applied to the MNK Landscape combinatorial problem. In the experiments, we evaluate the performance considering three different aspects: optimization abilities, robustness and learning efficiency. Results for instances of multi- and many-objective MNK-landscape show that, score-based structure learning algorithms appear to be the best choice. In particular, HMOBEDA\(_{k2}\) was capable of producing results comparable with the other variants in terms of the runtime of convergence and the coverage of the final Pareto front, with the additional advantage of providing solutions that are less sensible to noise while the variability of the corresponding Bayesian network models is reduced.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Notes

  1. The number of variables N is noted Q in Sects. 3 and 4.

  2. The source codes are available at https://bitbucket.org/marcella_engcomp/hmobeda.

  3. Usually high values for maximization problems: the ideal point \(Z^*\) is the maximum value of each objective achieved so far.

References

  • Aguirre, H.E., Tanaka, K.: Insights on properties of multiobjective MNK-landscapes. In: Proceedings of the 2004 Congress on Evolutionary Computation, vol. 1, pp. 196–203. IEEE, Portland (2004)

  • Aguirre, H.E., Tanaka, K.: Working principles, behavior, and performance of MOEAs on MNK-landscapes. Eur. J. Oper. Res. 181(3), 1670–1690 (2007)

    Article  Google Scholar 

  • Aragam, B., Gu, J., Zhou, Q.: Learning large-scale Bayesian networks with the sparsebn package. J. Stat. Softw. 91(11), 1–38 (2019)

    Article  Google Scholar 

  • Bengoetxea, E., Larrañaga, P., Bielza, C., Del Pozo, J.F.: Optimal row and column ordering to improve table interpretation using estimation of distribution algorithms. J. Heurist. 17(5), 567–588 (2011)

    Article  Google Scholar 

  • Bennett, K.P., Parrado-Hernández, E.: The interplay of optimization and machine learning research. J. Mach. Learn. Res. 7, 1265–1281 (2006)

    MathSciNet  MATH  Google Scholar 

  • Brownlee, A.E., McCall, J.A., Pelikan, M.: Influence of selection on structure learning in Markov network EDAs: an empirical study. In: Proceedings of the 14th Annual Conference on Genetic and Evolutionary Computation, pp. 249–256 (2012)

  • Buntine, W.: Theory refinement on Bayesian networks. In: Uncertainty Proceedings, pp. 52–60. Elsevier (1991)

  • Colombo, D., Maathuis, M.H.: Order-independent constraint-based causal structure learning. J. Mach. Learn. Res. 15(1), 3741–3782 (2014)

    MathSciNet  MATH  Google Scholar 

  • Cooper, G., Herskovits, E.: A Bayesian method for the induction of probabilistic networks from data. Mach. Learn. 9(4), 309–347 (1992)

    MATH  Google Scholar 

  • Deb, K., Agrawal, S., Pratab, A., Meyarivan, T.: A fast and elitist multi-objective genetic algorithm: NSGA-II. IEEE Trans. Evolut. Comput. 6, 182–197 (2002)

    Article  Google Scholar 

  • Ding, F., Zhuang, Y.: Distributed Bayesian network learning algorithm using storm topology. Int. J. Grid Distrib. Comput. 11(4), 113–126 (2018)

    Article  Google Scholar 

  • de Mattos Neto, P.S., Marinho, M.H., Siqueira, H., de Souza Tadano, Y., Machado, V., AntoniniAlves, T., de Oliveira, J.F.L., Madeiro, F.: A methodology to increase the accuracy of particulate matter predictors based on time decomposition. Sustainability 12(18), 7310 (2020)

    Article  Google Scholar 

  • Echegoyen, C., Lozano, J.A., Santana, R., Larrañaga, P.: Exact Bayesian network learning in estimation of distribution algorithms. In: 2007 IEEE Congress on Evolutionary Computation, pp. 1051–1058. IEEE (2007)

  • Echegoyen, C., Zhang, Q., Mendiburu, A., Santana, R., Lozano, J.A.: On the limits of effectiveness in estimation of distribution algorithms. In: 2011 IEEE Congress of Evolutionary Computation (CEC), pp. 1573–1580. IEEE (2011)

  • Heckerman, D., Geiger, D., Chickering, D.: Learning Bayesian networks: the combination of knowledge and statistical data. Mach. Learn. 20(3), 197–243 (1995)

    MATH  Google Scholar 

  • Henrion, M.: Propagating uncertainty in Bayesian networks by probabilistic logic sampling. In: Machine Intelligence and Pattern Recognition, vol. 5, pp. 149–163. Elsevier (1988)

  • Karshenas, H., Santana, R., Bielza, C., Larrañaga, P.: Multiobjective estimation of distribution algorithm based on joint modeling of objectives and variables. IEEE Trans. Evol. Comput. 18, 519–542 (2014)

    Article  Google Scholar 

  • Kauffman, S.A.: The Origins of Order: Self-organization and Selection in Evolution. Oxford University Press, New York (1993)

    Google Scholar 

  • Khan, N., Goldberg, D.E., Pelikan, M.: Multi-objective Bayesian optimization algorithm. Technical report, University of Illinois at Urbana-Champaign, Illinois Genetic Algorithms Laboratory-Tech Report no.2002009, Urbana, IL (2002)

  • Kollat, J.B., Reed, P., Kasprzyk, J.: A new epsilon-dominance hierarchical Bayesian optimization algorithm for large multiobjective monitoring network design problems. Adv. Water Resour. 31(5), 828–845 (2008)

    Article  Google Scholar 

  • Larrañaga, P., Karshenas, H., Bielza, C., Santana, R.: A review on probabilistic graphical models in evolutionary computation. J. Heuristics 18(5), 795–819 (2012)

    Article  Google Scholar 

  • Larrañaga, P., Lozano, J.A.: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation. Kluwer Academic Publishers, New York (2001)

    MATH  Google Scholar 

  • Lauritzen, S.L.: Graphical Models. Oxford Clarendon Press, Oxford (1996)

    MATH  Google Scholar 

  • Lima, C.F., Lobo, F.G., Pelikan, M., Goldberg, D.E.: Model accuracy in the Bayesian optimization algorithm. Soft. Comput. 15(7), 1351–1371 (2011)

    Article  Google Scholar 

  • Marti, L., Garcia, J., Berlanga, A., Molina, J.M.: Model-building algorithms for multiobjective EDAs: directions for improvement. In: IEEE Conference on Evolutionary Computation. CEC’2008, pp. 2843–2850. IEEE, Piscataway, NJ (2008)

  • Martins, M.S., Delgado, M., Lüders, R., Santana, R., Gonçalves, R.A., de Almeida, C.P.: Exploring the probabilistic graphic model of a hybrid multi-objective Bayesian estimation of distribution algorithm. Appl. Soft Comput. (2018). https://doi.org/10.1016/j.asoc.2018.08.039

    Article  Google Scholar 

  • Martins, M.S., Delgado, M.R., Lüders, R., Santana, R., Ricardo, Gonçalves, Almeida, C.P.d.: Probabilistic analysis of Pareto Front approximation for a hybrid multi-objective bayesian estimation of distribution algorithm. In: Proceedings of the 2017 Brazilian Conference on Intelligent Systems, BRACIS’17, pp. 384–389 (2017)

  • Martins, M.S., Delgado, M.R., Santana, R., Lüders, R., Gonçalves, R.A., Almeida, C.P.D.: HMOBEDA: hybrid multi-objective bayesian estimation of distribution algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO’16, pp. 357–364. ACM, New York, NY (2016)

  • Martins, M.S., El Yafrani, M., Santana, R., Delgado, M.R., Lüders, R., Ahiod, B.: On the performance of multi-objective estimation of distribution algorithms for combinatorial problems. In: IEEE Conference on Evolutionary Computation, CEC’18, pp. 1–8. arXiv:1806.09935 (2018)

  • Martins, M.S.R., Delgado, M.R.B.S., Lüders, R., Santana, R., Gonçalves, R.A., de Almeida, C.P.: Hybrid multi-objective Bayesian estimation of distribution algorithm: a comparative analysis for the multi-objective knapsack problem. J. Heuristics 8, 1–23 (2017)

    Google Scholar 

  • Meneghini, I.R., Guimaraes, F.G., Gaspar-Cunha, A.: Competitive coevolutionary algorithm for robust multi-objective optimization: the worst case minimization. In: 2016 IEEE Congress on Evolutionary Computation (CEC), pp. 586–593. IEEE (2016)

  • Moran, S., He, Y., Liu, K.: Choosing the best Bayesian classifier: an empirical study. IAENG Int. J. Comput. Sci. 36(4), 322–331 (2009)

    Google Scholar 

  • Mühlenbein, H., Paab, G.: From Recombination of Genes to the Estimation of Distributions I. Binary Parameters. Parallel Problem Solving from Nature. PPSN IV—Lecture Notes in Computer Science, vol. 1411, pp. 178–187. Springer, London (1996)

  • Pelikan, M., Goldberg, D.E., Tsutsui, S.: Hierarchical bayesian optimization algorithm: toward a new generation of evolutionary algorithms. In: SICE 2003 Annual Conference (IEEE Cat. No. 03TH8734), vol. 3, pp. 2738–2743. IEEE (2003)

  • Pelikan, M., Hauschild, M.W.: Learn from the past: improving model-directed optimization by transfer learning based on distance-based bias. Missouri Estimation of Distribution Algorithms Laboratory, University of Missouri in St. Louis, MO, United States. Technical Report 2012007 (2012)

  • Pelikan, M., Sastry, K., Goldberg, D.E.: iBOA: The incremental Bayesian optimization algorithm. In: Proceedings of the 10th Annual Conference on Genetic and Evolutionary Computation, pp. 455–462 (2008)

  • Puchta, E.D., Lucas, R., Ferreira, F.R., Siqueira, H.V., Kaster, M.S.: Gaussian adaptive PID control optimized via genetic algorithm applied to a step-down dc-dc converter. In: 2016 12th IEEE International Conference on Industry Applications (INDUSCON), pp. 1–6. IEEE (2016)

  • Puchta, E.D., Siqueira, H.V., Kaster, M.D.S.: Optimization tools based on metaheuristics for performance enhancement in a Gaussian adaptive PID controller. IEEE Trans. Cybern. 50(3), 1185–1194 (2020)

    Article  Google Scholar 

  • Ribeiro, V.H.A., Reynoso-Meza, G., Siqueira, H.V.: Multi-objective ensembles of echo state networks and extreme learning machines for streamflow series forecasting. Eng. Appl. Artif. Intell. 95, 103910 (2020)

    Article  Google Scholar 

  • Russell Stuart, J., Norvig, P.: Artificial Intelligence: A Modern Approach. Prentice Hall, Upper Saddle River (2009)

    MATH  Google Scholar 

  • Santana, R., Larrañaga, P., Lozano, J.A.: Combining variable neighborhood search and estimation of distribution algorithms in the protein side chain placement problem. J. Heuristics 14, 519–547 (2008)

    Article  Google Scholar 

  • Santana, R., Mendiburu, A., Lozano, J.A.: Evolving MNK-landscapes with structural constraints. In: IEEE Congress on Evolutionary Computation. CEC’15, pp. 1364–1371. IEEE, Sendai (2015)

  • Santana, R., Mendiburu, A., Lozano, J.A.: Multi-objective NM-landscapes. In: Proceedings of the Companion Publication of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO’15, pp. 1477–1478. ACM, Orlando, FL (2015)

  • Santana Jr., C.J., Macedo, M., Siqueira, H., Gokhale, A., Bastos-Filho, C.J.: A novel binary artificial bee colony algorithm. Future Gen. Comput. Syst. 98, 180–196 (2019)

    Article  Google Scholar 

  • Santhanam, N.P., Wainwright, M.J.: Information-theoretic limits of selecting binary graphical models in high dimensions. IEEE Trans. Inf. Theory 58(7), 4117–4134 (2012)

    Article  MathSciNet  Google Scholar 

  • Santos, P., Macedo, M., Figueiredo, E., Santana, C.J., Soares, F., Siqueira, H., Maciel, A., Gokhale, A., Bastos-Filho, C.J.: Application of pso-based clustering algorithms on educational databases. In: 2017 IEEE Latin American Conference on Computational Intelligence (LA-CCI), pp. 1–6. IEEE (2017)

  • Scanagatta, M., Salmerón, A., Stella, F.: A survey on Bayesian network structure learning from data. In: Progress in Artificial Intelligence, pp. 1–15 (2019)

  • Scutari, M.: Learning Bayesian networks with the bnlearn r package. arXiv:0908.3817 (2009)

  • Scutari, M., Graafland, C.E., Gutiérrez, J.M.: Who learns better Bayesian network structures: constraint-based, score-based or hybrid algorithms? In: International Conference on Probabilistic Graphical Models, pp. 416–427 (2018)

  • Shah, R., Reed, P.: Comparative analysis of multiobjective evolutionary algorithms for random and correlated instances of multiobjective d-dimensional knapsack problems. Eur. J. Oper. Res. 211(3), 466–479 (2011)

    Article  MathSciNet  Google Scholar 

  • Shim, V.A., Tan, K.C., Chia, J.Y., Al Mamun, A.: Multi-objective optimization with estimation of distribution algorithm in a noisy environment. Evol. Comput. 21(1), 149–177 (2013)

    Article  Google Scholar 

  • Srinivas, N., Deb, K.: Multiobjective optimization using nondominated sorting in genetic algorithms. Evol. Comput. 2, 221–248 (1994)

    Article  Google Scholar 

  • Tsagris, M.: Bayesian network learning with the pc algorithm: an improved and correct variation. Appl. Artif. Intell. 33(2), 101–123 (2019)

    Article  MathSciNet  Google Scholar 

  • Tsamardinos, I., Aliferis, C.F., Statnikov, A.R., Statnikov, E.: Algorithms for large scale Markov blanket discovery. In: FLAIRS Conference, vol. 2, pp. 376–380. AAAI Press, St. Augustine, FL (2003)

  • Tsamardinos, I., Brown, L.E., Aliferis, C.F.: The max–min hill-climbing Bayesian network structure learning algorithm. Mach. Learn. 65(1), 31–78 (2006)

    Article  Google Scholar 

  • Viinikka, J., Eggeling, R., Koivisto, M., et al.: Intersection-validation: a method for evaluating structure learning without ground truth. Proc. Mach. Learn. Res. 84, 1570–1578 (2018)

    Google Scholar 

  • Yuan, C., Malone, B.: Learning optimal Bayesian networks: a shortest path perspective. J. Artif. Intell. Res. 48(1), 23–65 (2013)

    Article  MathSciNet  Google Scholar 

  • Zitzler, E., Thiele, L.: Multiple objective evolutionary algorithms: a comparative case study and the strength Pareto approach. IEEE Trans. Evol. Comput. 3, 257–271 (1999)

    Article  Google Scholar 

Download references

Acknowledgements

M. Delgado acknowledges CNPq, grants 309935/2017-2 e 439226/2018-0. R. Santana acknowledges support by the TIN2016-78365-R (Spanish Ministry of Economy, Industry and Competitiveness), PID2019-104966GB-I00 (Spanish Ministry of Science and Innovation), the IT-1244-19 (Basque Government) program and project 3KIA (KK-2020/00049) funded by the SPRI-Basque Government through the ELKARTEK program.

Author information

Authors and Affiliations

  1. Federal University of Technology - Paraná (UTFPR), Curitiba, Brazil

    Marcella S. R. Martins, Myriam Delgado, Ricardo Lüders & Hugo V. Siqueira

  2. Operations Research Group, Aalborg University (AAU), Aalborg, Denmark

    Mohamed El Yafrani

  3. University of the Basque Country (UPV/EHU), San Sebastián, Spain

    Roberto Santana

  4. Akdeniz University (AKU), Antalya, Turkey

    Huseyin G. Akcay

  5. Mohammed V University in Rabat, Rabat, Morocco

    Belaïd Ahiod

Authors
  1. Marcella S. R. Martins
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohamed El Yafrani
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Myriam Delgado
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ricardo Lüders
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Roberto Santana
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Hugo V. Siqueira
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Huseyin G. Akcay
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Belaïd Ahiod
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Marcella S. R. Martins.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Tables 2 and 3 present the respective HV\(^-\) indicator and IGD metric to the approximated Pareto fronts provided by the two populations \(PF_{s}\) and \(PF_{sn}\). The values are averaged over the results of 30 executions of each algorithm. The Mann-Whitney-Wilcoxon test with \(\alpha =5\%\) is applied for the statistical analysis of the results. Values of \(PF_{s}\) and \(PF_{sn}\) for each algorithm and instance with background in light blue have no statistically significant differences. The values in bold correspond to the best values for the paiwise comparison between \(PF_{s}\) and \(PF_{sn}\) for each HMOBEDA variant.

Table 2 Average HV\(^-\) over 30 independent executions for the approximated \(PF_{s}\) and \(PF_{sn}\) for each HMOBEDA variant on each problem instance
Full size table
Table 3 Average IGD over 30 independent executions for the approximated \(PF_{s}\) and \(PF_{sn}\) for each HMOBEDA variant on each problem instance.
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Martins, M.S.R., Yafrani, M.E., Delgado, M. et al. Analysis of Bayesian Network Learning Techniques for a Hybrid Multi-objective Bayesian Estimation of Distribution Algorithm: a case study on MNK Landscape. J Heuristics 27, 549–573 (2021). https://doi.org/10.1007/s10732-021-09469-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10732-021-09469-x

Keywords

Search

Navigation