Skip to main content
Springer Nature Link
Log in

Exploiting Bivariate Dependencies to Speedup Structure Learning in Bayesian Optimization Algorithm

  • Regular Paper
  • Published:
Journal of Computer Science and Technology Aims and scope Submit manuscript

Abstract

Bayesian optimization algorithm (BOA) is one of the successful and widely used estimation of distribution algorithms (EDAs) which have been employed to solve different optimization problems. In EDAs, a model is learned from the selected population that encodes interactions among problem variables. New individuals are generated by sampling the model and incorporated into the population. Different probabilistic models have been used in EDAs to learn interactions. Bayesian network (BN) is a well-known graphical model which is used in BOA. Learning a proper model in EDAs and particularly in BOA is distinguished as a computationally expensive task. Different methods have been proposed in the literature to improve the complexity of model building in EDAs. This paper employs bivariate dependencies to learn accurate BNs in BOA efficiently. The proposed approach extracts the bivariate dependencies using an appropriate pairwise interaction-detection metric. Due to the static structure of the underlying problems, these dependencies are used in each generation of BOA to learn an accurate network. By using this approach, the computational cost of model building is reduced dramatically. Various optimization problems are selected to be solved by the algorithm. The experimental results show that the proposed approach successfully finds the optimum in problems with different types of interactions efficiently. Significant speedups are observed in the model building procedure as well.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Larrañaga P, Lozano J A. Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation. Kluwer Academic, 2002.

  2. Pelikan M, Goldberg D E, Cantu-Paz E. BOA: The Bayesian optimization algorithm. In Proc. Genetic and Evolutionary Computation Conference, Orlando, Florida, USA, July 13-17, 1999, pp.525–532.

  3. Mühlenbein H, Mahnig T (1999) FDA-A scalable evolutionary algorithm for the optimization of additively decomposed functions. Evolutionary Computation 7(4):353–376

    Article  Google Scholar 

  4. Etxeberria R, Larrañaga P. Global optimization using Bayesian networks. In Proc. the 2nd Symposium on Artificial Intelligence, La Habana, Cuba, July 15-17, 1999, pp.332–339.

  5. Pelikan M. Hierarchical Bayesian Optimization Algorithm: Toward a New Generation of Evolutionary Algorithm. Springer-Verlag, 2005.

  6. Mühlenbein H, Mahnig T. Evolutionary synthesis of Bayesian networks for optimization. In Advances in Evolutionary Synthesis of Intelligent Agents, Honavar V, Patel M, Balakrishnan K (eds.), MIT Press, 2001, pp.429–455.

  7. Soto M, Ochoa A, Acid S, Campos L M. Bayesian evolutionary algorithms based on simplified models. In Proc. the 2nd Symposium on Artificial Intelligence, La Habana, Cuba, July 15-17, 1999, pp.360–367.

  8. Chen T, Tang K, Chen G, Yao X (2010) Analysis of computational time of simple estimation of distribution algorithms. IEEE Transactions on Evolutionary Computation 14(1):1–22

    Article  MATH  Google Scholar 

  9. Sastry K, Pelikan M, Goldberg D E. Efficiency enhancement of estimation of distribution algorithms. In Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications, Pelikan M, Sastry K, Cantu-Paz E (eds.), Springer, 2006, pp.161–185.

  10. Pelikan M, Sastry K, Goldberg DE (2008) Sporadic model building for efficiency enhancement of the hierarchical BOA. Genetic Programming and Evolvable Machines 9(1):53–84

    Article  Google Scholar 

  11. Dong W, Yao X. NichingEDA: Utilizing the diversity inside a population of EDAs for continuous optimization. In Proc. IEEE Congress on Evolutionary Computation, Hong Kong, China, June 1-6, 2008, pp.1260–1267.

  12. Dong W, Chen T, Tino P, Yao X. Scaling up estimation of distribution algorithms for continuous optimization. Arxiv Preprint ArXiv: 1111.2221 v1, 2011.

  13. Dong W, Yao X (2008) Unified eigen analysis on multivariate Gaussian based estimation of distribution algorithms. Information Sciences 178(15):3000–3023

    Article  MathSciNet  Google Scholar 

  14. Luong HN, Nguyen HTT, Ahn CW (2012) Entropy-based efficiency enhancement techniques for evolutionary algorithms. Information Sciences 188(1):100–120

    Article  Google Scholar 

  15. Howard R A, Matheson J E. Influence diagrams. In Readings on the Principles and Applications of Decision Analysis, Sdg Decision Systems, 1981, pp.721–762.

  16. Pearl J. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.

  17. Chickering DM (2002) Learning equivalence classes of Bayesian network structures. J Machine Learning Research 2(3/1):445–498

    MathSciNet  MATH  Google Scholar 

  18. Chickering DM, Heckerman D, Meek C (2004) Large-sample learning of Bayesian networks is NP-hard. J Machine Learning Research 5(12/1):1287–1330

    MathSciNet  MATH  Google Scholar 

  19. Henrion M. Propagation of uncertainty in Bayesian networks by probabilistic logic sampling. In Proc. Uncertainty in Artificial Intelligence, Seattle, USA, July 10-12, 1988, pp.149–163.

  20. Cano R, Sordo C, Gutiérrez J M. Applications of Bayesian networks in meteorology. In Advances in Bayesian Networks, Gámez M S, Salmerón J A (eds.), Springer, 2004, pp.309–327.

  21. Friedman N, Nachman I, Peér D. Learning Bayesian network structure from massive datasets: The “sparse candidate” algorithm. In Proc. Uncertainty in Artificial Intelligence, Stockholm, Sweden, July 30-August 1, 1999, pp.206–215.

  22. Tsamardinos I, Brown LE, Aliferis CF (2006) The max-min hillclimbing Bayesian network structure learning algorithm. Machine Learning 65(1):31–78

    Article  Google Scholar 

  23. Gámez JA, Mateo JL, Puerta JM (2011) Learning Bayesian networks by hill climbing: Efficient methods based on progressive restriction of the neighborhood. Data Mining and Knowledge Discovery 22(1):106–148

    Article  MathSciNet  MATH  Google Scholar 

  24. Yu T L, Goldberg D E, Yassine A, Chen Y P. Genetic algorithm design inspired by organizational theory: Pilot study of a dependency structure matrix driven genetic algorithm. In Proc. Artificial Neural Networks in Engineering, St. Louis, Missouri, USA, November 2-5, 2003, pp.327–332.

  25. Yu TL, Goldberg DE, Sastry K, Lima CF, Pelikan M (2009) Dependency structure matrix, genetic algorithms, and effective recombination. Evolutionary Computation 17(4):595–626

    Article  Google Scholar 

  26. Yassine A, Joglekar N, Braha D, Eppinger S, Whitney D (2003) Information hiding in product development: The design churn effect. Research in Engineering Design 14(3):145–161

    Article  Google Scholar 

  27. Nikanjam A, Sharifi H, Helmi B H, Rahmani A. Enhancing the efficiency of genetic algorithm by identifying linkage groups using DSM clustering. In Proc. IEEE Congress on Evolutionary Computation, Barcelona, Spain, July 18-23, 2010, pp.1–8.

  28. Nikanjam A, Sharifi H, Rahmani AT (2010) Efficient model building in competent genetic algorithms using DSM clustering. AI Communications 24(3):213–231

    MathSciNet  Google Scholar 

  29. Duque T S P C, Goldberg D E. ClusterMI: Building probabilistic models using hierarchical clustering and mutual information. In Exploitation of Linkage Learning in Evolutionary Algorithms, Chen Y P (ed.), Springer, 2010, pp.123–137.

  30. Lu Q, Yao X (2005) Clustering and learning Gaussian distribution for continuous optimization. IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews 35(2):195–204

    Article  Google Scholar 

  31. Aporntewan C, Chongstitvatana P (2007) Building-block identification by simultaneity matrix. Soft Computing 11(6):541–548

    Article  Google Scholar 

  32. Hauschild M, Pelikan M, Lima C F, Sastry K. Analyzing probabilistic models in hierarchical BOA on traps and spin glasses. In Proc. Genetic and Evolutionary Computation Conference, London, England, July 7-11, 2007, pp.523–530.

  33. Goldberg D E, Sastry K. Genetic Algorithms: The Design of Innovation (2nd edition), Springer, 2010.

  34. Munetomo M, Goldberg D E. Identifying linkage groups by nonlinearity/non-monotonicity detection. In Proc. Genetic and Evolutionary Computation Conference, Orlando, Florida, USA, July 13-17, 1999, pp.433–440.

  35. Yu T L. A matrix approach for finding extreme: Problems with modularity, hierarchy and overlap [PhD Thesis]. University of Illinois at Urbana-Champaign, USA, 2006.

  36. Tsuji M, Munetomo M. Linkage analysis in genetic algorithms. In Computational Intelligence Paradigms: Innovative Applications, Springer, 2008, pp.251–279.

  37. Fischer K H, Hertz J A. Spin Glasses. Cambridge University Press, 1991.

  38. Mühlenbein H, Mahnig T, Rodriguez AO (1999) Schemata, distributions and graphical models in evolutionary optimization. J Heuristics 5(2):215–247

    Article  MATH  Google Scholar 

  39. Santana R (2005) Estimation of distribution algorithms with Kikuchi approximations. Evolutionary Computation 13(1):67–97

    Article  Google Scholar 

  40. Monien B, Sudborough IH (1988) Min cut is NP-complete for edge weighted trees. Theoretical Computer Science 58(1–3):209–229

    Article  MathSciNet  MATH  Google Scholar 

  41. Karshenas H, Nikanjam A, Helmi B H, Rahmani A T. Combinatorial effects of local structures and scoring metrics in Bayesian optimization algorithm. In Proc. World Summit on Genetic and Evolutionary Computation, Shanghai, China, June 12-14, 2008, pp.263–270.

  42. Ocenasek J, Schwarz J. The parallel Bayesian optimization algorithm. In Proc. European Symposium on Computational Intelligence, Kosice, Slovak Republic, August 30-September 1, 2000, pp.61–67.

  43. Lima C F, Lobo F G, Pelikan M. From mating pool distributions to model overfitting. In Proc. Genetic and Evolutionary Computation Conference, Atlanta, GA, USA, July 12-16, 2008, pp.431–438.

  44. Ackley D H. An empirical study of bit vector function optimization. In Genetic Algorithms and Simulated Annealing, Davies L (ed.), Morgan Kaufmann, 1987, pp.170–204.

Download references

Author information

Authors and Affiliations

  1. School of Computer Engineering, Iran University of Science and Technology, Tehran, 1684613114, Iran

    Amin Nikanjam & Adel Rahmani

Authors
  1. Amin Nikanjam
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Adel Rahmani
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Amin Nikanjam.

Electronic Supplementary Material

Below is the link to the electronic supplementary material.

(DOC 15.7 kb)

Rights and permissions

Reprints and permissions

About this article

Cite this article

Nikanjam, A., Rahmani, A. Exploiting Bivariate Dependencies to Speedup Structure Learning in Bayesian Optimization Algorithm. J. Comput. Sci. Technol. 27, 1077–1090 (2012). https://doi.org/10.1007/s11390-012-1285-1

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11390-012-1285-1

Keywords