Skip to main content
SpringerLink
Log in

A Review on Parallel Estimation of Distribution Algorithms

  • Chapter
Book cover Parallel and Distributed Computational Intelligence

Part of the book series: Studies in Computational Intelligence ((SCI,volume 269))

Abstract

Estimation of Distribution Algorithms (EDAs) are a set of techniques that belong to the field of Evolutionary Computation. They are similar to Genetic Algorithms (GAs), in the sense that, given a problem, they use a population of individuals to represent solutions, and this population is made to evolve towards the most promising solutions. However, instead of using the usual GA-operators such as mutation or crossover, EDAs learn a probabilistic model that tries to capture the main characteristics of the problem. Based on this idea, several EDAs have been introduced in the last years, showing a good performance and being able to solve problems of different complexity. One important drawback of EDAs is the significant computational effort required by the utilization of probabilistic models, when applied to real-world problems. This fact has led the research community to apply parallel schemes to EDAs, as a viable way to reduce execution times. Schemes already proposed for GAs have been used as the foundation for these parallel schemes. In this chapter, we make a review of parallel EDAs, with a main focus: identifying those parts that are susceptible of parallelization. Then we describe a collection of parallelization strategies proposed in the literature. Additionally, we provide some recommendations for those that are considering the implementation of parallel EDAs on state-of-the-art parallel computers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ahn, C.W., Goldberg, D.E., Ramakrishna, R.: Multiple-deme parallel estimation of distribution algorithms: Basic framework and application. In: Wyrzykowski, R., Dongarra, J., Paprzycki, M., Waśniewski, J. (eds.) PPAM 2004. LNCS, vol. 3019, pp. 544–551. Springer, Heidelberg (2004)

    Google Scholar 

  2. Alba, E.: Parallel Metaheuristics: A New Class of Algorithms. John Wiley & Sons Inc., Chichester (2005)

    MATH  Google Scholar 

  3. Alba, E., Cotta, C., Troya, J.: Numerical and real-time analysis of parallel distributed GAs with structured and panmictic populations. In: Proceedings of the IEEE Conference on Evolutionary Computing (CEC), vol. 2, pp. 1019–1026 (1999)

    Google Scholar 

  4. Alba, E., Tomassini, M.: Paralelism and evolutionary algorithms. IEEE Transactions on Evolutionary Computation 6(5), 443–462 (2002)

    Article  Google Scholar 

  5. Alba, E., Troya, J.M.: An analysis of synchronous and asynchronous parallel distributed genetic algorithms with structured and panmictic islands. In: Rolim, J.D.P., Mueller, F., Zomaya, A.Y., Erçal, F., Olariu, S., Ravindran, B., Gustafsson, J., Takada, H., Olsson, R.A., Kalé, L.V., Beckman, P.H., Haines, M., ElGindy, H.A., Caromel, D., Chaumette, S., Fox, G., Pan, Y., Li, K., Yang, T., Ghiola, G., Conte, G., Mancini, L.V., Méry, D., Sanders, B.A., Bhatt, D., Prasanna, V.K. (eds.) IPPS-WS 1999 and SPDP-WS 1999. LNCS, vol. 1586, pp. 248–256. Springer, Heidelberg (1999)

    Google Scholar 

  6. Bosman, P.A.N.: Design and application of iterated density-estimation evolutionary algorithms. Ph.D. thesis, Utrech University (2003)

    Google Scholar 

  7. Bosman, P.A.N., Thierens, D.: An algorithmic framework for density estimation based evolutionary algorithms. Tech. Rep. UU-CS-1999-46, Utrech University (1999)

    Google Scholar 

  8. Bosman, P.A.N., Thierens, D.: IDEAs bases on the normal kernels probability density function. Tech. Rep. UU-CS-2000-11, Utrech University (2000)

    Google Scholar 

  9. Bossert, W.: Mathematical optimization: Are there abstract limits on natural selection? In: Moorehead, P.S., Kaplan, M.M. (eds.) Mathematical Challenges to the Neo-Darwinian Interpretation of Evolution, pp. 35–46. The Wistar Institute Press, Philadelphia (1967)

    Google Scholar 

  10. Buntine, W.: Theory refinement in Bayesian networks. In: Proceedings of the Seventh Conference on Uncertainty in Artificial Intelligence, pp. 52–60 (1991)

    Google Scholar 

  11. Butenhof, D.R.: Programming with POSIX Threads. Addison-Wesley Professional Computing Series (1997)

    Google Scholar 

  12. Cantú-Paz, E.: Efficient and accurate parallel genetic algorithms. Kluwer Academic Publishers, Dordrecht (2000)

    MATH  Google Scholar 

  13. Castillo, E., Gutiérrez, J.M., Hadi, A.S.: Expert Systems and Probabilistic Network Models. Springer, New York (1997)

    Google Scholar 

  14. Chapman, B., Jost, G., van der Pas, R.: Using OpenMP: Portable Shared Memory Parallel Programming (Scientific and Engineering Computation). MIT Press, Cambridge (2007)

    Google Scholar 

  15. Cooper, G.F., Herskovits, E.A.: A Bayesian method for the induction of probabilistic networks from data. Machine Learning 9, 309–347 (1992)

    MATH  Google Scholar 

  16. De Bonet, J.S., Isbell, C.L., Viola, P.: MIMIC: Finding optima by estimating probability densities. In: Mozer, M., Jordan, M., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9 (1997)

    Google Scholar 

  17. Fogel, L.J.: Autonomous automata. Ind. Res. 4, 14–19 (1962)

    Google Scholar 

  18. Goldberg, D.E.: Genetic Algorithms in Search, Optimization, and Machine Learning. Addison/Wesley, Reading (1989)

    MATH  Google Scholar 

  19. Grefenstette, J.J.: Optimization of control parameters for genetic algorithms. IEEE Transactions on Systems, Man, and Cybernetics 16(1), 122–128 (1986)

    Article  Google Scholar 

  20. Harik, G.R., Lobo, F.G., Goldberg, D.E.: The compact Genetic Algorithm. IEEE Transactions on Evolutionary Computation 3(4), 287–297 (1999)

    Article  Google Scholar 

  21. Harik, G.R., Lobo, F.G., Sastry, K.: Linkage learning via probabilistic modeling in the extended compact genetic algorithm (ecga). In: Pelikan, M., Sastry, K., Cantú-Paz, E. (eds.) Scalable Optimization via Probabilistic Modeling. Studies in Computational Intelligence, vol. 33, pp. 39–61. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  22. Heckerman, D., Geiger, D., Chickering, D.M.: Learning Bayesian networks: The combination of knowledge and statistical data. Machine Learning 20, 197–243 (1995)

    MATH  Google Scholar 

  23. Henrion, M.: Propagating uncertainty in Bayesian networks by probabilistic logic sampling. In: Lemmer, J.F., Kanal, L.N. (eds.) Uncertainty in Artificial Intelligence, vol. 2, pp. 149–163. North-Holland, Amsterdam (1988)

    Google Scholar 

  24. Hiroyasu, T., Miki, M., Sano, M., Shimosaka, H., Tsutsui, S., Dongarra, J.: Distributed Probabilistic Model-Building Genetic Algorithm. In: Cantú-Paz, E., Foster, J.A., Deb, K., Davis, L., Roy, R., O’Reilly, U.M., Beyer, H.G., Standish, R.K., Kendall, G., Wilson, S.W., Harman, M., Wegener, J., Dasgupta, D., Potter, M.A., Schultz, A.C., Dowsland, K.A., Jonoska, N., Miller, J.F. (eds.) GECCO 2003. LNCS, vol. 2723, pp. 1015–1028. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  25. Holland, J.H.: Adaptation in Natural and Artificial Systems. The University of Michigan Press (1975)

    Google Scholar 

  26. Howard, R., Matheson, J.: Influence diagrams. In: Howard, R., Matheson, J. (eds.) Readings on the Principales and Applications of Decision Analysis, vol. 2, pp. 721–764. Strategic Decision Group, Menlo Park (1981)

    Google Scholar 

  27. Jaros, J., Schwarz, J.: Parallel BMDA with probability model migration. In: Proceeding of 2007 IEEE Congress on Evolutionary Computation, pp. 1059–1066. IEEE Computer Society, Los Alamitos (2007)

    Chapter  Google Scholar 

  28. Koza, J.R.: Genetic Programming: On the Programming of Computers by Means of Natural Selection. MIT Press, Cambridge (1992)

    MATH  Google Scholar 

  29. Larrañaga, P., Etxeberria, R., Lozano, J., Peña, J.: Optimization by learning and simulation of Bayesian and Gaussian networks. Tech. Rep. KZZA-IK-4-99, Department of Computer Science and Artificial Intelligence, University of the Basque Country (1999)

    Google Scholar 

  30. Larrañaga, P., Etxeberria, R., Lozano, J., Peña, J.: Optimization in continuous domains by learning and simulation of Gaussian networks. In: Proceedings of the Workshop in Optimization by Building and using Probabilistic Models. A Workshop within the 2000 Genetic and Evolutionary Computation Conference, GECCO 2000, Las Vegas, Nevada, USA, pp. 201–204 (2000)

    Google Scholar 

  31. Larrañaga, P., Etxeberria, R., Lozano, J.A., Peña, J.M.: Combinatorial optimization by learning and simulation of Bayesian networks. In: Proceedings of the Conference on Uncertainty in Artificial Intelligence, UAI 2000, Stanford, CA, USA, pp. 343–352 (2000)

    Google Scholar 

  32. Larrañaga, P., Lozano, J.A.: Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation. Kluwer Academic Publishers, Dordrecht (2002)

    MATH  Google Scholar 

  33. Larrañaga, P., Lozano, J.A., Bengoetxea, E.: Estimation of Distribution Algorithms based on multivariate normal and Gaussian networks. Tech. Rep. KZZA-IK-1-01, Department of Computer Science and Artificial Intelligence, University of the Basque Country (2001)

    Google Scholar 

  34. Lauritzen, S.L.: Graphical Models. Oxford University Press, Oxford (1996)

    Google Scholar 

  35. Lozano, J., Sagarna, R., Larrañaga, P.: Parallel estimation of distribution algorithms. In: Larrañaga, P., Lozano, J.A. (eds.) Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation, pp. 129–145. Kluwer Academic Publishers, Dordrecht (2002)

    Google Scholar 

  36. Lozano, J.A., Larrañaga, P., Inza, I., Bengoetxea, E.: Towards a New Evolutionary Computation. In: Advances on Estimation of Distribution Algorithms. Studies in Fuzziness and Soft Computing. Springer, New York (2006)

    Google Scholar 

  37. Madera, J., Alba, E., Ochoa, A.: A Parallel Island Model for Estimation of Distribution Algorithms. In: Lozano, J.A., Larrañaga, P., Inza, I., Bengoetxea, E. (eds.) Towards a New Evolutionary Computation. Advances on Estimation of Distribution Algorithms. Studies in Fuzziness and Soft Computing, vol. 192, pp. 159–186. Springer, Heidelberg (2005)

    Google Scholar 

  38. Mendiburu, A., Lozano, J.A., Miguel-Alonso, J.: Parallel implementation of EDAs based on probabilistic graphical models. IEEE Transactions on Evolutionary Computation 9(4), 406–423 (2005)

    Article  Google Scholar 

  39. Message Passing Interface Forum: MPI: A message-passing interface standard. International Journal of Supercomputer Applications (1994)

    Google Scholar 

  40. Mühlenbein, H.: The equation for response to selection and its use for prediction. Evolutionary Computation 5, 303–346 (1998)

    Article  Google Scholar 

  41. Mühlenbein, H., Mahning, T.: FDA - a scalable evolutionary algorithm for the optimization of additively decomposed functions. Evolutionary Computation 7(4), 353–376 (1999)

    Article  Google Scholar 

  42. Mühlenbein, H., Paaß, G.: From recombination of genes to the estimation of distributions i. binary parameters. In: Ebeling, W., Rechenberg, I., Voigt, H.-M., Schwefel, H.-P. (eds.) PPSN 1996. LNCS, vol. 1141, pp. 178–187. Springer, Heidelberg (1996)

    Google Scholar 

  43. Ocenasek, J.: Parallel estimation of distribution algorithms. Ph. D. thesis, Faculty of Information Technology, Brno University of Technology (2002)

    Google Scholar 

  44. Ocenasek, J., Schwarz, J.: Estimation of distribution algorithm for mixed continuous-discrete optimization problems. In: 2nd Euro-International Symposium on Computational Intelligence, pp. 227–232. IOS Press, Kosice (2002)

    Google Scholar 

  45. Ocenasek, J., Schwarz, J., Pelikan, M.: Design of multithreaded estimation of distribution algorithms. In: Cantú-Paz, E., Foster, J.A., Deb, K., Davis, L., Roy, R., O’Reilly, U.-M., Beyer, H.-G., Kendall, G., Wilson, S.W., Harman, M., Wegener, J., Dasgupta, D., Potter, M.A., Schultz, A., Dowsland, K.A., Jonoska, N., Miller, J., Standish, R.K. (eds.) GECCO 2003. LNCS, vol. 2724, pp. 1247–1258. Springer, Heidelberg (2003)

    Chapter  Google Scholar 

  46. De la Ossa, L., Gámez, J.A., Puerta, J.M.: Migration of probability models instead of individuals: An alternative when applying the island model to EDAs. In: Yao, X., Burke, E.K., Lozano, J.A., Smith, J., Merelo-Guervós, J.J., Bullinaria, J.A., Rowe, J.E., Tiňo, P., Kabán, A., Schwefel, H.-P. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 242–252. Springer, Heidelberg (2004)

    Google Scholar 

  47. De la Ossa, L., Gámez, J.A., Puerta, J.M.: Improving model combination through local search in parallel univariate edas. In: Congress on Evolutionary Computation, pp. 1426–1433. IEEE, Los Alamitos (2005)

    Chapter  Google Scholar 

  48. De la Ossa, L., Gámez, J.A., Puerta, J.M.: Initial approaches to the application of islands-based parallel EDAs in continuous domains. In: Skie, T., Yang, C.S. (eds.) ICPP Workshops, pp. 580–587. IEEE Computer Society, Los Alamitos (2005)

    Google Scholar 

  49. Pearl, J.: Probabilistic Reasoning in Intelligent Systems. Morgan Kaufmann, Palo Alto (1988)

    Google Scholar 

  50. Pelikan, M., Goldberg, D.E.: Genetic algorithms, clustering, and the breaking of symmetry. In: Deb, K., Rudolph, G., Lutton, E., Merelo, J.J., Schoenauer, M., Schwefel, H.-P., Yao, X. (eds.) PPSN 2000. LNCS, vol. 1917. Springer, Heidelberg (2000)

    Chapter  Google Scholar 

  51. Pelikan, M., Goldberg, D.E.: Hierarchical problem solving and the Bayesian optimization algorithm. In: Whitley, D., Goldberg, D., Cantú-Paz, E., Spector, L., Parmee, I., Beyer, H.G. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference, vol. 1, pp. 267–274. Morgan Kaufmann Publishers, San Francisco (2000)

    Google Scholar 

  52. Pelikan, M., Goldberg, D.E.: Research on the Bayesian optimization algorithm. In: Wu, A. (ed.) Proceedings of the 2000 Genetic and Evolutionary Computation Conference Workshop Program, vol. 1, pp. 212–215 (2000)

    Google Scholar 

  53. Pelikan, M., Goldberg, D.E., Cantú-Paz, E.: BOA: The Bayesian optimization algorithm. In: Banzhaf, W., Daida, J., Eiben, A.E., Garzon, M.H., Honavar, V., Jakiela, M., Smith, R.E. (eds.) Proceedings of the Genetic and Evolutionary Computation Conference GECCO 1999, Orlando FL, vol. 1, pp. 525–532. Morgan Kaufmann Publishers, San Francisco (1999)

    Google Scholar 

  54. Pelikan, M., Goldberg, D.E., Lobo, F.: A survey of optimization by building and using probabilistic models. Computational Optimization and Applications 21(1), 5–20 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  55. Pelikan, M., Laury Jr., J.D.: Order or not: does parallelization of model building in hboa affect its scalability? In: Lipson, H. (ed.) Genetic and Evolutionary Computation Conference, GECCO 2007, Proceedings, London, England, UK, July 7-11, pp. 555–561. ACM, New York (2007)

    Chapter  Google Scholar 

  56. Pelikan, M., Mühlenbein, H.: The bivariate marginal distribution algorithm. In: Roy, P.K.C.R., Furuhashi, T. (eds.) Advances in Soft Computing-Engineering Design and Manufacturing, pp. 521–535. Springer, London (1999)

    Google Scholar 

  57. Pelikan, M., Sastry, K., Cantú-Paz, E.: Scalable Optimization via Probabilistic Modeling: From Algorithms to Applications. Studies in Computational Intelligence. Springer, New York (2006)

    Book  MATH  Google Scholar 

  58. Rechenberg, I.: Evolutionsstrategie: Optimierung technischer Systeme nach Prinzipien der biologischen Evolution. Frommann–Holzboog, Stuttgart (1973)

    Google Scholar 

  59. Robles, V., Perez, M., Peña, J., Herves, V., Larrañaga, P.: Parallel interval estimation naive bayes. In: Actas de las XIV Jornadas de Paralelismo, pp. 349–353 (2003)

    Google Scholar 

  60. Santana, R.: Estimation of distribution algorithms with Kikuchi approximations. Evolutionary Computation 13(1), 67–97 (2005)

    Article  Google Scholar 

  61. Sastry, K., Goldberg, D.E., Llorà, X.: Towards billion-bit optimization via a parallel estimation of distribution algorithm. In: Lipson, H. (ed.) Genetic and Evolutionary Computation Conference, GECCO 2007, Proceedings, London, England, UK, July 7-11, pp. 577–584. ACM, New York (2007)

    Chapter  Google Scholar 

  62. Schaffer, J.D., Eshelman, L.J.: On crossover as an evolutionarily viable strategy. In: Belew, R.K., Booker, L.B. (eds.) ICGA, pp. 61–68. Morgan Kaufmann, San Francisco (1991)

    Google Scholar 

  63. Schwarz, G.: Estimating the dimension of a model. Annals of Statistics 7(2), 461–464 (1978)

    Article  Google Scholar 

  64. Schwarz, J., Jaros, J., Ocenásek, J.: Migration of probabilistic models for island-based bivariate eda algorithm. In: 2007 Genetic and Evolutionary Computational Conference, vol. I, p. 631. Association for Computing Machinery (2007)

    Google Scholar 

  65. Sebag, M., Ducoulombier, A.: Extending Population-Based Incremental Learning to Continuous Search Spaces. In: Eiben, A.E., Bäck, T., Schoenauer, M., Schwefel, H.-P. (eds.) PPSN 1998. LNCS, vol. 1498, pp. 418–427. Springer, Heidelberg (1998)

    Chapter  Google Scholar 

  66. Shachter, R., Kenley, C.: Gaussian influence diagrams. Management Science 35, 527–550 (1989)

    Article  Google Scholar 

  67. Shakya, S., McCall, J., Brown, D.: Updating the probability vector using MRF technique for a univariate EDA. In: Onaindia, E., Staab, S. (eds.) STAIRS 2004, Proceedings of the Second Starting AI Researchers’ Symposium. Frontiers in Artificial Intelligence and Applications, vol. 109. IOS Press, Valencia (2004)

    Google Scholar 

  68. Shimosaka, H., Hiroyasu, T., Miki, M.: Comparison of pulling back and penalty methods for constraints in BGA. In: Sarker, R., Reynolds, R., Abbass, H., Tan, K.C., McKay, B., Essam, D., Gedeon, T. (eds.) Proceedings of the 2003 Congress on Evolutionary Computation, CEC 2003, pp. 1941–1948. IEEE Press, Canberra (2003)

    Chapter  Google Scholar 

  69. Spirtes, P., Glymour, C., Scheines, R.: Causation, Prediction, and Search. Lecture Notes in Statistics, vol. 81. Springer, Heidelberg (1993)

    MATH  Google Scholar 

  70. Syswerda, G.: Simulated crossover in genetic algorithms. In: Foundations of Genetic Algorithms, vol. 2, pp. 239–255. Morgan Kaufmann, San Mateo (1993)

    Google Scholar 

  71. Whitley, L.D., Rana, S.B., Heckendorn, R.B.: Island model genetic algorithms and linearly separable problems. In: Corne, D., Shapiro, J.L. (eds.) Evolutionary Computing, AISB Workshop. LNCS, vol. 1305, pp. 109–125. Springer, Heidelberg (1997)

    Chapter  Google Scholar 

  72. Zhigljavsky, A.A.: Theory of Global Random Search. Kluwer Academic Publishers, Dordrecht (1991)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Intelligent Systems Group, The University of the Basque Country,  

    Alexander Mendiburu, Jose Miguel-Alonso & Jose A. Lozano

Authors
  1. Alexander Mendiburu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jose Miguel-Alonso
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jose A. Lozano
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. University of Extremadura, C/ Sta Teresa de Jornet, 38, Merida, Spain

    Francisco Fernández de Vega

  2. Yahoo! Inc., 701 First Avenue, 94087, Sunnyvale, CA, USA

    Erick Cantú-Paz

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Mendiburu, A., Miguel-Alonso, J., Lozano, J.A. (2010). A Review on Parallel Estimation of Distribution Algorithms. In: de Vega, F.F., Cantú-Paz, E. (eds) Parallel and Distributed Computational Intelligence. Studies in Computational Intelligence, vol 269. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-10675-0_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-10675-0_7

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-10674-3

  • Online ISBN: 978-3-642-10675-0

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation