Abstract
MapReduce is a quite popular paradigm, which allows to no specialized users to use large parallel computational platforms in a transparent way. Hadoop is the most used implementation of this paradigm, and in fact, for a large amount of users the word Hadoop and MapReduce are interchangeable. But, there are other frameworks that implement this programming paradigm, such as MapReduce-MPI. Since, optimization techniques can be greatly beneficiary of this kind of data-intensive computing modeling, in this paper, we analyze the performance effect of developing genetic algorithms (GA) using different frameworks of MapReduce (MRGA). In particular, we implement MRGA using Hadoop and MR-MPI frameworks. We analyze and compare both implementations considering relevant aspects such as efficiency and scalability to solve a large dimension problem. The results show a similar efficiency level between the algorithms but Hadoop presents a better scalability.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Welcome to Apache Hadoop! Technical report, The Apache Software Foundation (2014). http://hadoop.apache.org/
Cano, A., García-Martínez, C., Ventura, S.: Extremely high-dimensional optimization with MapReduce: scaling functions and algorithm. Inf. Sci. 415–416(Supplement C), 110–127 (2017)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: OSDI04: Proceedings of the 6th Conference on Symposium on Operating Systems Design and Implementation. USENIX Association (2004)
Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Ferrucci, F., Salza, P., Sarro, F.: Using Hadoop MapReduce for parallel genetic algorithms: a comparison of the global, grid and Island models. Evol. Comput. 1–33 (2017). https://doi.org/10.1162/evco_a_00213
Garey, M.R., Johnson, D.S.: Computers and Intractability: A Guide to the Theory of NP-Completeness. Freeman, New York (1979)
Di Geronimo, L., Ferrucci, F., Murolo, A., Sarro, F.: A parallel genetic algorithm based on Hadoop MapReduce for the automatic generation of JUnit test suites. In: 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, pp. 785–793, April 2012
Goldberg, D.E.: The Design of Innovation: Lessons from and for Competent Genetic Algorithms. Kluwer Academic Publishers, Dordrecht (2002)
Miller, B., Goldberg, D.: Genetic algorithms, tournament selection, and the effects of noise. Complex Syst. 9, 193–212 (1995)
Pisinger, D.: Core problems in knapsack algorithms. Oper. Res. 47, 570–575 (1999)
Plimpton, S., Devine, K.: MapReduce in MPI for large-scale graph algorithms. Parallel Comput. 37(9), 610–632 (2011)
Talbi, E.: Metaheuristics: From Design to Implementation. Wiley, Hoboken (2009)
Verma, A., Llorà, X., Goldberg, D.E., Campbell, R.: Scaling genetic algorithms using MapReduce. In: ISDA 2009, pp. 13–18 (2009)
Verma, A., Llorà, X., Venkataraman, S., Goldberg, D.E., Campbell, R.: Scaling eCGA model building via data-intensive computing. In: IEEE Congress on Evolutionary Computation, pp. 1–8, July 2010
Acknowledgments
This research has been partially funded by the Spanish MINECO and FEDER projects TIN2014-57341-R, TIN2016-81766-REDT, TIN2017-88213-R, and by Andalucía Tech, Universidad de Málaga. In addition, the research was funded by the National University of La Pampa, the Incentive Program from MINCyT, and CONICET.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Salto, C., Minetti, G., Alba, E., Luque, G. (2018). Developing Genetic Algorithms Using Different MapReduce Frameworks: MPI vs. Hadoop. In: Herrera, F., et al. Advances in Artificial Intelligence. CAEPIA 2018. Lecture Notes in Computer Science(), vol 11160. Springer, Cham. https://doi.org/10.1007/978-3-030-00374-6_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-00374-6_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00373-9
Online ISBN: 978-3-030-00374-6
eBook Packages: Computer ScienceComputer Science (R0)