Abstract
In this paper we explore the application of a recent breed of distributed systems, graph processing frameworks in particular, to solving complex research problems. These frameworks are designed to take full advantage of today’s abundant resources with their inherent distributed computing functionalities. Abstraction of many technical details, such as networking and coordination of multiple compute nodes is a desirable feature provided by these graph-processing frameworks. While these frameworks are largely used to process and analyse the web graphs and social networks, their capacity is not limited to this direct application. This paper is based on design and implementation of a genetic algorithm (GA) using a graph processing tool, GraphX for the task scheduling problem as a case study. Our experimental results show that GraphX can significantly aid in devising distributed solutions for complex problems.
This is a preview of subscription content, log in via an institution.
Notes
- 1.
GraphX (http://spark.apache.org/graphx/) is an open source implementation of the Google Pregel graph processing framework [1], provided as an API of Apache Spark [4].
References
Malewicz, G., Austern, M.H., Bik, A.J., Dehnert, J.C., Horn, I., Leiser, N., Czajkowski, G.: Pregel: a system for large-scale graph processing. In: Proceedings of the International Conference on Management of Data (SIGMOD 2010), pp. 135–146 (2010)
Topcuouglu, H., Hariri, S., Wu, M.: Performance-effective and low-complexity task scheduling for heterogeneous computing. IEEE Trans. Parallel Distrib. Syst. 13(3), 260–274 (2012)
Valiant, L.G.: A bridging model for parallel computation. Commun. ACM 33(8), 103–111 (2012)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing (HotCloud 2010) (2010)
Acknowledgement
Young Choon Lee acknowledges the support of the Australian Research Council (ARC) Linkage Grant LP140100980. Abhaya Nayak’s work is partially supported by ARC Discovery Project: DP150104133.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Mishra, S., Lee, Y.C., Nayak, A. (2016). Distributed Genetic Algorithm on GraphX. In: Kang, B.H., Bai, Q. (eds) AI 2016: Advances in Artificial Intelligence. AI 2016. Lecture Notes in Computer Science(), vol 9992. Springer, Cham. https://doi.org/10.1007/978-3-319-50127-7_48
Download citation
DOI: https://doi.org/10.1007/978-3-319-50127-7_48
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-50126-0
Online ISBN: 978-3-319-50127-7
eBook Packages: Computer ScienceComputer Science (R0)