Abstract
This paper shows how Erlang programming language can be used for creating a framework for distributing and coordinating the execution of many task computing problems. The goals of the proposed solution are (1) to disperse the computation into many tasks, (2) to support multiple well-known computation models (such as master-worker, map-reduce, pipeline), (3) to exploit the advantages of Erlang for developing an efficient and scalable framework and (4) to build a system that can scale from small to large number of tasks with minimum effort. We present the results of work on designing, implementing and testing ComputErl framework. The preliminary experiments with benchmarks as well as real scientific applications show promising scalability on a computing cluster.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Cesarini, F., Thompson, S.: Erlang Programming. O’Reilly Media, Sebastopol (2009)
DeCandia, G., Hastorun, D., Jampani, M., Kakulapati, G., Lakshman, A., Pilchin, A., Sivasubramanian, S., Vosshall, P., Vogels, W.: Dynamo: Amazon’s highly available key-value store. In: SOSP 2007: Proceedings of twenty-first ACM SIGOPS Symposium on Operating Systems Principles, vol. 41, pp. 205–220. ACM, New York (2007)
Foster, I.: Many Tasks Computing: What’s in a Name? (July 2008)
Wilde, M., Foster, I., Iskra, K., Beckman, P., Zhang, Z., Espinosa, A., Hategan, M., Clifford, B., Raicu, I.: Parallel scripting for applications at the petascale and beyond. Computer 42(11), 50–60 (2009)
AB Ericsson: OTP Design Principles User’s Guide (February 2010)
Foster, I.: Globus toolkit version 4: Software for service-oriented systems. In: Jin, H., Reed, D., Jiang, W. (eds.) NPC 2005. LNCS, vol. 3779, pp. 2–13. Springer, Heidelberg (2005), http://dx.doi.org/10.1007/11577188_2
Mościcki, J.T.: Diane - distributed analysis environment for grid-enabled simulation and analysis of physics data. In: Nuclear Science Symposium Conference Record, vol. 3, pp. 1617–1620. IEEE, Los Alamitos (2003)
Mościcki, J.T., Brochu, F., Ebke, J., Egede, U., Elmsheuser, J., Harrison, K., Jones, R.W.L., Lee, H.C., Liko, D., Maier, A.: Ganga: a tool for computational-task management and easy access to grid resources. Computer Physics Communications (June 2009)
Cole, M.: Algorithmic Skeletons: Structured Management of Parallel Computation. MIT Press, Pitman (1989)
Shao, G., Berman, F., Wolski, R.: Master/slave computing on the grid. In: Heterogeneous Computing Workshop, pp. 3–16 (2000)
Dean, J., Ghemawat, S.: Mapreduce: simplified data processing on large clusters. Commun. ACM 51(1), 107–113 (2008)
Bryliński, M., Prymula, K., Jurkowski, W., Kochańczyk, M., Stawowczyk, E., Konieczny, L., Roterman, I.: Prediction of functional sites based on the fuzzy oil drop model. PLoS Comput. Biol. 3(5), e94 (2007)
Massie, M.L., Chun, B.N., Culler, D.E.: The Ganglia Distributed Monitoring System: Design, Implementation, and Experience. Parallel Computing 30(7) (July 2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2011 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Ptaszek, M., Malawski, M. (2011). ComputErl – Erlang-Based Framework for Many Task Computing. In: Page, R., Horváth, Z., Zsók, V. (eds) Trends in Functional Programming. TFP 2010. Lecture Notes in Computer Science, vol 6546. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-22941-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-22941-1_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-22940-4
Online ISBN: 978-3-642-22941-1
eBook Packages: Computer ScienceComputer Science (R0)