Abstract
Problem-solving frameworks in large-scale and wide-area environments must handle connectivity issues (NATs and firewalls), maintain scalability with respect to connection management, accommodate dynamic processes joining/leaving at runtime, and provide simple means to tolerate communication/node failures. All of the above must be presented in a simple and flexible programming model. This paper designs and implements such a framework by minimally extending distributed object-oriented models for maximum generality and flexibility. To make parallelism manageable, we introduce an implicit serialization semantics on objects to relieve programmers from explicit synchronization, while avoiding the recursion deadlock problems from which some models based on active objects suffer. We show how this design nicely incorporate dynamically joining processes. In our implementation, participating nodes automatically construct a TCP overlay so as to address connectivity and scalability issues. We have implemented our framework, gluepy as a library for Python. For evaluation, we show on over 900 cores across 9 clusters with complex networks (involving NATs and firewalls) and process managements (involving SSH, torque, and SGE) configurations, how a simple branch-and-bound search application can be expressed simply and executed easily.
Keywords
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
OpenPBS, http://www-unix.mcs.anl.gov/openpbs/
Ayyub, S., Abramson, D., Enticott, C., Garic, S., Tan, J.: Executing Large Parameter Sweep Applications on a Multi-VO Testbed. In: CCGRID 2007: Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, pp. 73–82. IEEE Computer Society, Los Alamitos (2007)
Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the Condor experience. Concurrency - Practice and Experience 17(2-4), 323–356 (2005)
Taverna Project Website, http://taverna.sourceforge.net/
Dean, J., Ghemawat, S.: MapReduce: Simplified Data Processing on Large Clusters. In: OSDI, pp. 137–150 (2004)
Egner, M.T., Lorch, M., Biddle, E.: UIMA Grid: Distributed Large-scale Text Analysis. In: CCGRID 2007: Proceedings of the Seventh IEEE International Symposium on Cluster Computing and the Grid, pp. 317–326. IEEE Computer Society, Los Alamitos (2007)
van Nieuwpoort, R.V., Maassen, J., Wrzesinska, G., Hofman, R., Jacobs, C., Kielmann, T., Bal, H.E.: Ibis: a Flexible and Efficient Java-based Grid Programming Environment. Concurrency and Computation: Practice and Experience 17(7-8), 1079–1107 (2005)
Huet, F., Caromel, D., Bal, H.E.: A High Performance Java Middleware with a Real Application. In: Proceedings of the Supercomputing conference (2004)
Taura, K., Matsuoka, S., Yonezawa, A.: ABCL/f: A Future-Based Polymorphic Typed Concurrent Object-Oriented Language: Its Design and Implementation (1994)
Agha, G.: Actors: a Model of Concurrent Computation in Distributed Systems. MIT Press, Cambridge (1986)
3rd Grid Plugtest Report, http://www-sop.inria.fr/oasis/plugtest2006/plugtests_report_2006.pdf
Aida, K., Osumi, T.: A Case Study in Running a Parallel Branch and Bound Application on the Grid. In: SAINT 2005: Proceedings of the The 2005 Symposium on Applications and the Internet, pp. 164–173. IEEE Computer Society, Los Alamitos (2005)
Aoki, H., Nakada, H., Tanaka, K., Matsuoka, S.: A Programming Environment with Dynamic Node Configuration for Hierarchical Grid: Jojo2. In: SACSIS (2006)
Linderoth, J., Goux, J.P., Yoder, M.: Metacomputing and the Master-Worker Paradigm. Technical Report ANL/MCS-P792-0200, Mathematics and Computer Science Division, Argonne National Laboratory (February 2000)
Nakada, H., Matsuoka, S.: A Java-based programming environment for hierarchical Grid: Jojo. In: CCGRID, pp. 51–58 (2004)
Shudo, K., Tanaka, Y., Sekiguchi, S.: P3: P2P-based Middleware Enabling Transfer and Aggregation of Computational Resources. In: CCGRID 2005: Proceedings of the Fifth IEEE International Symposium on Cluster Computing and the Grid, vol. 1, pp. 259–266. IEEE Computer Society, Los Alamitos (2005)
Wrzesinska, G., van Nieuwpoort, R.V., Maassen, J., Kielmann, T., Bal, H.E.: Fault-tolerant Scheduling of Fine-grained Tasks in Grid Environments. International Journal of High Performance Computing Applications (IJHPCA) 20(1) (2006)
Frigo, M., Leiserson, C.E., Randall, K.H.: The Implementation of the Cilk-5 Multithreaded Language. In: Proceedings of the ACM SIGPLAN 1998 Conference on Programming Language Design and Implementation, pp. 212–223 (June 1998); Proceedings published ACM SIGPLAN Notices, 33(5) (May 1998)
Fuhner, T., Popp, S., Jung, T.: A novel framework for distributing computations dispyte distributing python tasks environment. Journal of Computational Electronics 5(4), 349–352 (2006)
van Nieuwpoort, R.V., Kielmann, T., Bal, H.E.: Efficient Load Balancing for Wide-area Divide-and-conquer Applications. In: PPoPP 2001: Proceedings of the eighth ACM SIGPLAN symposium on Principles and practices of parallel programming, pp. 34–43. ACM, New York (2001)
Denis, A., Pérez, C., Priol, T.: Padicotm: an open integration framework for communication middleware and runtimes. Future Gener. Comput. Syst. 19(4), 575–585 (2003)
Maassen, J., Bal, H.E.: Smartsockets: Solving the Connectivity Problems in Grid Computing. In: HPDC 2007: Proceedings of the 16th international symposium on High performance distributed computing, pp. 1–10. ACM, New York (2007)
Horita, Y., Taura, K., Chikayama, T.: A Scalable and Efficient Self-Organizing Failure Detector for Grid Applications. In: GRID 2005: Proceedings of the 6th IEEE/ACM International Workshop on Grid Computing, pp. 202–210. IEEE Computer Society, Los Alamitos (2005)
Perkins, C.: Ad-hoc On-demand Distance Vector Routing. In: MILCOM 1997 panel on Ad Hoc Networks (November 1997)
Taura, K.: GXP: An Interactive Shell for the Grid Environment. IWIA 00, 59–67 (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2008 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Hironaka, K., Saito, H., Takahashi, K., Taura, K. (2008). gluepy: A Simple Distributed Python Programming Framework for Complex Grid Environments. In: Amaral, J.N. (eds) Languages and Compilers for Parallel Computing. LCPC 2008. Lecture Notes in Computer Science, vol 5335. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-89740-8_17
Download citation
DOI: https://doi.org/10.1007/978-3-540-89740-8_17
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-89739-2
Online ISBN: 978-3-540-89740-8
eBook Packages: Computer ScienceComputer Science (R0)