Abstract
Applications that both access and generate large data sets increasingly draw our attention in high energy physics, astronomy, genomics and other disciplines. The Data Grids, like Gfarm, seek to harness geographically distributed resources for such large-scale data-intensive problems. However, scheduling is a challenging task in this context. In this paper, we discuss the integration of LSF with Gfarm. We will discuss how to enable LSF to support Gfarm applications requiring GSI authentication, the design and implementation of data aware scheduling and data management. The system is able to find data-affinity hosts for Gfarm jobs and to adjust the distribution of the data replicas dynamically according to the job load. Before job running, the system will setup the proper credential for it. Using the LSF scheduler plugin mechanism, we do not need to write a new scheduler from scratch or make a lot of changes to an existing scheduler.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Basney, J., Livny, M.: Managing Network Resources in Condor. In: Proceedings of the Ninth IEEE Symposium on High Performance Distributed Computing (HPDC9), Pittsburgh, Pennsylvania, August 2000, pp. 298–299 (2000)
Zhou, S., Zheng, X., Wang, J., et al.: Utopia: a Load Sharing Facility for Large, Heterogeneous Distributed Computer Systems. Software—Practice And Experience 23(12), 1305–1336 (1993)
James, P.J.: Portable Batch System: Exterernal Reference Specification Altair PBS Pro 5.3 (March 2003), http://www.mta.ca/torch/pdf/pbspro54/pbsproers.pdf
Sun Microsystems, Inc. Sun Grid Engine 5.3 Administration and User’s Guide (April 2002), http://gridengine.sunsource.net/project/gridengine-download/SGE53AdminUserDoc.pdf
Frey, J., Tannenbaum, T., Foster, I., et al.: Condor-G: A Computation Management Agent for Multi-Institutional Grids. Journal of Cluster Computing 5, 237–246 (2002)
Platform Computing Co. Open source metascheduling for Virtual Organizations with the Community Scheduler Framework, CSF (2004), http://www.cs.virginia.edu/~grimshaw/CS851-2004/Platform/CSF_architecture.pdf
MONARC Collaboration. Models of Networked Analysis at Regional Centres for LHC experiments: Phase 2 report. Technical Report CERN/LCB-001, CERN (2000), http://wwwcern.ch/MONARC/
Tatebe, O., Morita, Y., Matsuoka, S., et al.: Grid Datafarm Architecture for Petascale Data Intensive Computing. In: Proceedings of the 2nd IEEE/ACM International Symposium on Cluster Computing and the Grid, pp. 102–110 (2002)
Ranganathan, K., Foster, I.: Decoupling Computation and Data Scheduling in Distributed Data-Intensive Applications. In: Proceedings of 11th IEEE International Symposium on High Performance Distributed Computing (HPDC-11), Edinburgh, Scotland (July 2002), http://www.globus.org/research/papers/decouple.pdf
Schintke, F., Schutt, T., Alexander: A Framework for Self-Optimizing Grids Using P2P Components. In: Proceedings of the 14th International Workshop on Database and Expert Systems Applications, DEXA 2003 (2003), http://www.zib.de/reinefeld/Publications/dexa03.pdf
Blythe, J., Deelman, E., Gil, Y., et al.: The Role of Planning in Grid Computing. In: 13th International Conference on Automated Planning and Scheduling (ICAPS), Trento, Italy (June 2003), http://www.isi.edu/~gil/papers/icaps03-submission.pdf
Sakae, Y., et al.: Preliminary Evaluation of Dynamic Load Balancing Using Loop Re-partitioning on Omni/SCASH. In: The 3rd International Symposium on Cluster Computing and the Grid, Tokyo, Japan, May 2003, pp. 463–471 (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Wei, X., Li, W.W., Tatebe, O., Xu, G., Hu, L., Ju, J. (2005). Integrating Local Job Scheduler – LSFTM with GfarmTM . In: Pan, Y., Chen, D., Guo, M., Cao, J., Dongarra, J. (eds) Parallel and Distributed Processing and Applications. ISPA 2005. Lecture Notes in Computer Science, vol 3758. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11576235_25
Download citation
DOI: https://doi.org/10.1007/11576235_25
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29769-7
Online ISBN: 978-3-540-32100-2
eBook Packages: Computer ScienceComputer Science (R0)