Skip to main content
SpringerLink
Log in

Improving job scheduling performance with parallel access to replicas in Data Grid environment

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Data Grid has evolved to be the solution for data-intensive applications, such as High Energy Physics (HEP), astrophysics, and computational genomics. These applications usually have large input of data to be analyzed and these input data are widely replicated across Data Grid to improve the performance. The job scheduling performance on traditional computing jobs can be studied using queuing theory. However, with the addition of data transfer, the job scheduling performance is too complex to be modeled. In this research, we study the impact of data transfer on the performance of job scheduling in the Data Grid environment. We have proposed a parallel downloading system that supports replicating data fragments and parallel downloading of replicated data fragments, to improve the job scheduling performance. The performance of the parallel downloading system is compared with non-parallel downloading system, using three scheduling heuristics: Shortest Turnaround Time (STT), Least Relative Load (LRL) and Data Present (DP). Our simulation results show that the proposed parallel download approach greatly improves the Data Grid performance for all three scheduling algorithms, in terms of the geometric mean of job turnaround time. The advantage of parallel downloading system is most evident when the Data Grid has relatively low network bandwidth and relatively high computing power.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Foster I, Kesselman C (1999) The grid: blueprint for a new computing infrastructure. Morgan Kaufmann, Los Altos

    Google Scholar 

  2. Johnston WE (2002) The computing and Data Grid approach: infrastructure for distributed science applications. In: Computing and informatics, special issue on grid computing, Winter 2002

  3. Chervenak A, Foster I, Kesselman C, Salisbury C, Tuecke S (2000) The Data Grid: towards an architecture for the distributed management and analysis of large scientific data sets. J Netw Comput Appl 23(3):187–200

    Article  Google Scholar 

  4. Ranganathan K, Foster I (2001) Identifying dynamic replication strategies for high performance Data Grids. In: Proceedings of the second international workshop on grid computing, Denver, CO, November 2001, pp 75–86

  5. Tang M, Lee BS, Yeo CK, Tang XY (2005) Dynamic replication algorithms for the multi-tier Data Grid. Future Gener Comput Syst 21(5):775–790. Special issue: Parallel computing technologies

    Article  Google Scholar 

  6. Chang R, Chen P (2007) Complete and fragmented replica selection and retrieval in Data Grids. Future Gener Comput Syst 23(4):536–546

    Article  Google Scholar 

  7. Wang C, Hsu C, Chen H, Wu J (2006) Efficient multi-source data transfer in Data Grids. In: Proceedings of the sixth IEEE international symposium on cluster computing and the grid, May 2006, pp 421–424

  8. Vazhkudai S (2003) Enabling the co-allocation of Grid Data transfers. In: Proceedings of the fourth international workshop on grid computing, November 17, 2003

  9. Tang M, Lee BS, Tang X, Yeo C (2006) The impact of data replication on job scheduling performance in the Data Grid. Future Gener Comput Syst 22(3):254–268

    Article  MATH  Google Scholar 

  10. Rangmathan K, Foster I (2003) Simulation studies of computation and data scheduling algorithms for Data Grids. J Grid Comput, 53–62

  11. Takefusa A, Tatebe O, Matsuoka S, Morita Y (2003) Performance analysis of scheduling and replication algorithms on Grid Datafarm architecture for high-energy physics applications. In: Proceedings of the 12th IEEE international symposium on high performance distributed computing (HPDC’03)

  12. The GridFTP data transfer protocol http://www.globus.org/grid_software/data/gridftp.php

  13. Cirne W, Berman F (2003) When the herd is smart: aggregate behavior in the selection of job request. IEEE Trans Parallel Distrib Syst, 181–192

  14. Discrete Uniform Distribution http://planetmath.org/encyclopedia/DiscreteUniformDistribution.html

  15. Feitelson DG, Rudolph L (1998) Metrics and benchmarking for parallel job scheduling. In: Proceedings of the workshop on job scheduling strategies for parallel processing

  16. Chapin SJ, Cirne W, Feitelson DG, Jones JP, Leutenegger ST, Schwiegelshohn U, Smith W, Talby D (1999) Benchmarks and standards for the evaluation of parallel job schedulers. In: Proceedings of 5th job scheduling strategies for parallel processing, Apr 1999

  17. Zipf GK (1949) Human behavior and the principles of least effort. Addison-Wesley, Reading

    Google Scholar 

  18. Maheswaran M, Ali S, Siegel HJ, Hensgen D, Freund RF (1999) Dynamic mapping of a class of independent tasks onto heterogeneous computing systems. J Parallel Distrib Comput, 107–131

  19. Lee CB, Schwartzman Y, Hardy J, Snavely A (2004) Are user runtime estimates inherently inaccurate? In: Proceedings of 10th job scheduling strategies for parallel processing, Jun 2004

  20. The Pacific Rim Application and Grid Middleware Assembly (PRAGMA) http://www.pragma-grid.net/

  21. Grid Data farm (Gfarm) file system http://datafarm.apgrid.org/

  22. Keiser GE (1989) In: Local area networks. McGraw-Hill, New York, pp 185–188

    Google Scholar 

  23. Christodoulopoulos K, Gkamas V, Varvarigos E (2007) Statistical analysis and modeling of jobs in a grid environment. J Grid Comput, 77–101

  24. Yang C, Yang I, Chen C, Wang S (2006) Implementation of a dynamic adjustment mechanism with efficient replica selection in data grid environments. In: Proceedings of the 2006 ACM symposium on applied computing, Dijon, France, April 23–27, 2006, pp 797–804

  25. Yang C, Yang I, Li K, Wang S (2007) Improvements on dynamic adjustment mechanism in co-allocation data grid environments. J Super Comput, 269–280

  26. Yang C, Yang M, Chiang W (2008) Implementation of a cyber transformer for parallel download in co-allocation data grid environments. In: Seventh international conference on grid and cooperative computing (GCC’08), Shenzhen, 24–26 Oct 2008, pp 242–253

  27. Yang C, Yang I, Chiang W (2009) Enhancement of anticipative recursively adjusting mechanism for redundant parallel file transfer in data grids. J Netw Comput Appl 32(4):834–845

    Article  Google Scholar 

  28. Yang C, Wang S, Chu C (2009) Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids. J Supercomput, Online first, 17 June 2009

  29. Chang R, Guo M, Lin H (2008) A multiple parallel download scheme with server throughput and client bandwidth considerations for data grids. Future Gener Comput Syst, 798–805

  30. Chang R, Lin C, Ruey J, Hsi S (2008) An efficient and bandwidth sensitive parallel download scheme in data grids. In: 3rd international conference on communication systems software and middleware and workshops (COMSWARE 2008), 6–10 Jan 2008, pp 296–301

  31. Chang R, Lin C, Hsi S (2010) Accessing data from many servers simultaneously and adaptively in data grids. Future Gener Comput Syst 26(1):63–71

    Article  Google Scholar 

  32. Tikar S, Vadhiyar S (2008) Efficient reuse of replicated parallel data segments in computational grids. Future Gener Comput Syst 24(7):644–657

    Article  Google Scholar 

  33. Ranganathan K, Foster I (2002) Decoupling computation and data scheduling in distributed data-intensive applications. In: Proceedings of 11th IEEE international symposium on high performance distributed computing, Edinburgh, Scotland, July 2002

  34. Rangmathan K, Foster I (2003) Simulation studies of computation and data scheduling algorithms for Data Grids. J Grid Comput 1:53–62

    Article  Google Scholar 

  35. Cameron DG, Millar AP, Nicholson C, Carvajal-Schiaffino R, Zini F, Stockinger K (2005) Analysis of scheduling and replica optimisation strategies for Data Grids using OptorSim. J Grid Comput

  36. Bell W, Cameron D, Capozza L, Millar P, Stockinger K, Zini F (2003) OptorSim—A grid simulator for studying dynamic data replication strategies. Int J High Perform Comput Appl 17

Download references

Author information

Authors and Affiliations

  1. School of Computer Engineering, Nanyang Technological University, Singapore, Singapore

    Junwei Zhang, Bu-Sung Lee, Xueyan Tang & Chai-Kiat Yeo

Authors
  1. Junwei Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bu-Sung Lee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xueyan Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chai-Kiat Yeo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junwei Zhang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zhang, J., Lee, BS., Tang, X. et al. Improving job scheduling performance with parallel access to replicas in Data Grid environment. J Supercomput 56, 245–269 (2011). https://doi.org/10.1007/s11227-009-0365-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-009-0365-7

Search

Navigation