Skip to main content
SpringerLink
Log in

Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Co-allocation architecture was developed to enable parallel transferring of files from multiple replicas stored in the different servers. Several co-allocation strategies have been coupled and used to exploit the different transfer rates among various client-server links and to address dynamic rate fluctuations by dividing files into multiple blocks of equal sizes. The paper presents a dynamic file transfer scheme, called dynamic adjustment strategy (DAS), for co-allocation architecture in concurrently transferring a file from multiple replicas stored in multiple servers within a data grid. The scheme overcomes the obstacle of transfer performance due to idle waiting time of faster servers in co-allocation based file transfers and, therefore, provides reduced file transfer time. A tool with user friendly interface that can be used to manage replicas and downloading in a data grid environment is also described. Experimental results show that our DAS can obtain high-performance file transfer speed and reduce the time cost of reassembling data blocks.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Allcock B, Tuecke S, Foster I, Chervenak A, Kesselman C (2000) Protocols and services for distributed data-intensive science. In: ACAT2000 proceedings, pp 161–163

  2. Allcock B, Bester J, Bresnahan J, Chervenak A, Foster I, Kesselman C, Meder S, Nefedova V, Quesnel D, Tuecke S (2001) Secure, efficient data transport and replica management for high-performance data-intensive computing. In: Proceedings of the eighteenth IEEE symposium on mass storage systems and technologies, pp 13–28

  3. Allcock B, Bester J, Bresnahan J, Chervenak A, Foster I, Kesselman C, Meder S, Nefedova V, Quesnel D, Tuecke S (2002) Data management and transfer in high-performance computational grid environments. Parallel Comput 28(5):749–771

    Article  Google Scholar 

  4. Chervenak A, Foster I, Kesselman C, Salisbury C, Tuecke S (2001) The data grid: towards an architecture for the distributed management and analysis of large scientific datasets. J Netw Comput Appl 23:187–200

    Article  Google Scholar 

  5. Chervenak A, Deelman E, Foster I, Guy L, Hoschek W, Iamnitchi A, Kesselman C, Kunszt P, Ripeanu M (2002) Giggle: a framework for constructing scalable replica location services. In: Proceedings of supercomputing 2002, Baltimore, MD

  6. Czajkowski K, Foster I, Kesselman C (1999) Resource co-allocation in computational grids. In: Proceedings of the eighth IEEE international symposium on high performance distributed computing (HPDC-8’99), August 1999

  7. Czajkowski K, Fitzgerald S, Foster I, Kesselman C (2001) Grid information services for distributed resource sharing. In: Proceedings of the tenth IEEE international symposium on high-performance distributed computing (HPDC-10’01), August 2001, pp 181–194

  8. Donno F, Gaido L, Ghiselli A, Prelz F, Sgaravatto M (2002) DataGrid Prototype 1. In: Proceedings of the TERENA networking conference, June 2002. http://www.terena.nl/conferences/tnc2002/Papers/p5a2-ghiselli.pdf

  9. Foster I, Kesselman C (1997) Globus: a metacomputing infrastructure toolkit. Int J Supercomput Appl High Perform Comput 11(2):115–128

    Article  Google Scholar 

  10. Foster I, Kesselman C, Tuecke S (2001) The anatomy of the grid: enabling scalable virtual organizations. Int J Supercomput Appl High Perform Comput 15(3):200–222

    Article  Google Scholar 

  11. Hoschek W, Jaen-Martinez J, Samar A, Stockinger H, Stockinger K (2000) Data management in an international data grid project. In: First IEEE/ACM international workshop on grid computing—Grid 2000, Bangalore, India, December 2000

  12. IBM Red Books, Introduction to grid computing with Globus. IBM Press. http://www.redbooks.ibm.com/redbooks/pdfs/sg246895.pdf

  13. Open Grid Forum, http://www.ogf.org/

  14. Stockinger H, Samar A, Allcock B, Foster I, Holtman K, Tierney B (2002) File and object replication in data grids. J Clust Comput 5(3):305–314

    Google Scholar 

  15. SYSSTAT utilities home page, http://perso.wanadoo.fr/sebastien.godard/

  16. The Globus Alliance, http://www.globus.org/

  17. Vazhkudai S (2003) Enabling the co-allocation of grid data transfers. In: Proceedings of fourth international workshop on grid computing, November 2003, pp 41–51

  18. Vazhkudai S, Schopf J (2002) Predicting sporadic grid data transfers. In: Proceedings of 11th IEEE international symposium on high performance distributed computing (HPDC-11 ‘02), July 2002, pp 188–196

  19. Vazhkudai S, Schopf J (2003) Using regression techniques to predict large data transfers. Int J High Perform Comput Appl (IJHPCA) 17:249–268

    Article  Google Scholar 

  20. Vazhkudai S, Schopf J, Foster I (2002) Predicting the performance of wide area data transfers. In: Proceedings of the 16th international parallel and distributed processing symposium (IPDPS 2002), April 2002, pp 34–43

  21. Vazhkudai S, Tuecke S, Foster I (2002) Replica selection in the Globus data grid. In: Proceedings of the 1st international symposium on cluster computing and the grid (CCGRID 2001), May 2001, pp 106–113

  22. Venugopal S, Buyya R, Ramamohanarao K (2006) A taxonomy of data grids for distributed data sharing, management, and processing. ACM Comput Surv 38(1):1–53

    Article  Google Scholar 

  23. Wolski R, Spring N, Hayes J (1999) The network weather service: a distributed resource performance forecasting service for metacomputing. Future Gener Comput Syst 15(5–6):757–768

    Article  Google Scholar 

  24. Yang CT, Shih PC, Chen SY (2006) A domain-based model for efficient network information on grid computing environments. IEICE Trans Inf Syst E89-D(2):738–742. Special issue on parallel/distributed computing and networking

    Article  Google Scholar 

  25. Yang CT, Yang IH, Chen CH, Wang SY (2006) Implementation of a dynamic adjustment mechanism with efficient replica selection in co-allocation data grid environments. In: Proceedings of the 21st annual ACM symposium on applied computing (SAC 2006)—distributed systems and grid computing track, April 23–27, 2006, pp 797–804

  26. Yang CT, Wang SY, Fu CP (2007) A dynamic adjustment mechanism for data transfer in data grids. In: Network and parallel computing: IFIP international conference, NPC 2007, September 17–20. Lecture notes in computer science, vol 4672. Springer, Berlin, pp 61–70

    Google Scholar 

  27. Yang CT, Yang IH, Li KC, Wang SY (2007) Improvements on dynamic adjustment mechanism in co-allocation data grid environments. J Supercomput 40(3):269–280

    Article  MathSciNet  Google Scholar 

  28. Zhang X, Freschl J, Schopf J (2003) A performance study of monitoring and information services for distributed systems. In: Proceedings of 12th IEEE international symposium on high performance distributed computing (HPDC-12 ‘03), August 2003, pp 270–282

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Tunghai University, Taichung, 40704, Taiwan

    Chao-Tung Yang, Shih-Yu Wang & William Cheng-Chung Chu

  2. Communication System Division Service-Oriented Network System Department, Information and Communications Research Laboratories, Industrial Technology Research Institute, Hsinchu, 31040, Taiwan

    Shih-Yu Wang

Authors
  1. Chao-Tung Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shih-Yu Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. William Cheng-Chung Chu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chao-Tung Yang.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Yang, CT., Wang, SY. & Chu, W.CC. Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids. J Supercomput 54, 180–205 (2010). https://doi.org/10.1007/s11227-009-0307-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-009-0307-4

Keywords

Search

Navigation