Abstract
Co-allocation architecture was developed to enable parallel transferring of files from multiple replicas stored in the different servers. Several co-allocation strategies have been coupled and used to exploit the different transfer rates among various client-server links and to address dynamic rate fluctuations by dividing files into multiple blocks of equal sizes. The paper presents a dynamic file transfer scheme, called dynamic adjustment strategy (DAS), for co-allocation architecture in concurrently transferring a file from multiple replicas stored in multiple servers within a data grid. The scheme overcomes the obstacle of transfer performance due to idle waiting time of faster servers in co-allocation based file transfers and, therefore, provides reduced file transfer time. A tool with user friendly interface that can be used to manage replicas and downloading in a data grid environment is also described. Experimental results show that our DAS can obtain high-performance file transfer speed and reduce the time cost of reassembling data blocks.
Similar content being viewed by others
References
Allcock B, Tuecke S, Foster I, Chervenak A, Kesselman C (2000) Protocols and services for distributed data-intensive science. In: ACAT2000 proceedings, pp 161–163
Allcock B, Bester J, Bresnahan J, Chervenak A, Foster I, Kesselman C, Meder S, Nefedova V, Quesnel D, Tuecke S (2001) Secure, efficient data transport and replica management for high-performance data-intensive computing. In: Proceedings of the eighteenth IEEE symposium on mass storage systems and technologies, pp 13–28
Allcock B, Bester J, Bresnahan J, Chervenak A, Foster I, Kesselman C, Meder S, Nefedova V, Quesnel D, Tuecke S (2002) Data management and transfer in high-performance computational grid environments. Parallel Comput 28(5):749–771
Chervenak A, Foster I, Kesselman C, Salisbury C, Tuecke S (2001) The data grid: towards an architecture for the distributed management and analysis of large scientific datasets. J Netw Comput Appl 23:187–200
Chervenak A, Deelman E, Foster I, Guy L, Hoschek W, Iamnitchi A, Kesselman C, Kunszt P, Ripeanu M (2002) Giggle: a framework for constructing scalable replica location services. In: Proceedings of supercomputing 2002, Baltimore, MD
Czajkowski K, Foster I, Kesselman C (1999) Resource co-allocation in computational grids. In: Proceedings of the eighth IEEE international symposium on high performance distributed computing (HPDC-8’99), August 1999
Czajkowski K, Fitzgerald S, Foster I, Kesselman C (2001) Grid information services for distributed resource sharing. In: Proceedings of the tenth IEEE international symposium on high-performance distributed computing (HPDC-10’01), August 2001, pp 181–194
Donno F, Gaido L, Ghiselli A, Prelz F, Sgaravatto M (2002) DataGrid Prototype 1. In: Proceedings of the TERENA networking conference, June 2002. http://www.terena.nl/conferences/tnc2002/Papers/p5a2-ghiselli.pdf
Foster I, Kesselman C (1997) Globus: a metacomputing infrastructure toolkit. Int J Supercomput Appl High Perform Comput 11(2):115–128
Foster I, Kesselman C, Tuecke S (2001) The anatomy of the grid: enabling scalable virtual organizations. Int J Supercomput Appl High Perform Comput 15(3):200–222
Hoschek W, Jaen-Martinez J, Samar A, Stockinger H, Stockinger K (2000) Data management in an international data grid project. In: First IEEE/ACM international workshop on grid computing—Grid 2000, Bangalore, India, December 2000
IBM Red Books, Introduction to grid computing with Globus. IBM Press. http://www.redbooks.ibm.com/redbooks/pdfs/sg246895.pdf
Open Grid Forum, http://www.ogf.org/
Stockinger H, Samar A, Allcock B, Foster I, Holtman K, Tierney B (2002) File and object replication in data grids. J Clust Comput 5(3):305–314
SYSSTAT utilities home page, http://perso.wanadoo.fr/sebastien.godard/
The Globus Alliance, http://www.globus.org/
Vazhkudai S (2003) Enabling the co-allocation of grid data transfers. In: Proceedings of fourth international workshop on grid computing, November 2003, pp 41–51
Vazhkudai S, Schopf J (2002) Predicting sporadic grid data transfers. In: Proceedings of 11th IEEE international symposium on high performance distributed computing (HPDC-11 ‘02), July 2002, pp 188–196
Vazhkudai S, Schopf J (2003) Using regression techniques to predict large data transfers. Int J High Perform Comput Appl (IJHPCA) 17:249–268
Vazhkudai S, Schopf J, Foster I (2002) Predicting the performance of wide area data transfers. In: Proceedings of the 16th international parallel and distributed processing symposium (IPDPS 2002), April 2002, pp 34–43
Vazhkudai S, Tuecke S, Foster I (2002) Replica selection in the Globus data grid. In: Proceedings of the 1st international symposium on cluster computing and the grid (CCGRID 2001), May 2001, pp 106–113
Venugopal S, Buyya R, Ramamohanarao K (2006) A taxonomy of data grids for distributed data sharing, management, and processing. ACM Comput Surv 38(1):1–53
Wolski R, Spring N, Hayes J (1999) The network weather service: a distributed resource performance forecasting service for metacomputing. Future Gener Comput Syst 15(5–6):757–768
Yang CT, Shih PC, Chen SY (2006) A domain-based model for efficient network information on grid computing environments. IEICE Trans Inf Syst E89-D(2):738–742. Special issue on parallel/distributed computing and networking
Yang CT, Yang IH, Chen CH, Wang SY (2006) Implementation of a dynamic adjustment mechanism with efficient replica selection in co-allocation data grid environments. In: Proceedings of the 21st annual ACM symposium on applied computing (SAC 2006)—distributed systems and grid computing track, April 23–27, 2006, pp 797–804
Yang CT, Wang SY, Fu CP (2007) A dynamic adjustment mechanism for data transfer in data grids. In: Network and parallel computing: IFIP international conference, NPC 2007, September 17–20. Lecture notes in computer science, vol 4672. Springer, Berlin, pp 61–70
Yang CT, Yang IH, Li KC, Wang SY (2007) Improvements on dynamic adjustment mechanism in co-allocation data grid environments. J Supercomput 40(3):269–280
Zhang X, Freschl J, Schopf J (2003) A performance study of monitoring and information services for distributed systems. In: Proceedings of 12th IEEE international symposium on high performance distributed computing (HPDC-12 ‘03), August 2003, pp 270–282
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Yang, CT., Wang, SY. & Chu, W.CC. Implementation of a dynamic adjustment strategy for parallel file transfer in co-allocation data grids. J Supercomput 54, 180–205 (2010). https://doi.org/10.1007/s11227-009-0307-4
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11227-009-0307-4