Abstract
The Grid communities efforts on managing and transporting data have focused on very large data sets consisting of very large elements.We are interested in leveraging the benefits of solutions such GridFTP, in particular with respect to parallel data transfer and restartability (as well as security, third party control, etc.), for moving large data sets consisting of very large numbers of small objects, e.g., moving a file system subtree. In addition, we require a solution that 1) imposes constant memory overhead on the client and server systems, 2) is independent of the actual transfer mechanism used so we can easily take advantage of advances in technologies for transferring large files, 3) works well even for very large collections of very small files and 4) is a complete solution, i.e., reproduces the directory tree at the server. In this paper, we present Dynamo which is our tool built on GridFTP for transferring directory subtrees. In addition to describing the architecture and implementation of Dynamo, we present performance results on a range of networks showing that we have met our goals.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
W. Allcock, J. Bester, J. Bresnahan, A. Chernevak, I. Foster, C. Kesselman, S. Meder, V. Nefedova, D. Quesnal, and S. Tuecke, “Data management and transfer in high performance computational grid environments,” Journal of Parallel Computing, 2002.
A. Hanushevsky, A. Trunov, and L. Cottrell, “Peer-to-peer computing for secure high performance data copying,” Tech. Rep., Stanford University, 2001.
C. Baru, R. Moore, A. Rajasekar, and M. Wan, “The sdsc storage resource broker,” in Proc. CASCON’98 Conference, Toronto, Canada, November 1998.
W. Hoschek, J. Jaen-Martinez, A. Samar, H. Stockinger,, and K. Stockinger, “Data management in an international data grid project,” in Proceedings of the first IEEE/ACM International Workshop on Grid Computing, Banglore, India, Dec. 2000, Springer Verlag Press.
A. Shoshani, A. Sinm, and J. Gu, “Storage resource managers: Middleware components for grid stroage,” in Tenth Goddard Conference on Mass Storage Systems and Technologies in cooperation with the Nineteenth IEEE Symposium on Mass Storage Systems, B. Kolber and P. C. Hariharanan, Eds., College Park, MD, April 2002, NASA, p. 209.
H. Stockinger, A. Samar, B. Allcock, I. Foster, K. Holtman, and B. Tierney, “File and object replication in data grids,” in Proceedings of the Tenth International Symposium on High Performance Distributed Computing (HPDC-10). Aug. 2001, IEEE Press.
R. Oldfield, “Summary of existing and developing data grids,” White paper for the Remote Data Access groupof the Global Grid Forum, Mar. 2001.
H. Patterson, S. Manley, M. Federwisch, D. Hitz, S. Kleiman,, and S. Owara, “Snapmirror: File system based asynchronous mirroring for disaster recovery,” in Proceedings of the FAST 2002 Conference on File and Storage Technologies, Monterey, CA, Jan. 2002.
I. Foster and C. Kesselman, “Globus: A metacomputing infrastructure toolkit,” International Journal of Supercomputer Applications, vol. 11, no. 2, pp. 115–128, 1997.
D. Thain, J. Basney, S.-C. Son, and M. Livny, “The kangaroo approach to data movement on the grid,” in Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing (HPDC10), August 2001.
M. Loukides, System Performance Tuning, Nutshell Handbook Series. O’Reilly & Associates Inc, Dec. 1992, ISBN 0-937175-60-9.
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2002 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Silberstein, M., Factor, M., Lorenz, D. (2002). DYNAMO - DirectorY, Net Archiver and MOver. In: Parashar, M. (eds) Grid Computing — GRID 2002. GRID 2002. Lecture Notes in Computer Science, vol 2536. Springer, Berlin, Heidelberg. https://doi.org/10.1007/3-540-36133-2_23
Download citation
DOI: https://doi.org/10.1007/3-540-36133-2_23
Published:
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-00133-1
Online ISBN: 978-3-540-36133-6
eBook Packages: Springer Book Archive