skip to main content
10.1145/1048935.1050193acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
Article

The Livny and Plank-Beck Problems: Studies in Data Movement on the Computational Grid

Published: 15 November 2003 Publication History

Abstract

Over the last few years the Grid Computing research community has become interested in developing data intensive applications for the Grid. These applications face significant challenges because their widely distributed nature makes it difficult to access data with reasonable speed. In order to address this problem, we feel that the Grid community needs to develop and explore data movement challenges that represent problems encountered in these applications. In this paper, we will identify two such problems that we have dubbed the Livny Problem and the Plank-Beck Problem. We will also present data movement scheduling techniques that we have developed to address these problems.

References

[1]
{1} CERN homepage. http://public.web.cern.ch/ public/.
[2]
{2} Dr. James Plank's homepage. http://www.cs.utk. edu/~plank/.
[3]
{3} Dr. Micah Beck's homepage. http://www.cs.utk. edu/~mbeck/.
[4]
{4} Dr. Miron Livny's homepage. http://www.cs.wisc. edu/~miron/.
[5]
{5} European Data Grid Project homepage. http:// eu-datagrid.web.cern.ch/eu-datagrid/.
[6]
{6} Globus Data Grid homepage. http://www.globus. org/datagrid/.
[7]
{7} GrADS Project homepage. http://hipersoft.cs. rice.edu/grads/.
[8]
{8} M. Allen, R. Wolski, and J. Plank. Adaptive timeout discovery using the network weather service. In High Performance Distributed Computing, 2002.
[9]
{9} T. Arbogast and Z. Chen. On the implementation of mixed methods as nonconforming methods for second-order elliptic problems. Mathematics of Computation, 64(211):943- 972, 1995.
[10]
{10} S. Atchley, S. Soltesz, J. S. Plank, M. Beck, and T. Moore. Fault-tolerance in the network storage stack. In IEEE Workshop on Fault-Tolerant Parallel and Distributed Systems, Ft. Lauderdale, FL, April 2002.
[11]
{11} C. Baru, R. Moore, A. Rajasekar, and M. Wan. The sdsc storage resource broker, 1998.
[12]
{12} A. Bassi, M. Beck, T. Moore, and J. Plank. The logistical backbone: Scalable infrastructure for global data grids. In Asian Computing Science Conference, 2002.
[13]
{13} M. Beynon, R. Ferreira, T. M. Kurc, A. Sussman, and J. H. Saltz. Datacutter: Middleware for filtering very large scientific datasets on archival storage systems. In IEEE Symposium on Mass Storage Systems, pages 119-134, 2000.
[14]
{14} A. Chervenak, I. Foster, C. Kesselman, C. Salisbury, and S. Tuecke. The data grid: Towards an architecture for the distributed management and analysis of large scientific datasets, 1999.
[15]
{15} Condor home page - http://www.cs.wisc.edu/condor/.
[16]
{16} I. Foster and C. Kesselman. The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufmann Publishers, Inc., 1998.
[17]
{17} J. Plank, M. Beck, and W. Elwasif. IBP: The internet backplane protocol. Technical Report UT-CS-99-426, University of Tennessee, 1999.
[18]
{18} J. Plank, M. Beck, W. Elwasif, T. Moore, M. Swany, and R. Wolski. The internet backplane protocol: Storage in the network, 1999.
[19]
{19} J. S. Plank, S. Atchley, Y. Ding, and M. Beck. Algorithms for high performance, wide-area, distributed file downloads. Technical Report UT-CS-02-485, Department of Computer Science, University of Tennessee, October 2002.
[20]
{20} A. Shoshani, L. M. Bernardo, H. Nordberg, D. Rotem, and A. Sim. Storage managementfor high energy physics applications. In Computing in High Energy Physics, 1998.
[21]
{21} T. Tannenbaum and M. Litzkow. The condor distributed processing system. Dr. Dobbs Journal, February 1995.
[22]
{22} D. Thain, J. Basney, S.-C. Son, and M. Livny. The kangaroo approach to data movement on the grid. In Proceedings of the Tenth IEEE Symposium on High Performance Distributed Computing (HPDC10), August 2001.
[23]
{23} R. F. V. D. Wijngaart and M. Frumkin. Nas grid benchmarks version 1.0. Technical Report NAS-02-005, NASA Advanced Supercomputing, July 2002.
[24]
{24} R. Wolski. Dynamically forecasting network performance using the network weather service. Cluster Computing, 1:119-132, 1998. also available from http://www.cs.ucsb.edu/~rich/ publications/nws-tr.ps.gz.
[25]
{25} R. Wolski. Experiences with predicting resource performance on-line in computational grid settings. ACM SIGMETRICS Performance EvaluationReview, 30(4), March 2003.
[26]
{26} R. Wolski, N. Spring, and J. Hayes. The network weather service: A distributed resource performance forecasting service for metacomputing. Future Generation Computer Systems, 15(5-6):757-768, October 1999. http://www.cs.ucsb.edu/~rich/ publications/nws-arch.ps.gz.

Cited By

View all
  • (2019)A survey on data storage and placement methodologies for Cloud-Big Data ecosystemJournal of Big Data10.1186/s40537-019-0178-36:1Online publication date: 11-Feb-2019
  • (2015)ATLAS grid workload on NDGF resourcesFuture Generation Computer Systems10.1016/j.future.2014.12.00547:C(31-47)Online publication date: 1-Jun-2015
  • (2012)ATLAS grid workload on NDGF resourcesProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389104(1-11)Online publication date: 10-Nov-2012
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SC '03: Proceedings of the 2003 ACM/IEEE conference on Supercomputing
November 2003
859 pages
ISBN:1581136951
DOI:10.1145/1048935
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 November 2003

Permissions

Request permissions for this article.

Check for updates

Qualifiers

  • Article

Conference

SC '03
Sponsor:

Acceptance Rates

SC '03 Paper Acceptance Rate 60 of 207 submissions, 29%;
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)2
  • Downloads (Last 6 weeks)0
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2019)A survey on data storage and placement methodologies for Cloud-Big Data ecosystemJournal of Big Data10.1186/s40537-019-0178-36:1Online publication date: 11-Feb-2019
  • (2015)ATLAS grid workload on NDGF resourcesFuture Generation Computer Systems10.1016/j.future.2014.12.00547:C(31-47)Online publication date: 1-Jun-2015
  • (2012)ATLAS grid workload on NDGF resourcesProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/2388996.2389104(1-11)Online publication date: 10-Nov-2012
  • (2012)ATLAS grid workload on NDGF resources: Analysis, modeling, and workload generation2012 International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC.2012.21(1-11)Online publication date: Nov-2012
  • (2011)Wide area placement of data replicas for fast and highly available data accessProceedings of the fourth international workshop on Data-intensive distributed computing10.1145/1996014.1996016(1-8)Online publication date: 8-Jun-2011
  • (2010)Transparent on-demand co-allocation data access for gridsInternational Journal of Ad Hoc and Ubiquitous Computing10.1504/IJAHUC.2010.0329975:4(227-234)Online publication date: 1-May-2010
  • (2010)On‐demand data co‐allocation with user‐level cache for gridsConcurrency and Computation: Practice and Experience10.1002/cpe.158722:18(2488-2513)Online publication date: 12-Nov-2010
  • (2009)Memory-Mapped File Approach for On-Demand Data Co-allocation on GridsProceedings of the 2009 9th IEEE/ACM International Symposium on Cluster Computing and the Grid10.1109/CCGRID.2009.22(300-307)Online publication date: 18-May-2009
  • (2009)Workload characterization in a high-energy data grid and impact on resource managementCluster Computing10.1007/s10586-009-0081-312:2(153-173)Online publication date: 1-Jun-2009
  • (2008)File grouping for scientific data managementProceedings of the 17th international symposium on High performance distributed computing10.1145/1383422.1383429(153-164)Online publication date: 23-Jun-2008
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media