skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing: ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing

Journal Article · · Concurrency and Computation. Practice and Experience
DOI:https://doi.org/10.1002/cpe.3407· OSTI ID:1559755
 [1];  [1];  [1];  [1];  [1];  [2]; ORCiD logo [3]; ORCiD logo [3];  [1]
  1. Rutgers Univ., Piscataway, NJ (United States)
  2. California Inst. of Technology (CalTech), Pasadena, CA (United States)
  3. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
+ Show Author Affiliations

Managing the large volumes of data produced by emerging scientific and engineering simulations running on leadership-class resources has become a critical challenge. The data have to be extracted off the computing nodes and transported to consumer nodes so that it can be processed, analyzed, visualized, archived, and so on. Several recent research efforts have addressed data-related challenges at different levels. One attractive approach is to offload expensive input/output operations to a smaller set of dedicated computing nodes known as a staging area. However, even using this approach, the data still have to be moved from the staging area to consumer nodes for processing, which continues to be a bottleneck. In this paper, we investigate an alternate approach, namely moving the data-processing code to the staging area instead of moving the data to the data-processing code. Specifically, we describe the ActiveSpaces framework, which provides (1) programming support for defining the data-processing routines to be downloaded to the staging area and (2) runtime mechanisms for transporting codes associated with these routines to the staging area, executing the routines on the nodes that are part of the staging area, and returning the results. We also present an experimental performance evaluation of ActiveSpaces using applications running on the Cray XT5 at Oak Ridge National Laboratory. Finally, we use a coupled fusion application workflow to explore the trade-offs between transporting data and transporting the code required for data processing during coupling, and we characterize sweet spots for each option.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); National Science Foundation (NSF); USDOE Office of Science (SC), Fusion Energy Sciences (FES)
Contributing Organization:
L
Grant/Contract Number:
AC05-00OR22725; SC0007455; FG02-06ER54857; ACI 1339036; ACI 1310283; DMS 1228203; IIP 0758566
OSTI ID:
1559755
Alternate ID(s):
OSTI ID: 1401825
Journal Information:
Concurrency and Computation. Practice and Experience, Vol. 27, Issue 14; ISSN 1532-0626
Publisher:
WileyCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 8 works
Citation information provided by
Web of Science

References (17)

A New Flexible MPI Collective I/O Implementation conference September 2006
Addressing the petascale data challenge using in-situ analytics conference January 2011
Experiments with in-transit processing for data intensive grid workflows conference September 2007
PreDatA – preparatory data analytics on peta-scale machines conference April 2010
GPU Computing journal May 2008
CellSs: a Programming Model for the Cell BE Architecture conference November 2006
An API for Runtime Code Patching journal November 2000
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS journal August 2013
Plasma simulation studies using multilevel physics models journal May 1999
DataStager: scalable data staging services for petascale applications conference January 2009
Managing Variability in the IO Performance of Petascale Storage Systems
  • Lofstead, Jay; Zheng, Fang; Liu, Qing
  • 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.32
conference November 2010
Numerical study of neoclassical plasma pedestal in a tokamak geometry journal May 2004
DataSpaces: an interaction and coordination framework for coupled simulation workflows journal February 2011
Evaluation of active storage strategies for the lustre parallel file system conference January 2007
DataSpaces: an interaction and coordination framework for coupled simulation workflows conference January 2010
Active disks for large-scale data processing journal June 2001
Enabling high-speed asynchronous data extraction and transfer using DART journal January 2010

Figures / Tables (22)

Figure 1(p. 4)figure Figure 1
Figure 2(p. 6)figure Figure 2
Listing 1(p. 8)figure Listing 1
Listing 2(p. 8)figure Listing 2
Listing 3(p. 10)figure Listing 3
Listing 4(p. 10)figure Listing 4
Figure 3(p. 12)figure Figure 3
Figure 4(p. 13)figure Figure 4
Figure 5(p. 15)figure Figure 5
Figure 6(p. 15)figure Figure 6
Table I(p. 15)table Table I
Figure 7(p. 17)figure Figure 7
Figure 8(p. 17)figure Figure 8
Figure 10(p. 18)figure Figure 10
Figure 9(p. 18)figure Figure 9
Figure 11(p. 19)figure Figure 11
Figure 12(p. 19)figure Figure 12
Figure 13(p. 20)figure Figure 13
Figure 14(p. 21)figure Figure 14
Figure 15(p. 22)figure Figure 15
Figure 16(p. 23)figure Figure 16
Figure 17(p. 23)figure Figure 17

Similar Records

Specification of Fenix MPI Fault Tolerance library (V. 0.9)
Technical Report · Tue Jun 21 00:00:00 EDT 2016 · OSTI ID:1559755
+1 more
In-Staging Data Placement for Asynchronous Coupling of Task-Based Scientific Workflows, In: 2016 Second International Workshop on Extreme Scale Programming Models and Middlewar (ESPM2)
Conference · Fri Jan 01 00:00:00 EST 2016 · PROCEEDINGS OF SECOND INTERNATIONAL WORKSHOP ON EXTREME SCALE PROGRAMMING MODELS AND MIDDLEWARE (ESPM2 2016) · OSTI ID:1559755
+4 more
Enabling In-situ Execution of Coupled Scientific Workflow on Multi-core Platform
Conference · Thu Aug 16 00:00:00 EDT 2012 · 2012 IEEE 26TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS) · OSTI ID:1559755
+3 more

Related Subjects

97 MATHEMATICS AND COMPUTING