skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: IOrchestrator: improving the performance of multi-node I/O systems via inter-server coordination

Abstract

A cluster of I/O nodes and a parallel file system are often used to provide high-throughput I/O service to a parallel compute cluster. To exploit I/O parallelism parallel file systems stripe file data across the I/O nodes. While this practice is effective in serving asynchronous requests, it may break individual program's spatial locality, which can seriously degrade I/O performance when the I/O nodes concurrently serve synchronous requests from multiple I/O-intensive programs. In this paper we propose a scheme, Orchestrator, to improve the I/O performance of multi-I/O-node systems by orchestrating I/O services among programs when such inter-I/O-node coordination is dynamically determined to be cost effective. We have implemented IOrchestrator in the PVFS2 parallel file system. Our experiments with representative parallel benchmarks show that IOrchestrator can significantly improve I/O performance - by up to a factor of 2.5 - delivered by a cluster of I/O nodes servicing concurrently-running parallel programs. Notably, we have not observed any scenarios in which the use of IOrchestrator causes significant performance degradation.

Authors:
 [1];  [2];  [3]
  1. Los Alamos National Laboratory
  2. WAYNE STATE UNIV
  3. WAYNE STATE UNIV.
Publication Date:
Research Org.:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1009541
Report Number(s):
LA-UR-10-02601; LA-UR-10-2601
TRN: US201106%%973
DOE Contract Number:  
AC52-06NA25396
Resource Type:
Conference
Resource Relation:
Conference: Supercomputing 2010 ; November 13, 2010 ; New Orleans, LA
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; BENCHMARKS; PERFORMANCE; COMPUTER CODES; PARALLEL PROCESSING; PROGRAMMING

Citation Formats

Davis, Marion Kei, Zhang, Xuechen, and Jiang, Song. IOrchestrator: improving the performance of multi-node I/O systems via inter-server coordination. United States: N. p., 2010. Web.
Davis, Marion Kei, Zhang, Xuechen, & Jiang, Song. IOrchestrator: improving the performance of multi-node I/O systems via inter-server coordination. United States.
Davis, Marion Kei, Zhang, Xuechen, and Jiang, Song. 2010. "IOrchestrator: improving the performance of multi-node I/O systems via inter-server coordination". United States. https://www.osti.gov/servlets/purl/1009541.
@article{osti_1009541,
title = {IOrchestrator: improving the performance of multi-node I/O systems via inter-server coordination},
author = {Davis, Marion Kei and Zhang, Xuechen and Jiang, Song},
abstractNote = {A cluster of I/O nodes and a parallel file system are often used to provide high-throughput I/O service to a parallel compute cluster. To exploit I/O parallelism parallel file systems stripe file data across the I/O nodes. While this practice is effective in serving asynchronous requests, it may break individual program's spatial locality, which can seriously degrade I/O performance when the I/O nodes concurrently serve synchronous requests from multiple I/O-intensive programs. In this paper we propose a scheme, Orchestrator, to improve the I/O performance of multi-I/O-node systems by orchestrating I/O services among programs when such inter-I/O-node coordination is dynamically determined to be cost effective. We have implemented IOrchestrator in the PVFS2 parallel file system. Our experiments with representative parallel benchmarks show that IOrchestrator can significantly improve I/O performance - by up to a factor of 2.5 - delivered by a cluster of I/O nodes servicing concurrently-running parallel programs. Notably, we have not observed any scenarios in which the use of IOrchestrator causes significant performance degradation.},
doi = {},
url = {https://www.osti.gov/biblio/1009541}, journal = {},
number = ,
volume = ,
place = {United States},
year = {Fri Jan 01 00:00:00 EST 2010},
month = {Fri Jan 01 00:00:00 EST 2010}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: