Advising Big Data Transfer Over Dedicated Connections Based on Profiling Optimization
Big data transfer in next-generation scientific applications is currently commonly carried out over dedicated channels in high-performance networks (HPNs), where transport protocols play a critical role in maximizing application-level throughput. Optimizing the performance of these protocols is challenging: i) transport protocols perform differently in various network environments, and the protocol choice is not straightforward; ii) even for a given protocol in a given environment, different parameter settings of the protocol may lead to significantly different performance and oftentimes the default setting does not yield the best performance. However, it is prohibitively time-consuming to conduct exhaustive transport profiling due to the large parameter space. In this paper, we propose a PRofiling Optimization Based DAta Transfer Advisor (ProbData) to help end users determine the most effective transport method with the most appropriate parameter settings to achieve satisfactory performance for big data transfer over dedicated connections in HPNs. ProbData employs a fast profiling scheme based on the Simultaneous Perturbation Stochastic Approximation algorithm, namely, FastProf, to accelerate the exploration of the optimal operational zones of various transport methods to improve profiling efficiency. We first present a theoretical background of the optimized profiling approach in ProbData and then detail its design and implementation. The advising procedure and performance benefits of FastProf and ProbData are illustrated and evaluated by both extensive emulations based on real-life performance measurements and experiments over various physical connections in existing production HPNs.
- Research Organization:
- Argonne National Lab. (ANL), Argonne, IL (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- Harrisburg University of Science & Technology; National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- Grant/Contract Number:
- AC02-06CH11357; SC0015892; AC05-00OR22725; CNS-1828123
- OSTI ID:
- 1603359
- Alternate ID(s):
- OSTI ID: 1761635
- Journal Information:
- IEEE/ACM Transactions on Networking, Vol. 27, Issue 6; ISSN 1063-6692
- Publisher:
- IEEE - ACMCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Web of Science
Similar Records
Data Transfer Advisor with Transport Profiling Optimization
Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning