Cross-geography scientific data transferring trends and behavior
Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1468117
- Resource Relation:
- Conference: 27th International Symposium on High-Performance Parallel and Distributed Computing - Tempe, Arizona, United States of America - 6/12/2018 8:00:00 AM-6/15/2018 8:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Scientific User Behavior and Data-Sharing Trends in a Petascale File System
Characterization and identification of HPC applications at leadership computing facility