A Vision for Managing Extreme-Scale Data Hoards
Scientific data collections grow ever larger, both in terms of the size of individual data items and of the number and complexity of items. To use and manage them, it is important to directly address issues of robust and actionable provenance. We identify three key drivers as our focus: managing the size and complexity of metadata, lack of a priori information to match usage intents between publishers and consumers of data, and support for campaigns over collections of data driven by multi-disciplinary, collaborating teams. We introduce the Hoarde abstraction as an attempt to formalize a way of looking at collections of data to make them more tractable for later use. Hoarde leverages middleware and systems infrastructures for scientific and technical data management. Through the lens of a select group of challenging data usage scenarios, we discuss some of the aspects of implementation, usage, and forward portability of this new view on data management.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1558517
- Resource Relation:
- Conference: International Conference on Distributed Computing Systems (ICDCS 2019) - Dallas, Texas, United States of America - 7/7/2019 8:00:00 AM-7/9/2019 8:00:00 AM
- Country of Publication:
- United States
- Language:
- English
Similar Records
Brick : Metadata schema for portable smart building applications
Complex Network Analysis and Intelligent Monitoring Platform (Final Report)