A View from ORNL: Scientific Data Research Opportunities in the Big Data Age

Klasky, Scott; Wolf, Matthew; Ainsworth, Mark; Atkins, Chuck; Choi, Jong Youl; Eisenhauer, Greg; Geveci, Berk; Godoy, William; Kim, Mark B.; Kress, James; Kurc, Tahsin M.; Liu, Qing Gary; Logan, Jeremy; Maccabe, Arthur; Mehta, Kshitij; Ostrouchov, George; Parashar, Manish; Podhorszki, Norbert; Pugmire, Dave; Suchyta, Eric; Wan, Lipeng; Wang, Ruonan

doi:10.1109/ICDCS.2018.00136

Title: A View from ORNL: Scientific Data Research Opportunities in the Big Data Age

Conference · Sun Jul 01 00:00:00 EDT 2018

DOI:https://doi.org/10.1109/ICDCS.2018.00136· OSTI ID:1468120

^[1];

^[1]; Ainsworth, Mark ^[1]; Atkins, Chuck ^[2];

^[1]; Eisenhauer, Greg ^[3]; Geveci, Berk ^[2];

^[1];

^[1]; Kurc, Tahsin M. ^[1]; Liu, Qing Gary ^[1];

^[1];

^[1]; Parashar, Manish ^[4];

^[1];

^[1] more »

ORNL
Kitware
Georgia Institute of Technology, Atlanta
Rutgers University

One of the core issues across computer and computational science today is adapting to, managing, and learning from the influx of "Big Data". In the commercial space, this problem has led to a huge investment in new technologies and capabilities that are well adapted to dealing with the sorts of human-generated logs, videos, texts, and other large-data artifacts that are processed and resulted in an explosion of useful platforms and languages (Hadoop, Spark, Pandas, etc.). However, translating this work from the enterprise space to the computational science and HPC community has proven somewhat difficult, in part because of some of the fundamental differences in type and scale of data and timescales surrounding its generation and use. We describe a forward-looking research and development plan which centers around the concept of making Input/Output (I/O) intelligent for users in the scientific community, whether they are accessing scalable storage or performing in situ workflow tasks. Much of our work is based on our experience with the Adaptable I/O System (ADIOS 1.X), and our next generation version of the software ADIOS 2.X [1].

View Conference

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1468120

Resource Relation:: Conference: IEEE 38th International Conference on Distributed Computing Systems (ICDCS) - Vienna, , Austria - 7/2/2018 4:00:00 AM-7/5/2018 4:00:00 AM

Country of Publication:: United States

Language:: English

References (39)

Optimizing checkpoint data placement with guaranteed burst buffer endurance in large-scale hierarchical storage systems Wan, Lipeng; Cao, Qing; Wang, Feiyi Journal of Parallel and Distributed Computing, Vol. 100 https://doi.org/10.1016/j.jpdc.2016.10.002	journal	February 2017
GoldRush: resource efficient in situ scientific data analytics using fine-grained interference aware execution Zheng, Fang; Yu, Hongfeng; Hantas, Can Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503279	conference	January 2013
Topology Mapping for Blue Gene/L Supercomputer Yu, Hao; Chung, I-hsin; Moreira, Jose ACM/IEEE SC 2006 Conference (SC'06) https://doi.org/10.1109/SC.2006.63	conference	November 2006
The global version of the gyrokinetic turbulence code GENE Görler, T.; Lapillonne, X.; Brunner, S. Journal of Computational Physics, Vol. 230, Issue 18 https://doi.org/10.1016/j.jcp.2011.05.034	journal	August 2011
VTK-m: Accelerating the Visualization Toolkit for Massively Threaded Architectures Moreland, Kenneth; Sewell, Christopher; Usher, William IEEE Computer Graphics and Applications, Vol. 36, Issue 3 https://doi.org/10.1109/MCG.2016.48	journal	May 2016
Pegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems Deelman, Ewa; Singh, Gurmeet; Su, Mei-Hui Scientific Programming, Vol. 13, Issue 3 https://doi.org/10.1155/2005/128026	journal	January 2005
Visualization and Analysis Requirements for In Situ Processing for a Large-Scale Fusion Simulation Code Kress, James; Pugmire, David; Klasky, Scott 2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV) https://doi.org/10.1109/ISAV.2016.014	conference	November 2016
Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS Liu, Qing; Logan, Jeremy; Tian, Yuan Concurrency and Computation: Practice and Experience, Vol. 26, Issue 7 https://doi.org/10.1002/cpe.3125	journal	August 2013
Handling Failures in Parallel Scientific Workflows Using Clouds Costa, Flavio; de Oliveira, Daniel; Ocana, Kary 2012 SC Companion: High-Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis https://doi.org/10.1109/SC.Companion.2012.28	conference	November 2012
Extending Skel to Support the Development and Optimization of Next Generation I/O Systems Logan, Jeremy; Choi, Jong Youl; Wolf, Matthew 2017 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2017.30	conference	September 2017
Gyrokinetic neoclassical study of the bootstrap current in the tokamak edge pedestal with fully non-linear Coulomb collisions Hager, Robert; Chang, C. S. Physics of Plasmas, Vol. 23, Issue 4 https://doi.org/10.1063/1.4945615	journal	April 2016
SODA: Science-Driven Orchestration of Data Analytics Dayal, Jai; Lofstead, Jay; Eisenhauer, Greg 2015 IEEE 11th International Conference on e-Science https://doi.org/10.1109/eScience.2015.59	conference	August 2015
Event-based systems: opportunities and challenges at exascale Eisenhauer, Greg; Wolf, Matthew; Abbasi, Hasan Proceedings of the Third ACM International Conference on Distributed Event-Based Systems - DEBS '09 https://doi.org/10.1145/1619258.1619261	conference	January 2009
Service Augmentation for High End Interactive Data Services Wolf, Matthew; Abbasi, Hasan; Collins, Benjamin 2005 IEEE International Conference on Cluster Computing https://doi.org/10.1109/CLUSTR.2005.347053	conference	September 2005
Landrush: Rethinking In-Situ Analysis for GPGPU Workflows Goswami, Anshuman; Tian, Yuan; Schwan, Karsten 2016 16th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2016.58	conference	May 2016
DataSpaces: an interaction and coordination framework for coupled simulation workflows Docan, Ciprian; Parashar, Manish; Klasky, Scott Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851481	conference	January 2010
Global and local gyrokinetic simulations of high-performance discharges in view of ITER Jenko, F.; Told, D.; Görler, T. Nuclear Fusion, Vol. 53, Issue 7 https://doi.org/10.1088/0029-5515/53/7/073003	journal	May 2013
Big data provenance: Challenges, state of the art and opportunities Wang, Jianwu; Crawl, Daniel; Purawat, Shweta 2015 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2015.7364047	conference	October 2015
Global adjoint tomography: first-generation model Bozdağ, Ebru; Peter, Daniel; Lefebvre, Matthieu Geophysical Journal International, Vol. 207, Issue 3 https://doi.org/10.1093/gji/ggw356	journal	September 2016
Comprehensive Measurement and Analysis of the User-Perceived I/O Performance in a Production Leadership-Class Storage System Wan, Lipeng; Wolf, Matthew; Wang, Feiyi 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) https://doi.org/10.1109/ICDCS.2017.257	conference	June 2017
Electron Temperature Gradient Turbulence Dorland, W.; Jenko, F.; Kotschenreuther, M. Physical Review Letters, Vol. 85, Issue 26 https://doi.org/10.1103/PhysRevLett.85.5579	journal	December 2000
Exacution: Enhancing Scientific Data Management for Exascale Klasky, Scott; Suchyta, Eric; Ainsworth, Mark 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS) https://doi.org/10.1109/ICDCS.2017.256	conference	June 2017
Moving the Code to the Data - Dynamic Code Deployment Using ActiveSpaces Docan, Ciprian; Parashar, Manish; Cummings, Julian Distributed Processing Symposium (IPDPS), 2011 IEEE International Parallel & Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2011.120	conference	May 2011
TGE: Machine Learning Based Task Graph Embedding for Large-Scale Topology Mapping Choi, Jong Youl; Logan, Jeremy; Wolf, Matthew 2017 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2017.67	conference	September 2017
Topology-aware task mapping for reducing communication contention on large parallel machines Agarwal, T.; Sharma, A.; Laxmikant, A. Proceedings 20th IEEE International Parallel & Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2006.1639379	conference	January 2006
Compressed ion temperature gradient turbulence in diverted tokamak edge Chang, C. S.; Ku, S.; Diamond, P. H. Physics of Plasmas, Vol. 16, Issue 5 https://doi.org/10.1063/1.3099329	journal	May 2009
Machine Learning Predictions of Runtime and IO Traffic on High-End Clusters McKenna, Ryan; Herbein, Stephen; Moody, Adam 2016 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2016.58	conference	September 2016
Meteor: a middleware infrastructure for content‐based decoupled interactions in pervasive grid environments Jiang, Nanyan; Quiroz, Andres; Schmidt, Cristina Concurrency and Computation: Practice and Experience, Vol. 20, Issue 12 https://doi.org/10.1002/cpe.1278	journal	November 2007
GPUShare: Fair-Sharing Middleware for GPU Clouds Goswami, Anshuman; Young, Jeffrey; Schwan, Karsten 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) https://doi.org/10.1109/IPDPSW.2016.94	conference	May 2016
I/O performance challenges at leadership scale Lang, Samuel; Carns, Philip; Latham, Robert Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis - SC '09 https://doi.org/10.1145/1654059.1654100	conference	January 2009
Scientific workflow management and the Kepler system Ludäscher, Bertram; Altintas, Ilkay; Berkley, Chad Concurrency and Computation: Practice and Experience, Vol. 18, Issue 10 https://doi.org/10.1002/cpe.994	journal	January 2006
Generic topology mapping strategies for large-scale parallel architectures Hoefler, Torsten; Snir, Marc Proceedings of the international conference on Supercomputing - ICS '11 https://doi.org/10.1145/1995896.1995909	conference	January 2011
Data-intensive applications, challenges, techniques and technologies: A survey on Big Data Philip Chen, C. L.; Zhang, Chun-Yang Information Sciences, Vol. 275 https://doi.org/10.1016/j.ins.2014.01.015	journal	August 2014
Computing Just What You Need: Online Data Analysis and Reduction at Extreme Scales Foster, Ian 2017 IEEE 24th International Conference on High Performance Computing (HiPC) https://doi.org/10.1109/HiPC.2017.00042	conference	December 2017
Performance Modeling of In Situ Rendering Larsen, Matthew; Harrison, Cyrus; Kress, James SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.23	conference	November 2016
24/7 Characterization of petascale I/O workloads Carns, Philip; Latham, Robert; Ross, Robert 2009 IEEE International Conference on Cluster Computing and Workshops https://doi.org/10.1109/CLUSTR.2009.5289150	conference	August 2009
A Multiplatform Study of I/O Behavior on Petascale Supercomputers Luu, Huong; Winslett, Marianne; Gropp, William Proceedings of the 24th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '15 https://doi.org/10.1145/2749246.2749269	conference	January 2015
SSD-optimized workload placement with adaptive learning and classification in HPC environments Wan, Lipeng; Lu, Zheng; Cao, Qing 2014 30th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2014.6855552	conference	June 2014
Exascale Storage Systems the SIRIUS Way Klasky, S. A.; Abbasi, H.; Ainsworth, M. Journal of Physics: Conference Series, Vol. 759 https://doi.org/10.1088/1742-6596/759/1/012095	journal	October 2016

Similar Records

Extending the Publish/Subscribe Abstraction for High-Performance I/O and Data Management at Extreme Scale

Journal Article · Sun Mar 01 00:00:00 EST 2020 · Bulletin of the IEEE Technical Committee on Data Engineering · OSTI ID:1468120

Logan, Jeremy; Ainsworth, Mark; Atkins, Chuck; +23 more

Next Generation Workload Management System For Big Data on Heterogeneous Distributed Computing

Journal Article · Fri May 22 00:00:00 EDT 2015 · Journal of Physics. Conference Series · OSTI ID:1468120

Klimentov, A.; Buncic, P.; De, K.; +12 more

Computing for Finance

Multimedia · Wed Mar 24 00:00:00 EDT 2010 · OSTI ID:1468120

Title: A View from ORNL: Scientific Data Research Opportunities in the Big Data Age

Citation Formats

References (39)

Similar Records

Related Subjects