Loading [a11y]/accessibility-menu.js
Integrated HPC Scheduler Data Processing Workflow using Apache Zeppelin | IEEE Conference Publication | IEEE Xplore

Integrated HPC Scheduler Data Processing Workflow using Apache Zeppelin


Abstract:

Big data analytics pipeline often naturally involves the components with different programming language, various programming models, etc. And it presents steep learning c...Show More

Abstract:

Big data analytics pipeline often naturally involves the components with different programming language, various programming models, etc. And it presents steep learning curve not only on developing the tools but also on using them. Building a user friendly interface can hide these usage complexities, and provide a easy way to get the insights out of data. It's a challenging task to link those components together to make a smooth end-to-end workflow. Apache Zeppelin provides native support on multiple language and data processing backends so that different workflow components can be linked together on Zeppelin's framework. We developed a web interface for analyzing High Performance Computing data center scheduler log data through Apache Zeppelin's support on AngularJS, Spark, Python and Batch. An interactive PACE-Fast Analysis of Computational Trends (PACE-FACT) environment is built on the extension of our previous work, this environment seamlessly puts multiple log data analysis and visualization components together, and it allows to visualize the result data interactively without dealing with cumbersome command line user interface. In this work, we demonstrate that software ranking and analysis can be done through web GUI with user specified date range. And this system will be used in Georgia Institute of Technology (Georgia Tech)'s high performance computing (HPC) PACE center.
Date of Conference: 10-13 December 2018
Date Added to IEEE Xplore: 24 January 2019
ISBN Information:
Conference Location: Seattle, WA, USA

Contact IEEE to Subscribe

References

References is not available for this document.