Affiliations: [a] School of Computer Science, Nanjing University of Posts and Telecommunications, Nanjing 210003, China. E-mails: [email protected], [email protected], [email protected] | [b] Jiangsu High Technology Research Key Laboratory for Wireless Sensor Networks, Nanjing University of Posts and Telecommunications, Nanjing 210003, China. E-mails: [email protected], [email protected] | [c] School of Mathematics and Information Security, Royal Holloway, University of London, Surrey, TW200EX, UK. E-mail: [email protected]
Abstract: In Apache Spark cloud computing environment, computation capability of each node varies, together with the data size and uncertainties in application execution, these result in the differences in task execution time of a job. In order to enhance the accuracy in load execution time prediction, and reasonably guide the user to apply for Spark cluster resources, this paper researches on the execution flow of Spark job, collects the load time consumption index, and puts forward the time index fusion calculation scheme. And then, this paper researches on the multiple linear regression model and support vector machine model to explore the payload execution time and CPU Core, inputting data volume, memory size and other performance indicators. Based on the two models above, this paper proposes a Standard Regression Coefficient-based Weighted Support Vector Regression time prediction model (SRC-WSVR). Finally, through comparing the results from the prediction model proposed in this paper with conventional regression prediction model and Standard Support Vector Machine model, it proves that SRC-WSVR has a higher prediction accuracy, which can provide valid data reference for predicting Spark resource consumption.
Keywords: Weighted Support Vector Regression Machine, Standard Regression Coefficient, execution time prediction, performance benchmark, keyword five, Spark
