Abstract
As the most active project in the Hadoop ecosystem these days [1], Spark is a fast and general purpose engine for large-scale data processing. Spark runs programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk [2]. However, Spark performance is impacted by many factors especially memory and JVM related, which makes capacity planning and tuning for Spark clusters extremely difficult. Current estimation based solution are highly dependent on experience which are trial-and-error and far from efficient and accurate. Here, we propose a novel Spark simulator based on CSMethod [3], extension with a fine-grained multi-layered memory subsystem, well suitable for this scenario. The whole Spark application execution life cycle is simulated, hardware activities derived from software operations are dynamically mapped onto architecture models for processors, storage, and network devices. Experimental results with several popular micro benchmarks and a real case IoT workloads demonstrate that our Spark Simulator achieves high accuracy with an average error rate below 7%, with light weight computing resource. Case studies are also demonstrated to show the simulator’s capability.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bian, Z., Wang, K., Wang, Z., Munce, G., Cremer, I., Zhou, W., Chen, Q., Xu, G.: Simulating big data clusters for system planning, evaluation and optimization. In: ICPP-2014, 9–12 September 2014, Minneapolis, MN, USA (2014)
Xin, R.S., Rosen, J., Zaharia, M., Franklin, M.J., Shenker, S., Stoica, I.: Shark: SQL and rich analytics at scale. In: SIGMOD (2013)
Zaharia, M.: Spark: in-memory cluster computing for iterative and interactive applications. In: Invited Talk at NIPS 2011 Big Learning Workshop: Algorithms, Systems, and Tools for Learning at Scale (2011)
Zaharia, M., Chowdhury, M., Franklin, M.J., Shenker, S., Stoica, I.: Spark: cluster computing with working sets. In: HotCloud 2010 Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing, p. 10. CA (2010)
Apache Software Foundation: The Apache Software Foundation Announces Apache Spark as a Top-Level Project, 27 February 2014. Accessed 4 Mar 2014
Kolberg, W., Marcos, P.D.B., Anjos, J.C., Miyazaki, A.K., Geyer, C.R., Arantes, L.B.: MRSG – a MapReduce simulator over SimGrid. Parallel Comput. 39(4–5), 233–244 (2013)
Wang, G., Butt, A.R., Pandey, P., Gupta, K.: A simulation approach to evaluating design decisions in MapReduce setups. In: Proceedings of the 17th Annual Meeting of the IEEE/ACM International Symposium on Modelling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS 2011), London (2011)
Kennedy, P.R., Gopal, T.V.: A MR simulator in facilitating cloud computing. Int. J. Comput. Appl. 72(5), 43–49 (2013). Published by Foundation of Computer Science, New York, USA
Verma, A., Cherkasova, L., Campbell, R.H.: Play It Again, SimMR! In: Proceedings of IEEE International Conference Cluster Computing (Cluster 2011) (2011)
Intel, Simulation software. http://www.intel.com/content/www/ru/ru/cofluent/intel-cofluentstudio.html
Skiena, S.S.: The Algorithm Design Manual, Springer (2008)
https://www.phdata.io/real-time-analytics-on-medical-device-data/
Magnusson, P.S., Christensson, M., Eskilson, J., Forsgren, D., Hallberg, G., Hogberg, J., Larsson, F., Moestedt, A., Werner, B.: Simics: a full system simulation platform. IEEE Comput. 35(2), 50–58 (2002)
León, E.A., Riesen, R., Bridges, P.G., Maccabe, A.B.: Instruction-level simulation of a cluster at scale. In: HPCC, 14–20 November 2009, Portland, OR, USA (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this paper
Cite this paper
Chen, Q., Wang, K., Bian, Z., Cremer, I., Xu, G., Guo, Y. (2018). Cluster Performance Simulation for Spark Deployment Planning, Evaluation and Optimization. In: Obaidat, M., Ören, T., Merkuryev, Y. (eds) Simulation and Modeling Methodologies, Technologies and Applications. SIMULTECH 2016. Advances in Intelligent Systems and Computing, vol 676. Springer, Cham. https://doi.org/10.1007/978-3-319-69832-8_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-69832-8_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-69831-1
Online ISBN: 978-3-319-69832-8
eBook Packages: EngineeringEngineering (R0)