Abstract
Edge Computing emerges as a stable and efficient solution for IoT data processing and analytics. With big data distributed engines to be deployed on edge infrastructures, users seek solutions to evaluate the performance of their analytics queries. In this paper, we introduce SparkEdgeEmu, an interactive framework designed for researchers and practitioners who need to inspect the performance of Spark analytic jobs without the edge topology setup burden. SparkEdgeEmu provides: (i) parameterizable template-based use cases for edge infrastructures, (ii) real-time emulated environments serving ready-to-use Spark clusters, (iii) a unified and interactive programming interface for the framework’s execution and query submission, and (vi) utilization metrics from the underlying emulated topology as well as performance and quantitative metrics from the deployed queries. We evaluate the usability of our framework in a smart city use case and extract useful performance hints for the Apache Spark code execution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Possible issues regarding storage, like security concerns, are out of our scope. However, we plan to introduce an emulated distributed storage as a future extension.
- 3.
- 4.
References
5G Automotive Association: C-ITS vehicle to infrastructure services: How C-V2X technology completely changes the cost equation for road operators. White paper (2019)
Austria, S.: Federal ministry for climate action, environment, energy, mobility, innovation and technology. https://www.senderkataster.at/
Beilharz, J., et al.: Towards a staging environment for the internet of things. In: 2021 IEEE International Conference on Pervasive Computing and Communications Workshops (PerCom Workshops), pp. 312–315 (2021)
Catlett, C.E., Beckman, P.H., Sankaran, R., Galvin, K.K.: Array of things: a scientific research instrument in the public way: platform design and early lessons learned. In: Proceedings of the 2nd International Workshop on Science of Smart City Operations and Platforms Engineering, pp. 26–33. ACM (2017)
Chen, B., Wan, J., Celesti, A., Li, D., Abbas, H., Zhang, Q.: Edge computing in IoT-based manufacturing. IEEE Commun. Mag. 56(9), 103–109 (2018)
Chintapalli, S., et al.: Benchmarking streaming computation engines: storm, flink and spark streaming. In: IEEE IPDPSW (2016)
Cooper, B.F., Silberstein, A., Tam, E., Ramakrishnan, R., Sears, R.: Benchmarking cloud serving systems with YCSB. In: SoCC. ACM (2010)
Coutinho, A., Greve, F., Prazeres, C., Cardoso, J.: Fogbed: a rapid-prototyping emulation environment for fog computing. In: IEEE ICC (2018)
Dignan., L.: IoT devices to generate 79.4 ZB of data in 2025, says IDC (2019). https://bit.ly/3MVTY15
Georgiou, J., Symeonides, M., Kasioulis, M., Trihinas, D., Pallis, G., Dikaiakos, M.D.: BenchPilot: repeatable & reproducible benchmarking for edge micro-DCs. In: Proceedings of the 27th IEEE ISCC (2022)
Hasenburg, J., Grambow, M., Bermbach, D.: MockFog 2.0: automated execution of fog application experiments in the cloud. IEEE TCC 11(01), 1 (2021)
Intel: Intel NUC for edge compute. https://www.intel.com/content/www/us/en/products/docs/boards-kits/nuc/edge-compute.html
Karimov, J., Rabl, T., Katsifodimos, A., Samarev, R., Heiskanen, H., Markl, V.: Benchmarking distributed stream data processing systems. In: IEEE ICDE (2018)
Lantz, B., Heller, B., Mckeown, N.: A network in a laptop: rapid prototyping for software-defined networks. In: ACM SIGCOMM HotNets Workshop (2010)
Li, M., Tan, J., Wang, Y., Zhang, L., Salapura, V.: Sparkbench: a comprehensive benchmarking suite for in memory data analytic platform spark. In: ACM International Conference on Computing Frontiers (2015)
Nikolaidis, F., Chazapis, A., Marazakis, M., Bilas, A.: Frisbee: a suite for benchmarking systems recovery. In: Proceedings of the 1st Workshop on High Availability and Observability of Cloud Systems. HAOC (2021)
Rathijit, S., Abhishek, R., Alekh, J.: Predictive price-performance optimization for serverless query processing. In: International Conference on Extending Database Technology, EDBT (2023)
Rausch, T., Lachner, C., Frangoudis, P.A., Raith, P., Dustdar, S.: Synthesizing plausible infrastructure configurations for evaluating edge computing systems. In: 3rd USENIX Workshop on Hot Topics in Edge Computing (HotEdge 20). USENIX Association (2020)
Symeonides, M., Georgiou, Z., Trihinas, D., Pallis, G., Dikaiakos, M.D.: Fogify: a fog computing emulation framework. In: IEEE/ACM SEC (2020)
Symeonides, M., Trihinas, D., Georgiou, Z., Pallis, G., Dikaiakos, M.: Query-driven descriptive analytics for IoT and edge computing. In: Proceedings of IEEE International Conference on Cloud Engineering (IC2E 2019) (2019)
Symeonides, M., Trihinas, D., Pallis, G., Dikaiakos, M.D., Psomas, C., Krikidis, I.: 5G-slicer: an emulator for mobile IoT applications deployed over 5G network slices. In: IEEE/ACM IoTDI (2022)
Zeng, Y., Chao, M., Stoleru, R.: EmuEdge: a hybrid emulator for reproducible and realistic edge computing experiments. In: IEEE ICFC (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Symeonides, M., Trihinas, D., Pallis, G., Dikaiakos, M.D. (2023). SparkEdgeEmu: An Emulation Framework for Edge-Enabled Apache Spark Deployments. In: Cano, J., Dikaiakos, M.D., Papadopoulos, G.A., Pericàs, M., Sakellariou, R. (eds) Euro-Par 2023: Parallel Processing. Euro-Par 2023. Lecture Notes in Computer Science, vol 14100. Springer, Cham. https://doi.org/10.1007/978-3-031-39698-4_11
Download citation
DOI: https://doi.org/10.1007/978-3-031-39698-4_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-39697-7
Online ISBN: 978-3-031-39698-4
eBook Packages: Computer ScienceComputer Science (R0)