Abstract
The rise of the Internet of Things and Fog computing has increased substantially the number of interconnected devices at the edge of the network. As a result, a large amount of computations is now performed in the fog generating vast amounts of data. To process this data in near real time, stream processing is typically employed due to its efficiency in handling continuous streams of information in a scalable manner. However, most stream processing approaches do not consider the underlying network devices as candidate resources for processing data. Moreover, many existing works do not take into account the incurred network latency of performing computations on multiple devices in a distributed way. Consequently, the fog computing resources may not be fully exploited by existing stream processing approaches. To avoid this, we formulate an optimization problem for utilizing the existing fog resources, and we design heuristics for solving this problem efficiently. Furthermore, we integrate our heuristics into Apache Storm, and we perform experiments that show latency-related benefits compared to alternatives.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Apache software foundation: apache storm documentation: resource aware scheduler. https://storm.apache.org/releases/2.4.0/Resource_Aware_Scheduler_overview.html#Enhancements-on-original-DefaultResourceAwareStrategy (2022). Accessed 31 Mar 2022
Apache software foundation: apache storm documentation: scheduler. https://storm.apache.org/releases/2.4.0/Storm-Scheduler.html (2022). Accessed 31 Mar 2022
de Assunção, M.D., Veith, A.D.S., Buyya, R.: Distributed data stream processing and edge computing: a survey on resource elasticity and future directions. Netw. Comput. Appl. 103, 1–17 (2018)
Axenie, C., Tudoran, R., Bortoli, S., Hassan, M.A.H., Sánchez, C.S., Brasche, G.: Dimensionality reduction for low-latency high-throughput fraud detection on datastreams. In: 18th IEEE International Conference On Machine Learning And Applications, pp. 1170–1177. IEEE (2019)
Cardellini, V., Grassi, V., Presti, F.L., Nardelli, M.: Distributed QoS-aware scheduling in storm. In: 9th ACM International Conference on Distributed Event-Based Systems, pp. 344–347. ACM (2015)
Cardellini, V., Grassi, V., Presti, F.L., Nardelli, M.: On QoS-aware scheduling of data stream applications over fog computing infrastructures. In: 2015 IEEE Symposium on Computers and Communication, pp. 271–276. IEEE (2015)
Dorigo, M., Maniezzo, V., Colorni, A.: Ant system: optimization by a colony of cooperating agents. IEEE Trans. Syst. Man Cybern. Part B 26(1), 29–41 (1996)
Eskandari, L., Huang, Z., Eyers, D.M.: P-Scheduler: adaptive hierarchical scheduling in apache storm. In: Australasian Computer Science Week Multiconference, p. 26. ACM (2016)
Hasenburg, J., Grambow, M., Bermbach, D.: Mockfog 2.0: automated execution of fog application experiments in the cloud. IEEE Trans. Cloud Comput. 11(01), 58–70 (2021)
d Hasenburg, J., Grambow, M., Grünewald, E., Huk, S., Bermbach, D.: MockFog: emulating fog computing infrastructure in the cloud. In: IEEE International Conference on Fog Computing, pp. 144–152. IEEE (2019)
Hiessl, T., Karagiannis, V., Hochreiner, C., Schulte, S., Nardelli, M.: Optimal placement of stream processing operators in the fog. In: 3rd IEEE International Conference on Fog and Edge Computing, pp. 1–10. IEEE (2019)
IEEE: IEEE Standard 1934–2018 for adoption of OpenFog reference architecture for fog computing (2018)
Karagiannis, V., Frangoudis, P.A., Dustdar, S., Schulte, S.: Context-aware routing in fog computing systems. IEEE Trans. Cloud Comput. 11(01), 532–549 (2021)
Kobourov, S.G.: Spring embedders and force directed graph drawing algorithms. CoRR abs/1201.3011 (2012)
Mayer, R., Graser, L., Gupta, H., Saurez, E., Ramachandran, U.: EmuFog: extensible and scalable emulation of large-scale fog computing infrastructures. In: IEEE Fog World Congress, pp. 1–6. IEEE (2017)
Nardelli, M., Cardellini, V., Grassi, V., Presti, F.L.: Efficient operator placement for distributed data stream processing applications. IEEE Trans. Parallel Distrib. Syst. 30(8), 1753–1767 (2019)
Peng, B., Hosseini, M., Hong, Z., Farivar, R., Campbell, R.H.: R-Storm: resource-aware scheduling in storm. In: 16th Annual Middleware Conference, pp. 149–161. ACM (2015)
Pietzuch, P.R., Ledlie, J., Shneidman, J., Roussopoulos, M., Welsh, M., Seltzer, M.I.: Network-aware operator placement for stream-processing systems. In: 22nd International Conference on Data Engineering, p. 49. IEEE (2006)
Prosperi, L., Costan, A., Silva, P., Antoniu, G.: Planner: Cost-efficient execution plans placement for uniform stream analytics on edge and cloud. In: 2nd IEEE/ACM Workflows in Support of Large-Scale Science, pp. 42–51. IEEE (2018)
Skarlat, O., Nardelli, M., Schulte, S., Dustdar, S.: Towards QoS-aware fog service placement. In: IEEE International Conference on Fog and Edge Computing, pp. 89–96. IEEE (2017)
Stützle, T., Hoos, H.H.: MAX-MIN ant system. Futur. Gener. Comput. Syst. 16(8), 889–914 (2000)
Szymaniak, M., Presotto, D.L., Pierre, G., van Steen, M.: Practical large-scale latency estimation. Comput. Netw. 52(7), 1343–1364 (2008)
Tsai, C., Rodrigues, J.J.P.C.: Metaheuristic scheduling for cloud: a survey. IEEE Syst. J. 8(1), 279–291 (2014)
Varshney, P., Simmhan, Y.: Characterizing application scheduling on edge, fog and cloud computing resources. Softw. Pract. Experience 50(5), 558–595 (2020)
Yousefpour, A., et al.: All one needs to know about fog computing and related edge computing paradigms: a complete survey. J. Syst. Architect. 98, 289–330 (2019)
Acknowledgements
The financial support by the Austrian Federal Ministry for Digital and Economic Affairs, the National Foundation for Research, Technology and Development as well as the Christian Doppler Research Association is gratefully acknowledged.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Ecker, R., Karagiannis, V., Sober, M., Ebrahimi, E., Schulte, S. (2024). Latency-Aware Placement of Stream Processing Operators. In: Zeinalipour, D., et al. Euro-Par 2023: Parallel Processing Workshops. Euro-Par 2023. Lecture Notes in Computer Science, vol 14351. Springer, Cham. https://doi.org/10.1007/978-3-031-50684-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-50684-0_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-50683-3
Online ISBN: 978-3-031-50684-0
eBook Packages: Computer ScienceComputer Science (R0)