Abstract
The concept of stream data processing is becoming challenging in most business sectors where try to improve their operational efficiency by deriving valuable information from unstructured, yet, contentiously generated high volume raw data in an expected time spans. A modern streamlined data processing platform is required to execute analytical pipelines over a continues flow of data-items that might arrive in a high rate. In most cases, the platform is also expected to dynamically adapt to dynamic characteristics of the incoming traffic rates and the ever-changing condition of underlying computational resources while fulfill the tight latency constraints imposed by the end-users. Apache Storm has emerged as an important open source technology for performing stream processing with very tight latency constraints over a cluster of computing nodes. To increase the overall resource utilization, however, the service provider might be tempted to use a consolidation strategy to pack as many applications as possible in a (cloud-centric) cluster with limited number of working nodes. However, collocated applications can negatively compete with each other, for obtaining the resource capacity in a shared platform that, in turn, the result may lead to a severe performance degradation among all running applications.
The main objective of this work is to develop an elastic solution in a modern stream processing ecosystem, for addressing the shared resource contention problem among collocated applications. We propose a mechanism, based on design principles of Model Predictive Control theory, for coping with the extreme conditions in which the collocated analytical applications have different quality of service (QoS) levels while the shared-resource interference is considered as a key performance limiting parameter. Experimental results confirm that the proposed controller can successfully enhance the \(p\)-99 latency of high priority applications by 67%, compared to the default round robin resource allocation strategy in Storm, during the high traffic load, while maintaining the requested quality of service levels.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Allen, A.O.: Probability, Statistics, Queueing Theory. Academic Press (1990)
Andrade, H.C.M., Gedik, B., Turaga, D.S.: Fundamentals of Stream Processing. Cambridge University, New York, NY, USA (2014)
Aniello, L., Baldoni, R., Querzoni, L.: Adaptive online scheduling in storm. In: Proceedings of the 7th ACM International Conference on Distributed Event-Based Systems, pp. 207–218. ACM (2013)
Box, G., et al.: Time Series: Forecasting & Control. Wiley (2008)
Casalicchio, E., et al.: Autonomic resource provisioning in cloud systems with availability goals. In: Proceedings of the 2013 ACM Cloud and Autonomic Computing Conference, pp. 1–10. ACM (2013)
De Matteis, T., Mencagli, G.: Keep calm and react with foresight: strategies for low-latency and energy-efficient elastic data stream processing. In: Proceedings of the 21st ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, PPoPP 2016, ACM, New York, NY, USA, pp. 13:1–13:12 (2016)
DeMatteis, T., et al.: Proactive elasticity and energy awareness in data stream processing. J. Syst. Softw. 127, 302–319 (2016)
Gedik, B., et al.: Elastic scaling for data stream processing. Trans. Parallel Distrib. Syst. 25(6), 1447–1463 (2014)
Jain, A.: Mastering Apache Storm. Packt Publishing (2017)
Kim, Y., et al.: Scalable & high-performance scheduling algorithm for multiple memory controllers. In: HPCA-16 2010 The Sixteenth International Symposium on High-Performance Computer Architecture, pp. 1–12. IEEE (2010)
Kusic, D., et al.: Power and performance management of virtualized computing environments via lookahead control. In: International Conference on Autonomic Computing. ICAC 2008, IEEE, Washington, DC, pp. 3–12 (2008)
Moraveji, R., Taheri, J., HoseinyF., M., et al.: Data-intensive workload consolidation for the hdfs systems. In: ACM/IEEE 13th International Conference on Grid Computing, pp. 95–103. IEEE (2012)
Mutlu, O., Moscibroda, T.: Stall-time fair memory access scheduling for chip multiproc. In: 40th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO 2007), pp. 146–160. IEEE (2007)
Nathuji, R., et al.: Q-clouds: managing performance interference effects for QoS-aware clouds. In: Proceedings of the 5th European conference on Computer systems, pp. 237–250. ACM (2010)
Onur, M., Thomas, M.: Parallelism-aware batch scheduling: enhancing both performance & fairness of shared dram systems. In: International Symposium on Computer Architecture, pp. 63–74. IEEE (2008)
Rawlings, J., et al.: Model predictive control: theory & design. NobHill (2009)
Stonebraker, M., et al.: The 8 requirements of real-time stream processing. ACM Sigmod Rec. 34(4), 42–47 (2005)
Subramanian, L., et al.: MISE: Providing performance predictability & improving fairness in shared memory systems. In: High Performance Computer Architecture, pp. 639–650. IEEE (2013)
Tembey, P., et al.: Application & platform-aware RA in consolidated servers. In: SOCC, pp. 1–14 (2014)
Wang, H., et al.: A-DRM: architecture-aware distributed RA of virtualized clusters. In: ACM SIGPLAN/SIGOPS on Virtual Execution Env. pp. 93–106 (2015)
Xu, J., et al.: T-storm: traffic-aware online scheduling in storm. In: IEEE 34th International Conference on Distributed Computing Systems, pp. 535–544. IEEE (2014)
Yang, H., et al.: Precise online QoS man. for increased util. in warehouse computer, pp. 607–618 (2013)
Acknowledgments
Professor Albert Y. Zomaya would like to acknowledge the support of the Australian Research Council Discovery scheme (grant DP200103494). Professor Zahir Tari would like to acknowledge the support of the Australian Research Council Discovery scheme (grant DP200100005). Professor Javid Taheri would like to acknowledge the support of the Knowledge Foundation of Sweden through the AIDA project. Dr. M.Reza HoseinyFarahabady would like to acknowledge continued support of The Centre for Distributed and High Performance Computing in The University of Sydney for providing access to advanced high-performance computing and cloud facilities, digital platforms and tools.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
HoseinyFarahabady, M.R., Taheri, J., Zomaya, A.Y., Tari, Z. (2021). Graceful Performance Degradation in Apache Storm. In: Zhang, Y., Xu, Y., Tian, H. (eds) Parallel and Distributed Computing, Applications and Technologies. PDCAT 2020. Lecture Notes in Computer Science(), vol 12606. Springer, Cham. https://doi.org/10.1007/978-3-030-69244-5_35
Download citation
DOI: https://doi.org/10.1007/978-3-030-69244-5_35
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69243-8
Online ISBN: 978-3-030-69244-5
eBook Packages: Computer ScienceComputer Science (R0)