Abstract
Replication, the widely adapted technique for crash fault tolerance introduces additional infrastructural costs for resource limited clusters. In this paper we take a different approach for maintaining stream program performance during crash failures. It is based on the concepts of automatic code generation. Albatross, the middleware we introduce for this task maintains the same performance level during crash failures based on predetermined priority values assigned to each stream program. Albatross constructs different versions of the input stream programs (sample programs) with different levels of performance characteristics, and assigns the best performing programs for normal operations. During node failure or node recovery, potential use of a different version of sample program is evaluated in order to bring the performance of each job back to its original level. We evaluated effectiveness of this approach with three different real world stream computing applications on System S distributed stream processing platform. We show that our approach is capable of maintaining stream program performance even if half of the nodes of the cluster has been crashed using both Apnoea, and Regex applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Andrade, H., Gedik, B., Wu, K.-L., Yu, P.S.: Scale-up strategies for processing high-rate data streams in systems. In: IEEE 25th International Conference on Data Engineering, ICDE 2009, March 29-April 2, pp. 1375–1378 (2009)
Babcock, B., Datar, M., Motwani, R.: Load shedding in data stream systems. In: Data Streams, vol. 31, pp. 127–147. Springer US (2007)
Catley, C., et al.: A framework to model and translate clinical rules to support complex real-time analysis of physiological and clinical data. In: Proceedings of the 1st ACM International Health Informatics Symposium, IHI 2010, pp. 307–315. ACM, New York (2010)
Dayarathna, M., Suzumura, T.: Hirundo: a mechanism for automated production of optimized data stream graphs. In: Proceedings of the Third Joint WOSP/SIPEW International Conference on Performance Engineering, ICPE 2012, pp. 335–346. ACM, New York (2012)
Furht, B., Escalante, A.: Handbook of Cloud Computing. Springer-Verlag New York, Inc. (2010)
Gedik, B., et al.: Spade: the system s declarative stream processing engine. In: SIGMOD 2008, pp. 1123–1134. ACM, New York (2008)
Gu, Y., Zhang, Z., Ye, F., Yang, H., Kim, M., Lei, H., Liu, Z.: An empirical study of high availability in stream processing systems. In: Middleware 2009, pp. 23:1–23:9. Springer-Verlag New York, Inc., New York (2009)
Hwang, J.-H., Cetintemel, U., Zdonik, S.: Fast and highly-available stream processing over wide area networks, pp. 804–813 (April 2008)
Hwang, J.-H., et al.: High-availability algorithms for distributed stream processing. In: Proceedings of the 21st International Conference on Data Engineering, ICDE 2005, pp. 779–790. IEEE Computer Society, Washington, DC (2005)
IBM. Ibm infosphere streams version 1.2.1: Installation and administration guide (October 2010)
Ishii, A., Suzumura, T.: Elastic stream computing with clouds. In: 2011 IEEE International Conference on Cloud Computing (CLOUD), pp. 195–202 (July 2011)
Khandekar, R., Hildrum, K., Parekh, S., Rajan, D., Wolf, J., Wu, K.-L., Andrade, H., Gedik, B.: COLA: Optimizing stream processing applications via graph partitioning. In: Bacon, J.M., Cooper, B.F. (eds.) Middleware 2009. LNCS, vol. 5896, pp. 308–327. Springer, Heidelberg (2009)
Kleiminger, W., Kalyvianaki, E., Pietzuch, P.: Balancing load in stream processing with the cloud. In: Data Engineering Workshops (ICDEW), pp. 16–21 (April 2011)
Logothetis, D., Trezzo, C., Webb, K.C., Yocum, K.: In-situ mapreduce for log processing. In: USENIXATC 2011, Berkeley, CA, USA, p. 9. USENIX Association, Berkeley (2011)
Tanenbaum, A.S., Steen, M.V.: Distributed Systems. Pearson Education, Inc. (2007)
White, T.: Hadoop: The Definitive Guide. O’Reilly Media, Inc. (2010)
Wolf, J., et al.: Soda: an optimizing scheduler for large-scale stream-based distributed computer systems. In: Middleware 2008, pp. 306–325. Springer-Verlag New York, Inc., New York (2008)
Zhang, Z., et al.: A hybrid approach to high availability in stream processing systems. In: Distributed Computing Systems (ICDCS), pp. 138–148 (June 2010)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Dayarathna, M., Suzumura, T. (2013). A Mechanism for Stream Program Performance Recovery in Resource Limited Compute Clusters. In: Meng, W., Feng, L., Bressan, S., Winiwarter, W., Song, W. (eds) Database Systems for Advanced Applications. DASFAA 2013. Lecture Notes in Computer Science, vol 7826. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-37450-0_12
Download citation
DOI: https://doi.org/10.1007/978-3-642-37450-0_12
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-37449-4
Online ISBN: 978-3-642-37450-0
eBook Packages: Computer ScienceComputer Science (R0)