Abstract
In a distributed processing environment, the static placement of query operators may result in unsatisfactory system performance due to unpredictable factors such as changes of servers’ load, data arrival rates, etc. The problem is exacerbated for continuous (and long running) monitoring queries over data streams as any suboptimal placement will affect the system for a very long time. In this paper, we formalize and analyze the operator placement problem in the context of a locally distributed continuous query system. We also propose a solution, that is asynchronous and local, to dynamically manage the load across the system nodes. Essentially, during runtime, we migrate query operators/fragments from overloaded nodes to lightly loaded ones to achieve better performance. Heuristics are also proposed to maintain good data flow locality. Results of a performance study shows the effectiveness of our technique.
An erratum to this chapter can be found at http://dx.doi.org/10.1007/11914853_71.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Arasu, A., et al.: Stream: The stanford stream data manager. IEEE Data Eng. Bull. 26(1), 19–26 (2003)
Ayad, A., Naughton, J.F.: Static optimization of conjunctive queries with sliding windows over infinite streams. In: SIGMOD, pp. 419–430 (2004)
Ahmad, Y., Çetintemel, U.: Networked query processing for distributed stream-based applications. In: VLDB, pp. 456–467 (2004)
Carney, D., et al.: Monitoring streams - a new class of data management applications. In: VLDB, pp. 215–226 (2002)
Carney, D., et al.: Operator scheduling in a data stream manager. In: VLDB, pp. 838–849 (2003)
Chandrasekaran, S., et al.: Telegraphcq: Continuous dataflow processing for an uncertain world. In: CIDR (2003)
Cherniack, M., et al.: Scalable distributed stream processing. In: CIDR (2003)
Pietzuch, P., et al.: Network-aware operator placement for stream-processing systems. In: ICDE, p. 49 (2006)
Shah, M.A., et al.: Flux: An adaptive partitioning operator for continuous query systems. In: ICDE, pp. 25–36 (2003)
Srivastava, U., et al.: Operator Placement for In-Network Stream Query Processing. In: PODS, pp. 250–258 (2005)
Tian, F., DeWitt, D.J.: Tuple routing strategies for distributed eddies. In: VLDB, pp. 333–344 (2003)
Viglas, S., Naughton, J.F.: Rate-based query optimization for streaming information sources. In: SIGMOD, pp. 37–48 (2002)
Willebeek-LeMair, M., Reeves, A.P.: Strategies for dynamic load balancing on highly parallel computers. IEEE Trans. Parallel Distrib. Syst. 4(9), 979–993 (1993)
Xing, Y., et al.: Dynamic load distribution in the Borealis stream processor. In: ICDE, pp. 791–802 (2005)
Zhou, Y., et al.: Adaptive reorganization of coherency-preserving dissemination tree for streaming data. In: ICDE, p. 55 (2006)
Zhou, Y., et al.: Dynamic load management for distributed continuous query systems. Unpublished manuscript (2005), http://www.comp.nus.edu.sg/~zhouyong/papers/op.html
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Zhou, Y., Ooi, B.C., Tan, KL., Wu, J. (2006). Efficient Dynamic Operator Placement in a Locally Distributed Continuous Query System. In: Meersman, R., Tari, Z. (eds) On the Move to Meaningful Internet Systems 2006: CoopIS, DOA, GADA, and ODBASE. OTM 2006. Lecture Notes in Computer Science, vol 4275. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11914853_5
Download citation
DOI: https://doi.org/10.1007/11914853_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-48287-1
Online ISBN: 978-3-540-48289-5
eBook Packages: Computer ScienceComputer Science (R0)