Abstract
Because of increased stream data, managing stream data has become quite important. This paper describes our data stream management system, which employs an architecture combining a stream processing engine and DBMS. Based on the architecture, the system processes both continuous queries and traditional one-shot queries. Our proposed query language supports not only filtering, join, and projection over data streams, but also continuous persistence requirements for stream data. Users can also specify continuous queries that integrate streaming data and historical data stored in DBMS. Another contribution of this paper is feasibility validation of queries. Processing queries on streams with frequent inputs may cause the system to overflow its capacity. Specifically, the maximum writing rate to DBMS is a significant bottleneck when we try to store stream data into DBMS. Our system detects infeasible queries in advance.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Abadi, D.J., et al.: Aurora: a New Model and Architecture for Data Stream Management. VLDB Journal 12(2), 120–139 (2003)
Abadi, D.J., et al.: The Design of the Borealis Stream Processing Engine. In: Proc. CIDR, pp. 277–289 (2005)
Arasu, A., et al.: The CQL Continuous Query Language: Semantic Foundations and Query Execution. VLDB Journal 15(2) (2006)
Ayad, A.M., et al.: Static Optimization of Conjunctive Queries with Sliding Windows Over Infinite Streams. In: Proc. ACM SIGMOD, pp. 419–430 (2004)
Babcock, B., et al.: Load Shedding for Aggregation Queries over Data Streams. In: Proc. ICDE, pp. 350–361 (2004)
Chandrasekaran, S., et al.: TelegraphCQ: Continuous Dataflow Processing for an Uncertain World. In: Proc. CIDR (2003)
Chen, J., et al.: NiagaraCQ: A Scalable Continuous Query System for Internet Databases. In: Proc. ACM SIGMOD, pp. 379–390 (2000)
Motwani, R., et al.: Query Processing, Resource Management, and Approximation in a Data Stream Management System. In: Proc. CIDR (2003)
Tatbul, N., et al.: Load Shedding in a Data Stream Manager. In: Proc. VLDB, pp. 309–320 (2003)
Viglas, S.D., et al.: Rate-based Query Optimization for Streaming Information Sources. In: Proc. ACM SIGMOD, pp.37–48 (2002)
Wang, S., et al.: State-Slice: New Paradigm of Multi-query Optimization of Window-based Stream Queries. In: Proc. VLDB, pp. 619–630 (2006)
StreamSpinner. http://www.streamspinner.org/
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Watanabe, Y., Yamada, S., Kitagawa, H., Amagasa, T. (2007). Integrating a Stream Processing Engine and Databases for Persistent Streaming Data Management. In: Wagner, R., Revell, N., Pernul, G. (eds) Database and Expert Systems Applications. DEXA 2007. Lecture Notes in Computer Science, vol 4653. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74469-6_41
Download citation
DOI: https://doi.org/10.1007/978-3-540-74469-6_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74467-2
Online ISBN: 978-3-540-74469-6
eBook Packages: Computer ScienceComputer Science (R0)