Skip to main content
SpringerLink
Log in

Window-based multiple continuous query algorithm for data streams

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

As more data are being collected and analyzed in real time, stream processing is attracting greater attention. In traditional network, economic and financial analysis, or processing of sensors data in Internet of Things, efficient and timely methods of handling continuous data generation are required. Data are being produced at higher frequency, and the volume of data to be processed within a particular period of time is increasing rapidly. This is especially true for continuous window aggregation, which involves intensive computation. An increase in the number of query windows also generates scalability problems involving aggregate queries. Traditional query handling algorithms can perform many repeated operations. In this study, to enable window-based shared processing of continuous queries, a window reuse algorithm for a multi-query environment based on pace and results is proposed: the multiple continuous query algorithm (MCQA). The aggregation is simplified by gradually shrinking the set of multiple values so that the operation is reduced at each step, eventually achieving result sharing. The algorithm is implemented with the Storm stream processing framework. Experiments prove that the MCQA performance is more efficient and effectively reduces memory usage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

References

  1. Naidu KVM, Rastogi R, Satkin S, Srinivasan A (2011) Memory constrained aggregate computation over data streams. In: Proceedings of 27th IEEE International Conference on Data Engineering (ICDE), Hannover, Germany, pp 852–863

  2. Liu W, Shen YM, Wang P (2016) An efficient approach of processing multiple continuous queries. J Comput Sci Technol 31(6):1212–1227

    Article  Google Scholar 

  3. Gulisano V, Jimenez-Peris R, Patio-Martinez M (2012) StreamCloud: an elastic and scalable data streaming system. IEEE Trans Parallel Distrib Syst 23(12):2351–2365

    Article  Google Scholar 

  4. Krishnamurthy S, Franklin MJ, Davis J, Farina D, Golovko P, Li A, Thombre N (2010) Continuous analytics over discontinuous streams. In: Proceedings of 29th ACM SIGMOD International Conference on Management of Data, Indianapolis, Indiana, pp 1081–1092

  5. Roy P, Seshadri S, Sudarshan S, Bhobe S (2000) Efficient and extensible algorithms for multi query optimization. ACM SIGMOD Rec. 29(2):249–260

    Article  Google Scholar 

  6. Storm[EB/OL] http://storm.apache.org/. Accessed 13 June 2018

  7. Gray J, Chaudhuri S, Bosworth A, Layman A, Reichart D, Venkatrao M, Pellow F, Pirahesh H (1997) Data cube: a relational aggregation operator generalizing groupby, cross-tab, and sub-totals. Data Min Knowl Discov 1(1):29–53

    Article  Google Scholar 

  8. Abadi DJ, Carney D, Cetintemel U, Cherniack M, Convey C, Lee S, Stonebraker M, Tatbul N, Zdonik S (2003) Aurora: a new model and architecture for data stream management. VLDB J 12(2):120–139

    Article  Google Scholar 

  9. Huebsch R, Garofalakis M, Hellerstein JM, Stoica I (2010) Sharing aggregate computation for distributed queries. In: Proceedings of 26th ACM SIGMOD International Conference on Management of Data, Beijing, China, pp 485–496

  10. Babu S, Widom J (2001) Continuous queries over data streams. ACM SIGMOD Rec 30(3):109–120

    Article  Google Scholar 

  11. Seshadri S, Kumar V, Cooper BF (2007) Optimizing Multiple Distributed Stream Queries Using Hierarchical Network Partitions. In: IEEE International Parallel and Distributed Processing Symposium, pp 1–10

  12. Aniello L, Baldoni R, Querzoni L (2013) Adaptive online scheduling in storm. In: Proceedings of 7th ACM International Conference on Distributed Event-Based Systems (DEBS), Arlington, Texas, USA, pp 207–218

  13. Arasu A, Babu S, Widom J (2006) The CQL continuous query language: semantic foundations and query execution. VLDB J 15(2):121–142

    Article  Google Scholar 

  14. Krishnamurthy S, Franklin MJ, Davis J, Farina D, Golovko P, Li A, Thombre N (2010) Continuous analytics over discontinuous streams. In: Proceedings of 29th ACM SIGMOD International Conference on Management of Data, Indianapolis, Indiana, pp 1081–1092

  15. Guirguis S, Sharaf MA, Chrysanthis PK, Labrinidis A (2011) Optimized processing of multiple aggregate continuous queries. In: Proceedings of 20th ACM International Conference on Information and Knowledge Management, Glasgow, United Kingdom, pp 1515–1524

  16. Guirguis S, Sharaf MA, Chrysanthis PK, Labrinidis A (2012) Three-level processing of multiple aggregate continuous queries. In: Proceedings of 28th IEEE International Conference on Data Engineering (ICDE), Washington, DC, pp 929–940

  17. Patroumpas K, Sellis T (2010) Multi-granular time-based sliding windows over data stream. International Symposium on Temporal Representation and Reasoning, IEEE Computer Society, pp 146–153

  18. Patroumpas K, Sellis T (2011) Subsuming multiple sliding windows for shared stream computation. Adv Databases Inf Syst 36:56–69

    Article  Google Scholar 

Download references

Acknowledgements

This work is supported by Young Doctoral Science and Technology Talents of Xinjiang (2017Q085), Natural Science Foundation of Xinjiang (2017D01B09), Scientific Research Program of the Higher Education Institution of Xinjiang (XJEDU2016I049) and Doctoral Initiation Fund of Xinjiang Institute of Engineering (2017XGY022110).

Author information

Author notes

  1. Wen Liu

    Present address: , No. 1350, Aidinghu Road, Economic and Technological Development District, Ürümqi, 830023, Xinjiang Province, People’s Republic of China

Authors and Affiliations

  1. School of Control Engineering, Xinjiang Institute of Engineering, Ürümqi, 830023, China

    Wen Liu & Junxia Liu

  2. School of Computer Science and Technology, Dalian University of Technology, Dalian, China

    Tuqian Zhang

Authors
  1. Wen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tuqian Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junxia Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wen Liu.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, W., Zhang, T. & Liu, J. Window-based multiple continuous query algorithm for data streams. J Supercomput 75, 5782–5807 (2019). https://doi.org/10.1007/s11227-019-02856-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-019-02856-z

Keywords

Search

Navigation