skip to main content
10.1145/3030207.3030227acmconferencesArticle/Chapter ViewAbstractPublication PagesicpeConference Proceedingsconference-collections
research-article

Latency Aware Elastic Switching-based Stream Processing Over Compressed Data Streams

Published: 17 April 2017 Publication History

Abstract

Elastic scaling of event stream processing systems has gained significant attention recently due to the prevalence of cloud computing technologies. We investigate on the complexities associated with elastic scaling of an event processing system in a private/public cloud scenario. We develop an Elastic Switching Mechanism (ESM) which reduces the overall average latency of event processing jobs by significant amount considering the cost of operating the system. ESM is augmented with adaptive compressing of upstream data. The ESM conducts one of the two types of switching where either part of the data is sent to the public cloud (data switching) or a selected query is sent to the public cloud (query switching) based on the characteristics of the query. We model the operation of the ESM as the function of two binary switching functions. We show that our elastic switching mechanism with compression is capable of handling out-of-order events more efficiently compared to techniques which does not involve compression. We used two application benchmarks called EmailProcessor and a Social Networking Benchmark (SNB2016) to conduct multiple experiments to evaluate the effectiveness of our approach. In a single query deployment with EmailProcessor benchmark we observed that our elastic switching mechanism provides 1.24 seconds average latency improvement per processed event which is 16.70% improvement compared to private cloud only deployment. When presented the option of scaling EmailProcessor with four public cloud VMs ESM further reduced the average latency by 37.55% compared to the single public cloud VM. In a multi-query deployment with both EmailProcessor and SNB2016 we obtained a reduction of average latency of both the queries by 39.61 seconds which is a decrease of 7% of overall latency. These performance figures indicate that our elastic switching mechanism with compressed data streams can effectively reduce the average elapsed time of stream processing happening in private/public clouds.

References

[1]
M. Blount, M. Ebling, J. Eklund, A. James, C. McGregor, N. Percival, K. Smith, and D. Sow. Real-time analysis for intensive care: Development and deployment of the artemis analytic system. Engineering in Medicine and Biology Magazine, IEEE, 29(2):110--118, March 2010.
[2]
J. Cervino, E. Kalyvianaki, J. Salvachua, and P. Pietzuch. Adaptive provisioning of stream processing systems in the cloud. In Data Engineering Workshops (ICDEW), 2012 IEEE 28th International Conference on, pages 295--301, April 2012.
[3]
R. Cocci, T. Tran, Y. Diao, and P. Shenoy. Efficient data interpretation and compression over rfid streams. In 2008 IEEE 24th International Conference on Data Engineering, pages 1445--1447, April 2008.
[4]
A. Cuzzocrea and S. Chakravarthy. Event-based lossy compression for effective and efficient ØLAP over data streams. Data & Knowledge Engineering, 69(7):678--708, 2010. Advanced Knowledge-based Systems.
[5]
Datapath. Datapath.io. URL: http://console.datapath.io/map, 2016.
[6]
M. Dayarathna and T. Suzumura. Automatic optimization of stream programs via source program operator graph transformations. Distributed and Parallel Databases, 31(4):543--599, 2013.
[7]
M. Dayarathna and T. Suzumura. A Mechanism for Stream Program Performance Recovery in Resource Limited Compute Clusters, pages 164--178. Springer Berlin Heidelberg, Berlin, Heidelberg, 2013.
[8]
N. Dindar, Ã. Balkesen, K. Kromwijk, and N. Tatbul. Event processing support for cross-reality environments. IEEE Pervasive Computing, 8(3):34--41, July 2009.
[9]
C. Fehling, F. Leymann, R. Retter, W. Schupeck, and P. Arbitter. Cloud Computing Fundamentals, pages 21--78. Springer Vienna, Vienna, 2014.
[10]
C. Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the Forty-first Annual ACM Symposium on Theory of Computing, STOC '09, pages 169--178, New York, NY, USA, 2009. ACM.
[11]
S. Halevi and V. Shoup. Algorithms in HElib, pages 554--571. Springer Berlin Heidelberg, Berlin, Heidelberg, 2014.
[12]
J. Hazra, K. Das, D. P. Seetharam, and A. Singhee. Stream computing based synchrophasor application for power grids. In Proceedings of the First International Workshop on High Performance Computing, Networking and Analytics for the Power Grid, HiPCNA-PG '11, pages 43--50, New York, NY, USA, 2011. ACM.
[13]
S. Hemminger. Network emulation with netem. 2005.
[14]
W. Hummer, B. Satzger, and S. Dustdar. Elastic stream processing in the cloud. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(5):333--345, 2013.
[15]
T. Hunter, T. Das, M. Zaharia, P. Abbeel, and A. Bayen. Large-scale estimation in cyberphysical systems using streaming data: A case study with arterial traffic estimation. Automation Science and Engineering, IEEE Transactions on, 10(4):884--898, Oct 2013.
[16]
S. Jayasekara, S. Perera, M. Dayarathna, and S. Suhothayan. Continuous analytics on geospatial data streams with wso2 complex event processor. In Proceedings of the 9th ACM International Conference on Distributed Event-Based Systems, DEBS '15, pages 277--284, New York, NY, USA, 2015. ACM.
[17]
M. Jayasinghe, A. Jayawardena, B. Rupasinghe, M. Dayarathna, S. Perera, S. Suhothayan, and I. Perera. Continuous analytics on graph data streams using wso2 complex event processor. In Proceedings of the 10th ACM International Conference on Distributed and Event-based Systems, DEBS '16, pages 301--308, New York, NY, USA, 2016. ACM.
[18]
S. R. Jeffery, M. J. Franklin, and M. Garofalakis. An adaptive rfid middleware for supporting metaphysical data independence. The VLDB Journal, 17(2):265--289, 2008.
[19]
W. Kleiminger, E. Kalyvianaki, and P. Pietzuch. Balancing load in stream processing with the cloud. In Data Engineering Workshops (ICDEW), 2011 IEEE 27th International Conference on, pages 16--21, April 2011.
[20]
B. Klimt and Y. Yang. Introducing the enron corpus. In CEAS 2004 - First Conference on Email and Anti-Spam, July 30-31, 2004, Mountain View, California, USA, 2004.
[21]
S. Loesing, M. Hentschel, T. Kraska, and D. Kossmann. Stormy: An elastic and highly available streaming service in the cloud. In Proceedings of the 2012 Joint EDBT/ICDT Workshops, EDBT-ICDT '12, pages 55--60, New York, NY, USA, 2012. ACM.
[22]
C. Mutschler and M. Philippsen. Distributed low-latency out-of-order event processing for high data rate sensor streams. In Parallel Distributed Processing (IPDPS), 2013 IEEE 27th International Symposium on, pages 1133--1144, May 2013.
[23]
Z. Nabi, E. Bouillet, A. Bainbridge, and C. Thomas. Of streams and storms. IBM White Paper, 2014.
[24]
Y. Nie, R. Cocci, Z. Cao, Y. Diao, and P. Shenoy. Spire: Efficient data inference and compression over rfid streams. IEEE Transactions on Knowledge and Data Engineering, 24(1):141--155, Jan 2012.
[25]
A. Page, O. Kocabas, S. Ames, M. Venkitasubramaniam, and T. Soyata. Cloud-based secure health monitoring: Optimizing fully-homomorphic encryption for streaming algorithms. In 2014 IEEE Globecom Workshops (GC Wkshps), pages 48--52, Dec 2014.
[26]
B. Theeten, I. Bedini, P. Cogan, A. Sala, and T. Cucinotta. Towards the optimization of a parallel streaming engine for telco applications. Bell Labs Technical Journal, 18(4):181--197, 2014.
[27]
M. Thompson, D. Farley, M. Barker, P. Gee, and A. Stewart. High performance alternative to bounded queues for exchanging data between concurrent threads. technical paper, LMAX Exchange, 2011.
[28]
WSO2. Wso2 complex event processor. URL: http://wso2.com/products/complex-event-processor/, 2016.

Cited By

View all
  • (2021)Resource aware scheduler for distributed stream processing in cloud native environmentsConcurrency and Computation: Practice and Experience10.1002/cpe.637333:20Online publication date: 20-May-2021
  • (2019)Fast Coflow Scheduling via Traffic Compression and Stage Pipelining in Datacenter NetworksIEEE Transactions on Computers10.1109/TC.2019.293171668:12(1755-1771)Online publication date: 1-Dec-2019
  • (2019)Latency-Aware Secure Elastic Stream Processing with Homomorphic EncryptionData Science and Engineering10.1007/s41019-019-00100-5Online publication date: 12-Sep-2019
  • Show More Cited By

Index Terms

  1. Latency Aware Elastic Switching-based Stream Processing Over Compressed Data Streams

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering
        April 2017
        450 pages
        ISBN:9781450344043
        DOI:10.1145/3030207
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 17 April 2017

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. cloud computing
        2. compressed event processing
        3. data compression
        4. elastic data stream processing
        5. event-based systems
        6. iass
        7. software performance engineering
        8. system sizing and capacity planning

        Qualifiers

        • Research-article

        Conference

        ICPE '17
        Sponsor:

        Acceptance Rates

        ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;
        Overall Acceptance Rate 252 of 851 submissions, 30%

        Upcoming Conference

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)2
        • Downloads (Last 6 weeks)1
        Reflects downloads up to 19 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2021)Resource aware scheduler for distributed stream processing in cloud native environmentsConcurrency and Computation: Practice and Experience10.1002/cpe.637333:20Online publication date: 20-May-2021
        • (2019)Fast Coflow Scheduling via Traffic Compression and Stage Pipelining in Datacenter NetworksIEEE Transactions on Computers10.1109/TC.2019.293171668:12(1755-1771)Online publication date: 1-Dec-2019
        • (2019)Latency-Aware Secure Elastic Stream Processing with Homomorphic EncryptionData Science and Engineering10.1007/s41019-019-00100-5Online publication date: 12-Sep-2019
        • (2019)Privacy Preserving Elastic Stream Processing with Clouds Using Homomorphic EncryptionDatabase Systems for Advanced Applications10.1007/978-3-030-18579-4_16(264-280)Online publication date: 24-Apr-2019
        • (2018)Swallow: Joint Online Scheduling and Coflow Compression in Datacenter Networks2018 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS.2018.00060(505-514)Online publication date: May-2018
        • (2017)A Stepwise Auto-Profiling Method for Performance Optimization of Streaming ApplicationsACM Transactions on Autonomous and Adaptive Systems10.1145/313261812:4(1-33)Online publication date: 14-Nov-2017
        • (2017)Analysis and Autonomic Elasticity Control for Multi-Server Queues under Traffic Surges2017 International Conference on Cloud and Autonomic Computing (ICCAC)10.1109/ICCAC.2017.16(92-103)Online publication date: Sep-2017

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media