research-article

The non-expert tax: quantifying the cost of auto-scaling in cloud-based data stream analytics

Authors:

Vasiliki KalavriAuthors Info & Claims

BiDEDE '22: Proceedings of the International Workshop on Big Data in Emergent Distributed Environments

Article No.: 7, Pages 1 - 7

https://doi.org/10.1145/3530050.3532925

Published: 12 June 2022 Publication History

Abstract

All major Cloud providers offer Data Stream Analytics as managed services, allowing non-expert users to easily extract knowledge from data streams. These services lift the burden of job deployment and maintenance off users by provisioning resources automatically. However, this automation often comes at a cost, as auto-scaling policies may choose to over-provision to meet performance goals. We term this cost the Non-Expert Tax: the relative error between the ideal cost of deployment if an expert user would carefully configure resource allocation and the actual cost incurred by executing the same job with auto-scaling enabled. We conduct an empirical evaluation study of auto-scaling in two popular Cloud-based data stream analytics services and we find that they aggressively scale out, allocating resources quickly to meet high demand, but are conservative when scaling down, thus, charging users for underutilized resources. We quantify the Non-Expert Tax and show it can be as high as 544% for short-term jobs and up to 332% per month for periodic workloads.

References

[1]

Alibaba. 2022. Configure Autopilot for fully managed Flink. https://www.alibabacloud.com/help/en/doc-detail/173651.htm Last access: March 2022.

[2]

Amazon. 2022. Amazon Kinesis. https://aws.amazon.com/kinesis/ Last access: March 2022.

[3]

Amazon. 2022. Amazon Kinesis Data Analytics pricing. https://aws.amazon.com/kinesis/data-analytics/pricing/ Last access: March 2022.

[4]

Amazon. 2022. AWS Kinesis Data Analytics Automatic Scaling. https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-scaling.html#how-scaling-auto Last access: March 2022.

[5]

Apache. 2022. Apache Beam. https://beam.apache.org/ Last access: March 2022.

[6]

Apache. 2022. Apache Flink. https://flink.apache.org/ Last access: March 2022.

[7]

Apache Beam. 2022. Nexmark Benchmark Suite. https://beam.apache.org/documentation/sdks/java/testing/nexmark/ Last access: March 2022.

[8]

Paris Carbone, Marios Fragkoulis, Vasiliki Kalavri, and Asterios Katsifodimos. 2020. Beyond Analytics: The Evolution of Stream Processing Systems. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data (SIGMOD '20). 2651--2658.

Digital Library

[9]

Eli Cortez, Anand Bonde, Alexandre Muzio, Mark Russinovich, Marcus Fontoura, and Ricardo Bianchini. 2017. Resource Central: Understanding and Predicting Workloads for Improved Resource Management in Large Cloud Platforms. In Proceedings of the 26th Symposium on Operating Systems Principles (Shanghai, China) (SOSP '17). 153--167.

Digital Library

[10]

Miyuru Dayarathna and Srinath Perera. 2018. Recent Advancements in Event Processing. ACM Comput. Surv. 51, 2, Article 33 (2018).

Digital Library

[11]

Avrilia Floratou, Ashvin Agrawal, Bill Graham, Sriram Rao, and Karthik Ramasamy. 2017. Dhalion: Self-Regulating Stream Processing in Heron. Proc. VLDB Endow. 10, 12 (aug 2017), 1825--1836.

Digital Library

[12]

Marios Fragkoulis, Paris Carbone, Vasiliki Kalavri, and Asterios Katsifodimos. 2020. A Survey on the Evolution of Stream Processing Systems. (08 2020). https://arxiv.org/pdf/2008.00842.pdf

[13]

Gareth George, Rich Wolski, Chandra Krintz, and John Brevik. 2019. Analyzing AWS Spot Instance Pricing. In 2019 IEEE International Conference on Cloud Engineering (IC2E). 222--228.

[14]

Google. 2022. Autotuning features - Google Cloud Dataflow. https://cloud.google.com/dataflow/docs/guides/deploying-a-pipeline#autotuning-features Last access: March 2022.

[15]

Google. 2022. Google Cloud Dataflow. https://cloud.google.com/dataflow Last access: March 2022.

[16]

Google. 2022. SKU Groups - Dataflow - Google Cloud. https://cloud.google.com/skus/sku-groups/dataflow Last access: March 2022.

[17]

Vasiliki Kalavri, John Liagouris, Moritz Hoffmann, Desislava Dimitrova, Matthew Forshaw, and Timothy Roscoe. 2018. Three Steps is All You Need: Fast, Accurate, Automatic Scaling Decisions for Distributed Streaming Dataflows. In Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation.

[18]

Lin Ma, Dana Van Aken, Ahmed Hefny, Gustavo Mezerhane, Andrew Pavlo, and Geoffrey J. Gordon. 2018. Query-Based Workload Forecasting for Self-Driving Database Management Systems. In Proceedings of the 2018 International Conference on Management of Data (Houston, TX, USA) (SIGMOD '18). 631--645.

[19]

Yuan Mei, Luwei Cheng, and Vanish et al. Talwar. 2020. Turbine: Facebook's Service Management Platform for Stream Processing. In 2020 IEEE 36th International Conference on Data Engineering (ICDE). 1591--1602.

[20]

Microsoft. 2022. Autoscale Stream Analytics jobs using Azure Automation. https://docs.microsoft.com/en-us/azure/stream-analytics/stream-analytics-autoscale Last access: March 2022.

[21]

Ricky K. P. Mok, Hongyu Zou, Rui Yang, Tom Koch, Ethan Katz-Bassett, and K C Claffy. 2021. Measuring the Network Performance of Google Cloud Platform. In Proceedings of the 21st ACM Internet Measurement Conference. Association for Computing Machinery, 54--61.

Digital Library

[22]

Andy Pavlo. 2022. You are Overpaying Jeff Bezos for Your Databases (and the things he does with that extra money). https://ottertune.com/blog/overpaying-jeff-bezos-for-aws-databases/ Last access: March 2022.

[23]

Krzysztof Rzadca, Paweł Findeisen, and Jacek Świderski et al. 2020. Autopilot: Workload Autoscaling at Google Scale. In Proceedings of the Fifteenth European Conference on Computer Systems.

[24]

Huangshi Tian, Yunchuan Zheng, and Wei Wang. 2019. Characterizing and Synthesizing Task Dependencies of Data-Parallel Jobs in Alibaba Cloud. In Proceedings of the ACM Symposium on Cloud Computing (Santa Cruz, CA, USA) (SoCC '19). 139--151.

Digital Library

[25]

Pete Tucker, Kristin Tufte, Vassilis Papadimos, and David Maier. 2002. NEXMark---A Benchmark for Queries over Data Streams. Technical Report. OGI School of Science & Engineering at OHSU.

[26]

Le Xu, Boyang Peng, and Indranil Gupta. 2016. Stela: Enabling Stream Processing Systems to Scale-in and Scale-out On-demand. In 2016 IEEE International Conference on Cloud Engineering (IC2E). 22--31.

[27]

Tianyi Yu, Qingyuan Liu, Dong Du, Yubin Xia, Binyu Zang, Ziqian Lu, Pingchao Yang, Chenggang Qin, and Haibo Chen. 2020. Characterizing Serverless Platforms with Serverlessbench. In Proceedings of the 11th ACM Symposium on Cloud Computing (Virtual Event, USA) (SoCC '20). 30--44.

Digital Library

Cited By

Ezzeddine MBaude FHuet FLaaziz F(2025)Latency Aware and Resource-Efficient Bin Pack Autoscaling for Distributed Event Queues: Parameters Impact and SettingSN Computer Science10.1007/s42979-025-03740-96:3Online publication date: 24-Feb-2025
https://doi.org/10.1007/s42979-025-03740-9
Jamieson SForshaw MKemme BRiviere ESchiavoni VPasin M(2023)On Improving Streaming System Autoscaler Behaviour using Windowing and Weighting MethodsProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596886(68-79)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596886
Ezzeddine MMigliorini GBaude FHuet F(2022)Cost-Efficient and Latency-Aware Event Consuming in Workload-Skewed Distributed Event QueuesProceedings of the 2022 6th International Conference on Cloud and Big Data Computing10.1145/3555962.3555973(62-70)Online publication date: 18-Oct-2022
https://doi.org/10.1145/3555962.3555973

Index Terms

The non-expert tax: quantifying the cost of auto-scaling in cloud-based data stream analytics
1. Information systems
  1. Data management systems
    1. Database management system engines
      1. Stream management
2. Networks
  1. Network services
    1. Cloud computing

Recommendations

Heterogeneity-aware adaptive auto-scaling heuristic for improved QoS and resource usage in cloud environments

Cloud computing is a promising utility-based distributed computing environment in which resources (hardware/software) are offered as a service over the Internet on a pay per use basis. It involves elastic resource provisioning capabilities and hence ...
OCReM: OpenStack-based cloud datacentre resource monitoring and management scheme

Managing virtualised computing, network and storage resources at large-scale in both public and private cloud datacentres is a challenging task. As an open source cloud operating system, OpenStack needs to be enhanced for managing cloud datacentre ...
Prediction-Based Instant Resource Provisioning for Cloud Applications
UCC '14: Proceedings of the 2014 IEEE/ACM 7th International Conference on Utility and Cloud Computing

Dynamic provisioning of computing resources to fulfill the application requirement based on its current demand is one of the key challenges in cloud environment. However, availability of a resource to the application is not possible just by launching ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

BiDEDE '22: Proceedings of the International Workshop on Big Data in Emergent Distributed Environments

June 2022

77 pages

ISBN:9781450393461

DOI:10.1145/3530050

Conference Chairs:
Sven Groppe
University of Lübeck, Germany
,
Le Gruenwald
University of Oklahoma
,
Ching-Hsien Hsu
Asia University, Taiwan

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGMOD: ACM Special Interest Group on Management of Data

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 June 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGMOD/PODS '22

Sponsor:

SIGMOD

SIGMOD/PODS '22: International Conference on Management of Data

June 12, 2022

Pennsylvania, Philadelphia

Acceptance Rates

BiDEDE '22 Paper Acceptance Rate 10 of 15 submissions, 67%;

Overall Acceptance Rate 25 of 47 submissions, 53%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
166
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)1

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ezzeddine MBaude FHuet FLaaziz F(2025)Latency Aware and Resource-Efficient Bin Pack Autoscaling for Distributed Event Queues: Parameters Impact and SettingSN Computer Science10.1007/s42979-025-03740-96:3Online publication date: 24-Feb-2025
https://doi.org/10.1007/s42979-025-03740-9
Jamieson SForshaw MKemme BRiviere ESchiavoni VPasin M(2023)On Improving Streaming System Autoscaler Behaviour using Windowing and Weighting MethodsProceedings of the 17th ACM International Conference on Distributed and Event-based Systems10.1145/3583678.3596886(68-79)Online publication date: 27-Jun-2023
https://dl.acm.org/doi/10.1145/3583678.3596886
Ezzeddine MMigliorini GBaude FHuet F(2022)Cost-Efficient and Latency-Aware Event Consuming in Workload-Skewed Distributed Event QueuesProceedings of the 2022 6th International Conference on Cloud and Big Data Computing10.1145/3555962.3555973(62-70)Online publication date: 18-Oct-2022
https://doi.org/10.1145/3555962.3555973

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten