research-article

Towards a Resource Elasticity Benchmark for Cloud Environments

Authors:

Nikolas Herbst,

Henning Groenda,

Samuel KounevAuthors Info & Claims

HotTopiCS '14: Proceedings of the 2nd International Workshop on Hot Topics in Cloud service Scalability

Article No.: 5, Pages 1 - 8

https://doi.org/10.1145/2649563.2649571

Published: 22 March 2014 Publication History

Abstract

Auto-scaling features offered by today's cloud infrastructures provide increased flexibility especially for customers that experience high variations in the load intensity over time. However, auto-scaling features introduce new system quality attributes when considering their accuracy, timing, and boundaries. Therefore, distinguishing between different offerings has become a complex task, as it is not yet supported by reliable metrics and measurement approaches. In this paper, we discuss shortcomings of existing approaches for measuring and evaluating elastic behavior and propose a novel benchmark methodology specifically designed for evaluating the elasticity aspects of modern cloud platforms. The benchmark is based on open workloads with realistic load variation profiles that are calibrated to induce identical resource demand variations independent of the underlying hardware performance. Furthermore, we propose new metrics that capture the accuracy of resource allocations and de-allocations, as well as the timing aspects of an auto-scaling mechanism explicitly.

References

[1]

M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A. Rabkin, I. Stoica, and M. Zaharia. A View of Cloud Computing. Commun. ACM, 53(4):50--58, Apr. 2010.

Digital Library

[2]

C. Binnig, D. Kossmann, T. Kraska, and S. Loesing. How is the weather tomorrow?: towards a benchmark for the cloud. In Proceedings of the Second International Workshop on Testing Database Systems, DBTest '09, pages 9:1--9:6, New York, NY, USA, 2009. ACM.

Digital Library

[3]

T. Dory, B. Mejías, P. V. Roy, and N.-L. Tran. Measuring Elasticity for Cloud Databases. In Proceedings of the The Second International Conference on Cloud Computing, GRIDs, and Virtualization, 2011.

[4]

L. Duboc, D. S. Rosenblum, and T. Wicks. A framework for modelling and analysis of software systems scalability. In Proceedings of the 28th international conference on Software engineering, ICSE '06, pages 949--952, New York, NY, USA, 2006. ACM.

Digital Library

[5]

E. Folkerts, A. Alexandrov, K. Sachs, A. Iosup, V. Markl, and C. Tosun. Benchmarking in the Cloud: What It Should, Can, and Cannot Be. In R. Nambiar and M. Poess, editors, Selected Topics in Performance Evaluation and Benchmarking, volume 7755 of Lecture Notes in Computer Science, pages 173--188. Springer Berlin Heidelberg, 2012.

[6]

G. Galante and L. C. E. d. Bona. A Survey on Cloud Computing Elasticity. In Proceedings of the 2012 IEEE/ACM Fifth International Conference on Utility and Cloud Computing, UCC '12, pages 263--270, Washington, DC, USA, 2012. IEEE Computer Society.

Digital Library

[7]

N. R. Herbst, S. Kounev, and R. Reussner. Elasticity in Cloud Computing: What it is, and What it is Not (Short Paper). In Proceedings of the 10th International Conference on Autonomic Computing (ICAC 2013). USENIX, June 2013.

[8]

S. Islam, K. Lee, A. Fekete, and A. Liu. How a consumer can measure elasticity for cloud platforms. In Proceedings of the 3rd ACM/SPEC International Conference on Performance Engineering, ICPE '12, pages 85--96, New York, NY, USA, 2012. ACM.

Digital Library

[9]

B. Jennings and R. Stadler. Resource management in clouds: Survey and research challenges. Journal of Network and Systems Management, pages 1--53, 2014.

[10]

E. J. Keogh and M. J. Pazzani. Scaling up dynamic time warping to massive dataset. In Proceedings of the Third European Conference on Principles of Data Mining and Knowledge Discovery, PKDD '99, pages 1--11, London, UK, 1999.

Digital Library

[11]

M. Kuperberg, N. R. Herbst, J. G. von Kistowski, and R. Reussner. Defining and Quantifying Elasticity of Resources in Cloud Computing and Scalable Platforms. Technical report, Karlsruhe Institute of Technology (KIT), 2011.

[12]

A. Li, X. Yang, S. Kandula, and M. Zhang. CloudCmp: Comparing Public Cloud Providers. In Proceedings of the 10th ACM SIGCOMM Conference on Internet Measurement, IMC '10, pages 1--14, New York, NY, USA, 2010. ACM.

Digital Library

[13]

M. Salehie and L. Tahvildari. Self-adaptive Software: Landscape and Research Challenges. ACM Trans. Auton. Adapt. Syst., 4(2):14:1--14:42, May 2009.

Digital Library

[14]

P. Shivam, V. Marupadi, J. Chase, T. Subramaniam, and S. Babu. Cutting corners: Workbench automation for server benchmarking. In USENIX 2008 Annual Technical Conference on Annual Technical Conference, ATC'08, pages 241--254, Berkeley, CA, USA, 2008. USENIX Association.

Digital Library

[15]

M. van Steen, S. Van der Zijden, and H. J. Sips. Software engineering for the scalable distributed applications. In Computer Software and Applications Conference, 1998. COMPSAC '98. Proceedings., pages 285--292, 1998.

Digital Library

[16]

J. G. von Kistowski, N. R. Herbst, and S. Kounev. LIMBO: A Tool For Modeling Variable Load Intensities (Demo Paper). In Proceedings of the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014). ACM, March 2014.

Digital Library

[17]

J. G. von Kistowski, N. R. Herbst, and S. Kounev. Modeling Variations in Load Intensity over Time. In Proceedings of the 3rd International Workshop on Large-Scale Testing (LT 2014), co-located with the 5th ACM/SPEC International Conference on Performance Engineering (ICPE 2014). ACM, March 2014.

Digital Library

Cited By

Henning SHasselbring W(2022)A configurable method for benchmarking scalability of cloud-native applicationsEmpirical Software Engineering10.1007/s10664-022-10162-127:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10162-1
Henning SHasselbring WBourcier JJiang ZBezemer CCortellessa V(2021)How to Measure Scalability of Distributed Stream Processing Engines?Companion of the ACM/SPEC International Conference on Performance Engineering10.1145/3447545.3451190(85-88)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3447545.3451190
Feng DWu ZZuo DZhang Z(2020)Auto-Scaling Provision Basing on Workload Prediction in the Virtualized Data CenterInternational Journal of Grid and High Performance Computing10.4018/IJGHPC.202001010412:1(53-69)Online publication date: Jan-2020
https://doi.org/10.4018/IJGHPC.2020010104
Show More Cited By

Index Terms

Towards a Resource Elasticity Benchmark for Cloud Environments

Recommendations

Managing elasticity across multiple cloud providers
MultiCloud '13: Proceedings of the 2013 international workshop on Multi-cloud applications and federated clouds

In the context of cloud computing, elasticity is the capacity to scale computing resources up and down easily. Currently, most Platforms as a Service (PaaS) manage application elasticity within a single cloud provider. However, the not so infrequent ...
Cloud Elasticity: going beyond demand as user load
ARMS-CC'16: Proceedings of the Third International Workshop on Adaptive Resource Management and Scheduling for Cloud Computing

Cloud computing systems have become not only popular, but extensively used. They are supported and exploited by both industry and academia. Cloud providers have diversified and so did the software offered by their systems. Infrastructure as a Service (...
An Experimental Performance Evaluation of Autoscaling Policies for Complex Workflows
ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Simplifying the task of resource management and scheduling for customers, while still delivering complex Quality-of-Service (QoS), is key to cloud computing. Many autoscaling policies have been proposed in the past decade to decide on behalf of cloud ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

HotTopiCS '14: Proceedings of the 2nd International Workshop on Hot Topics in Cloud service Scalability

March 2014

39 pages

ISBN:9781450330596

DOI:10.1145/2649563

Copyright © 2014 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

SAP
SPEC: SPEC Research Group

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 March 2014

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

HotTopiCS '14

HotTopiCS '14: International Workshop on Hot Topics in Cloud service Scalability

March 22, 2014

Dublin, Ireland

Acceptance Rates

Overall Acceptance Rate 10 of 15 submissions, 67%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
190
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Henning SHasselbring W(2022)A configurable method for benchmarking scalability of cloud-native applicationsEmpirical Software Engineering10.1007/s10664-022-10162-127:6Online publication date: 1-Nov-2022
https://dl.acm.org/doi/10.1007/s10664-022-10162-1
Henning SHasselbring WBourcier JJiang ZBezemer CCortellessa V(2021)How to Measure Scalability of Distributed Stream Processing Engines?Companion of the ACM/SPEC International Conference on Performance Engineering10.1145/3447545.3451190(85-88)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3447545.3451190
Feng DWu ZZuo DZhang Z(2020)Auto-Scaling Provision Basing on Workload Prediction in the Virtualized Data CenterInternational Journal of Grid and High Performance Computing10.4018/IJGHPC.202001010412:1(53-69)Online publication date: Jan-2020
https://doi.org/10.4018/IJGHPC.2020010104
Kuhlenkamp JWerner SBorges MErnst D(2020)All but oneACM SIGAPP Applied Computing Review10.1145/3429204.342920520:3(5-19)Online publication date: 8-Oct-2020
https://dl.acm.org/doi/10.1145/3429204.3429205
Funaro LBen-Yehuda OSchuster A(2020)Memory Elasticity BenchmarkProceedings of the 13th ACM International Systems and Storage Conference10.1145/3383669.3398277(1-12)Online publication date: 30-May-2020
https://dl.acm.org/doi/10.1145/3383669.3398277
Kuhlenkamp JWerner SBorges MErnst DWenzel DHung CCerny TShin DBechini A(2020)Benchmarking elasticity of FaaS platforms as a foundation for objective-driven design of serverless applicationsProceedings of the 35th Annual ACM Symposium on Applied Computing10.1145/3341105.3373948(1576-1585)Online publication date: 30-Mar-2020
https://dl.acm.org/doi/10.1145/3341105.3373948
Dupont SBouri SAlvares FLedoux TShin SShin DLencastre M(2017)ElaScriptProceedings of the Symposium on Applied Computing10.1145/3019612.3019631(392-398)Online publication date: 3-Apr-2017
https://dl.acm.org/doi/10.1145/3019612.3019631
Yazhou Hu Bo Deng Fuyang Peng Bin Hong Yuchao Zhang Dongxia Wang (2016)A survey on evaluating elasticity of cloud computing platform2016 World Automation Congress (WAC)10.1109/WAC.2016.7583052(1-4)Online publication date: Jul-2016
https://doi.org/10.1109/WAC.2016.7583052
Hu YDeng BYang YWang D(2016)Elasticity Evaluation of IaaS Cloud Based on Mixed Workloads2016 15th International Symposium on Parallel and Distributed Computing (ISPDC)10.1109/ISPDC.2016.28(157-164)Online publication date: 2016
https://doi.org/10.1109/ISPDC.2016.28
Rosa Righi RCosta CRodrigues VRostirolla G(2016)Joint-analysis of performance and energy consumption when enabling cloud elasticity for synchronous HPC applicationsConcurrency and Computation: Practice & Experience10.1002/cpe.371028:5(1548-1571)Online publication date: 10-Apr-2016
https://dl.acm.org/doi/10.1002/cpe.3710
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten