short-paper

Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments

Authors:

Ali Abedi,

Tim BrechtAuthors Info & Claims

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

Pages 287 - 292

https://doi.org/10.1145/3030207.3030229

Published: 17 April 2017 Publication History

Get Access

Abstract

Previous work has shown that benchmark and application performance in public cloud computing environments can be highly variable. Utilizing Amazon EC2 traces that include measurements affected by CPU, memory, disk, and network performance, we study commonly used methodologies for comparing performance measurements in cloud computing environments. The results show considerable flaws in these methodologies that may lead to incorrect conclusions. For instance, these methodologies falsely report that the performance of two identical systems differ by 38% using a confidence level of 95%. We then study the efficacy of the Randomized Multiple Interleaved Trials (RMIT) methodology using the same traces. We demonstrate that RMIT could be used to conduct repeatable experiments that enable fair comparisons in this cloud computing environment despite the fact that changing conditions beyond the user's control make comparing competing alternatives highly challenging.

References

[1]

http://cs.uwaterloo.ca/brecht/data/icpe-rmit-2017.

Google Scholar

[2]

ABEDI, A., HEARD, A., AND BRECHT, T. Conducting repeatable experiments and fair comparisons using 802.11N MIMO networks. SIGOPS Oper. Syst. Rev. 49, 1 (2015).

Digital Library

Google Scholar

[3]

BALLANI, H., COSTA, P., KARAGIANNIS, T., AND ROWSTRON, A. Towards predictable datacenter networks. In SIGCOMM (2011).

Digital Library

Google Scholar

[4]

BALLANI, H., JANG, K., KARAGIANNIS, T., KIM, C., GUNAWARDENA, D., AND O'SHEA, G. Chatty tenants and the cloud network sharing problem. In NSDI(2013).

Google Scholar

[5]

CARDOSA, M., WANG, C., NANGIA, A., CHANDRA, A., AND WEISSMAN, J. Exploring MapReduce efficiency with highly-distributed data. In MapReduce (2011).

Digital Library

Google Scholar

[6]

FARLEY, B., JUELS, A., VARADARAJAN, V., RISTENPART, T., BOWERS, K. D., AND SWIFT, M. M. More for your money: Exploiting performance heterogeneity in public clouds. In SoCC (2012).

Google Scholar

[7]

IORDACHE, A., MORIN, C., PARLAVANTZAS, N., FELLER, E., AND RITEAU, P. Resilin: Elastic MapReduce over multiple clouds. In CCGrid (2013).

Google Scholar

[8]

IOSUP, A., YIGITBASI, N., AND EPEMA, D. On the performance variability of production cloud services. In CCGRID (2011).

Digital Library

Google Scholar

[9]

JIANG, D., OOI, B., SHI, L., AND WU, S. The performance of MapReduce: An in-depth study. Proc. VLDB Endow. 3, 1--2 (2010), 472--483.

Digital Library

Google Scholar

[10]

JUVE, G., DEELMAN, E., BERRIMAN, G. B., BERMAN, B. P., AND M AECHLING, P. An evaluation of the cost and performance of scientific workflows on Amazon EC2. J. Grid Comput. 10, 1 (2012).

Digital Library

Google Scholar

[11]

LEITNER, P., AND CITO, J. Patterns in the chaos - a study of performance variation and predictability in public iaas clouds. ACM Trans. Internet Technol. 16, 3 (2016).

Digital Library

Google Scholar

[12]

MEHROTRA, P., D JOMEHRI, J., HEISTAND, S., HOOD, R., JIN, H., LAZANOFF, A., SAINI, S., AND BISWAS, R. Performance evaluation of Amazon EC2 for NASA HPC applications. In ScienceCloud (2012).

Google Scholar

[13]

MONTGOMERY, D. Design and Analysis of Experiments. Wiley, 2012.

Digital Library

Google Scholar

[14]

SCHAD, J., DITTRICH, J., AND QUIANÉ-RUIZ, J.-A. Runtime Measurements in the Cloud: Observing, Analyzing, and Reducing Variance. Proc. VLDB Endow. (2010)

Digital Library

Google Scholar

Cited By

View all

Dixon IForshaw MMatthews J(2025)Robust Streaming Benchmark Design in the Presence of BackpressureComputer Performance Engineering10.1007/978-3-031-80932-3_10(137-152)Online publication date: 13-Feb-2025
https://doi.org/10.1007/978-3-031-80932-3_10
Bauer AGonthier MPan HChard RGrzenda DStraesser MPauloski JKamatar ABaughman MHudson NFoster IChard K(2024)An Empirical Investigation of Container Building Strategies and Warm Times to Reduce Cold Starts in Scientific Computing Serverless Functions2024 IEEE 20th International Conference on e-Science (e-Science)10.1109/e-Science62913.2024.10678668(1-10)Online publication date: 16-Sep-2024
https://doi.org/10.1109/e-Science62913.2024.10678668
Rese TJapke NKoch SPfandzelter TBermbach D(2024)Increasing Efficiency and Result Reliability of Continuous Benchmarking for FaaS Applications2024 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E61754.2024.00017(93-100)Online publication date: 24-Sep-2024
https://doi.org/10.1109/IC2E61754.2024.00017
Show More Cited By

Index Terms

Conducting Repeatable Experiments in Highly Variable Cloud Computing Environments
1. General and reference
  1. Cross-computing tools and techniques
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles
        Cloud computing

Recommendations

Analysis of Cloud Computing Delivery Architecture Models
WAINA '11: Proceedings of the 2011 IEEE Workshops of International Conference on Advanced Information Networking and Applications

Cloud computing is one of the emerging technologies that will lead to the next generation of Internet. It provides optimized and efficient computing through enhanced collaboration, agility, scalability, and availability. In this paper, the evolution of ...
CloudCmp: comparing public cloud providers
IMC '10: Proceedings of the 10th ACM SIGCOMM conference on Internet measurement

While many public cloud providers offer pay-as-you-go computing, their varying approaches to infrastructure, virtualization, and software services lead to a problem of plenty. To help customers pick a cloud that fits their needs, we develop CloudCmp, a ...
An early comparison of commercial and open-source cloud platforms for scientific environments
KES-AMSTA'12: Proceedings of the 6th KES international conference on Agent and Multi-Agent Systems: technologies and applications

Cloud computing promises efficient use of hardware resources through virtualization and elastic computing facilities. Various cloud computing solutions have emerged on the market from open-source communities and commercial vendors. In this paper we ...

Comments

Information & Contributors

Information

Published In

ICPE '17: Proceedings of the 8th ACM/SPEC on International Conference on Performance Engineering

April 2017

450 pages

ISBN:9781450344043

DOI:10.1145/3030207

General Chairs:
Walter Binder
Università della Svizzera Italiana, Switzerland
,
Vittorio Cortellessa
Università dell'Aquila, Italy
,
Program Chairs:
Anne Koziolek
Karlsruhe Institute of Technology, Germany
,
Evgenia Smirni
College of William & Mary, USA
,
Meikel Poess
Oracle Corporation, USA

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 17 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Natural Sciences and Engineering Research Council of Canada (NSERC)

Conference

ICPE '17

Sponsor:

ICPE '17: ACM/SPEC International Conference on Performance Engineering

April 22 - 26, 2017

L'Aquila, Italy

Acceptance Rates

ICPE '17 Paper Acceptance Rate 27 of 83 submissions, 33%;

Overall Acceptance Rate 252 of 851 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
298
Total Downloads

Downloads (Last 12 months)42
Downloads (Last 6 weeks)8

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Dixon IForshaw MMatthews J(2025)Robust Streaming Benchmark Design in the Presence of BackpressureComputer Performance Engineering10.1007/978-3-031-80932-3_10(137-152)Online publication date: 13-Feb-2025
https://doi.org/10.1007/978-3-031-80932-3_10
Bauer AGonthier MPan HChard RGrzenda DStraesser MPauloski JKamatar ABaughman MHudson NFoster IChard K(2024)An Empirical Investigation of Container Building Strategies and Warm Times to Reduce Cold Starts in Scientific Computing Serverless Functions2024 IEEE 20th International Conference on e-Science (e-Science)10.1109/e-Science62913.2024.10678668(1-10)Online publication date: 16-Sep-2024
https://doi.org/10.1109/e-Science62913.2024.10678668
Rese TJapke NKoch SPfandzelter TBermbach D(2024)Increasing Efficiency and Result Reliability of Continuous Benchmarking for FaaS Applications2024 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E61754.2024.00017(93-100)Online publication date: 24-Sep-2024
https://doi.org/10.1109/IC2E61754.2024.00017
Schirmer TPfandzelter TBermbach D(2024)ElastiBench: Scalable Continuous Benchmarking on Cloud FaaS Platforms2024 IEEE International Conference on Cloud Engineering (IC2E)10.1109/IC2E61754.2024.00016(83-92)Online publication date: 24-Sep-2024
https://doi.org/10.1109/IC2E61754.2024.00016
Grambow MDockenfuß TSchirmer TJapke NBermbach D(2023)Efficiently Detecting Performance Changes in FaaS Application ReleasesProceedings of the 9th International Workshop on Serverless Computing10.1145/3631295.3631395(13-17)Online publication date: 11-Dec-2023
https://dl.acm.org/doi/10.1145/3631295.3631395
Laaber CYue TAli SSchwitalla TNygård JChandra SBlincoe KTonella P(2023)Automated Test Generation for Medical Rules Web Services: A Case Study at the Cancer Registry of NorwayProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613882(1937-1948)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613882
Japke NWitzko CGrambow MBermbach DFu SCarnevale LRana OVillari M(2023)The Early Microbenchmark Catches the Bug -- Studying Performance Issues Using Micro- and Application BenchmarksProceedings of the IEEE/ACM 16th International Conference on Utility and Cloud Computing10.1145/3603166.3632128(1-10)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3603166.3632128
Hasenoot JRellermeyer JUta AVieira MCardellini VDi Marco ATuma P(2023)The Performance of Distributed Applications: A Traffic Shaping PerspectiveProceedings of the 2023 ACM/SPEC International Conference on Performance Engineering10.1145/3578244.3583733(207-220)Online publication date: 15-Apr-2023
https://dl.acm.org/doi/10.1145/3578244.3583733
Straesser MBauer ALeppich RHerbst NChard KFoster IKounev S(2023)An Empirical Study of Container Image Configurations and Their Impact on Start Times2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)10.1109/CCGrid57682.2023.00019(94-105)Online publication date: May-2023
https://doi.org/10.1109/CCGrid57682.2023.00019
Baresi LDolci TQuattrocchi GRasi N(2023)A multi‐faceted analysis of the performance variability of virtual machinesSoftware: Practice and Experience10.1002/spe.324453:11(2067-2091)Online publication date: 24-Jul-2023
https://doi.org/10.1002/spe.3244
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Cited By

Index Terms

Recommendations

Analysis of Cloud Computing Delivery Architecture Models

CloudCmp: comparing public cloud providers

An early comparison of commercial and open-source cloud platforms for scientific environments