research-article

Heterogeneity and dynamicity of clouds at scale: Google trace analysis

Authors:

Alexey Tumanov,

Gregory R. Ganger,

Michael A. KozuchAuthors Info & Claims

SoCC '12: Proceedings of the Third ACM Symposium on Cloud Computing

Article No.: 7, Pages 1 - 13

https://doi.org/10.1145/2391229.2391236

Published: 14 October 2012 Publication History

Abstract

To better understand the challenges in developing effective cloud-based resource schedulers, we analyze the first publicly available trace data from a sizable multi-purpose cluster. The most notable workload characteristic is heterogeneity: in resource types (e.g., cores:RAM per machine) and their usage (e.g., duration and resources needed). Such heterogeneity reduces the effectiveness of traditional slot- and core-based scheduling. Furthermore, some tasks are constrained as to the kind of machine types they can use, increasing the complexity of resource assignment and complicating task migration. The workload is also highly dynamic, varying over time and most workload features, and is driven by many short jobs that demand quick scheduling decisions. While few simplifying assumptions apply, we find that many longer-running jobs have relatively stable resource utilizations, which can help adaptive resource schedulers.

References

[1]

ISTC-CC: Research Overview, 2012. http://www.istc-cc.cmu.edu/research.

[2]

B. Abrahao and A. Zhang. Characterizing application workloads on CPU utilization for utility computing. Technical Report HPL-2004-157, HP Labs, 2004.

[3]

P. Bodik, A. Fox, M. J. Franklin, M. I. Jordan, and D. A. Patterson. Characterizing, modeling, and generating workload spikes for stateful services. In Proc. of the 1st ACM Symposium on Cloud Computing, 2010.

Digital Library

[4]

Y. Chen, S. Alspaugh, and R. H. Katz. Design insights for mapreduce from diverse production workloads. Technical Report UCB/EECS-2012-17, EECS Dept, University of California, Berkeley, Jan 2012.

[5]

Y. Chen, A. Ganapathi, R. Griffith, and R. Katz. The case for evaluating MapReduce performance using workload suites. In Proc. of 19th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems (MASCOTS), pages 390--399, July 2011.

Digital Library

[6]

A. Clauset, C. R. Shalizi, and M. E. J. Newman. Power-law distributions in empirical data. SIAM Rev., 51(4): 661--703, Nov. 2009.

Digital Library

[7]

J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1): 107--113, 2008.

Digital Library

[8]

S. Di, D. Kondo, and W. Cirne. Characterization and Comparison of Google Cloud Load versus Grids. http://hal.archives-ouvertes.fr/hal-00705858, 2012.

[9]

A. B. Downey and D. G. Feitelson. The elusive goal of workload characterization. SIGMETRICS Perform. Eval. Rev., 26: 14--29, March 1999.

Digital Library

[10]

M. Isard, M. Budiu, Y. Yu, A. Birrell, and D. Fetterly. Dryad: distributed data-parallel programs from sequential building blocks. In Proc. of the 2nd ACM SIGOPS/EuroSys European Conference on Computer Systems, 2007.

Digital Library

[11]

S. Kavulya, J. Tan, R. Gandhi, and P. Narasimhan. An analysis of traces from a production MapReduce cluster. In Proc. of the 10th IEEE/ACM International Conference on Cluster, Cloud and Grid Computing, May 2010.

Digital Library

[12]

M. Kozuch, M. Ryan, R. Gass, S. Schlosser, D. O'Hallaron, J. Cipar, E. Krevat, J. López, M. Stroucken, and G. Ganger. Tashi: location-aware cluster management. In Proc. of the 1st Workshop on Automated Control for Datacenters and Clouds, 2009.

Digital Library

[13]

Z. Liu and S. Cho. Characterizing machines and workloads on a Google cluster. In Eighth International Workshop on Scheduling and Resource Management for Parallel and Distributed Systems, September 2012.

Digital Library

[14]

U. Lublin and D. Feitelson. The workload on parallel supercomputers: modeling the characteristics of rigid jobs. Journal of Parallel and Distributed Computing, 63(11): 1105--1122, 2003.

Digital Library

[15]

A. Oliner and J. Stearley. What supercomputers say: a study of five system logs. In Proc. of the 37th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN '07), pages 575--584, June 2007.

Digital Library

[16]

N. Poggi, D. Carrera, R. Gavalda, J. Torres, and E. Ayguadé. Characterization of workload and resource consumption for an online travel and booking site. In Proc. of the IEEE International Symposium on Workload Characterization (IISWC), 2010.

Digital Library

[17]

C. Reiss, A. Tumanov, G. R. Ganger, R. H. Katz, and M. A. Kozuch. Towards understanding heterogeneous clouds at scale: Google trace analysis. Technical Report ISTC-CC-TR-12-101, Intel Science and Technology Center for Cloud Computing, Apr 2012.

[18]

C. Reiss, J. Wilkes, and J. L. Hellerstein. Google cluster-usage traces: format + schema, 2011. http://goo.gl/5uJri.

[19]

C. Reiss, J. Wilkes, and J. L. Hellerstein. Obfuscatory obscanturism: making workload traces of commercially-sensitive systems safe to release. In Proc. of IEEE/IFIP International Workshop on Cloud Management (CloudMan'12), April 2012.

[20]

M. Schwarzkopf, D. G. Murray, and S. Hand. The seven deadly sins of cloud computing research. In Proc. of the 4th USENIX Workshop on Hot Topics in Cloud Computing, 2012.

Digital Library

[21]

B. Sharma, V. Chudnovsky, J. L. Hellerstein, R. Rifaat, and C. R. Das. Modeling and synthesizing task placement constraints in Google compute clusters. In Proc. of the 2nd ACM Symposium on Cloud Computing, SOCC '11, 2011.

Digital Library

[22]

A. Tumanov, J. Cipar, M. A. Kozuch, and G. R. Ganger. alsched: algebraic scheduling of mixed workloads in heterogeneous clouds. In Proc. of the 3nd ACM Symposium on Cloud Computing, SOCC '12, 2012.

Digital Library

[23]

J. Wilkes. Omega: Cluster management at google. Video from 2011 Google Faculty Summit; https://youtu.be/0ZFMlO98Jkc, July 2011.

[24]

J. Wilkes and C. Reiss. Details of the ClusterData-2011-1 trace, 2011. https://code.google.com/p/googleclusterdata/wiki/ClusterData2011_1.

[25]

M. Zaharia, B. Hindman, A. Konwinski, A. Ghodsi, A. Joseph, R. Katz, S. Shenker, and I. Stoica. The datacenter needs an operating system. In HotCloud, June 2011.

Digital Library

[26]

Q. Zhang, M. F. Zhani, S. Zhang, Q. Zhu, R. Boutaba, and J. L. Hellerstein. Dynamic energy-aware capacity provisioning for cloud computing enviornments. In Proceedings of the 9th ACM International Conference on Autonomic Computing, 2012.

Digital Library

[27]

Z. Zheng, L. Yu, W. Tang, Z. Lan, R. Gupta, N. Desai, S. Coghlan, and D. Buettner. Co-analysis of RAS log and job log on Blue Gene/P. In Proc. of IEEE International Parallel Distributed Processing Symposium (IPDPS), May 2011.

Digital Library

Cited By

Shailin Saraiya (2025)Technical Evolution and Performance Analysis of MapReduce in Modern Distributed SystemsInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology10.32628/CSEIT2511120611:1(29-35)Online publication date: 3-Jan-2025
https://doi.org/10.32628/CSEIT25111206
Chippagiri S(2025)Optimization of Kubernetes for the High-Performance Computing with Kubernetes, Performance Analysis, and Dynamic Workload Placement toward the Enhancement of Cloud ComputingSSRN Electronic Journal10.2139/ssrn.5073127Online publication date: 2025
https://doi.org/10.2139/ssrn.5073127
Reidys BZardoshti PGoiri ÍIrvene CBerger DMa HArya KCortez EStark TBak EIyigun MNovakovic SHsu LTrueba KPan ABansal CRajmohan SHuang JBianchini REeckhout LSmaragdakis GLiang KSampson AKim MRossbach C(2025)Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud PlatformsProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707226(164-181)Online publication date: 30-Mar-2025
https://dl.acm.org/doi/10.1145/3669940.3707226
Show More Cited By

Index Terms

Heterogeneity and dynamicity of clouds at scale: Google trace analysis
1. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles

Recommendations

Resource provisioning and scheduling in clouds: QoS perspective

Resource provisioning of appropriate resources to cloud workloads depends on the quality of service (QoS) requirements of cloud applications and is a challenging task. In cloud environment, heterogeneity, uncertainty and dispersion of resources ...
Heterogeneity and Dynamicity of Clouds at Scale: Google Trace Analysis
SoCC '21: Proceedings of the ACM Symposium on Cloud Computing

Test of Time Award Talk for Heterogeneity and Dynamicity of Clouds at Scale: Google Trace Analysis, SoCC 2012.
Is the Same Instance Type Created Equal? Exploiting Heterogeneity of Public Clouds

Public cloud platforms might start with homogeneous hardware; nevertheless, because of inevitable hardware upgrades, or adding more capacity, the initial homogeneous platform will gradually evolve into heterogeneous as time passes by. The consequent ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SoCC '12: Proceedings of the Third ACM Symposium on Cloud Computing

October 2012

325 pages

ISBN:9781450317610

DOI:10.1145/2391229

Program Chairs:
Michael Carey
UC Irvine
,
Steven Hand
University of Cambridge

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 October 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Conference

SOCC '12

Sponsor:

SOCC '12: ACM Symposium on Cloud Computing

October 14 - 17, 2012

California, San Jose

Acceptance Rates

Overall Acceptance Rate 169 of 722 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

771
Total Citations
View Citations
3,935
Total Downloads

Downloads (Last 12 months)285
Downloads (Last 6 weeks)30

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Shailin Saraiya (2025)Technical Evolution and Performance Analysis of MapReduce in Modern Distributed SystemsInternational Journal of Scientific Research in Computer Science, Engineering and Information Technology10.32628/CSEIT2511120611:1(29-35)Online publication date: 3-Jan-2025
https://doi.org/10.32628/CSEIT25111206
Chippagiri S(2025)Optimization of Kubernetes for the High-Performance Computing with Kubernetes, Performance Analysis, and Dynamic Workload Placement toward the Enhancement of Cloud ComputingSSRN Electronic Journal10.2139/ssrn.5073127Online publication date: 2025
https://doi.org/10.2139/ssrn.5073127
Reidys BZardoshti PGoiri ÍIrvene CBerger DMa HArya KCortez EStark TBak EIyigun MNovakovic SHsu LTrueba KPan ABansal CRajmohan SHuang JBianchini REeckhout LSmaragdakis GLiang KSampson AKim MRossbach C(2025)Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud PlatformsProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707226(164-181)Online publication date: 30-Mar-2025
https://dl.acm.org/doi/10.1145/3669940.3707226
Olliaro DAnggraito AMarsan MBalsamo SMarin A(2025)The Impact of Service Demand Variability on Data Center PerformanceIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2024.349779236:2(120-132)Online publication date: Feb-2025
https://doi.org/10.1109/TPDS.2024.3497792
Landré DPhilippe LPierson J(2025)Sufficiency power consideration to run a workload on renewable energy operated datacenterFuture Generation Computer Systems10.1016/j.future.2025.107710167(107710)Online publication date: Jun-2025
https://doi.org/10.1016/j.future.2025.107710
Wu JXu MHe YYe KXu C(2025)Cloudnativesim: A Toolkit for Modeling and Simulation of Cloud‐Native ApplicationsSoftware: Practice and Experience10.1002/spe.3417Online publication date: 23-Feb-2025
https://doi.org/10.1002/spe.3417
Nair AKumar SPrasad AHuang YRudoff ASubramoney SBagchi SZhang Y(2024)TelescopeProceedings of the 2024 USENIX Conference on Usenix Annual Technical Conference10.5555/3691992.3692018(409-424)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691992.3692018
Lazarev NGohil VTsai JAnderson AChitlur BZhang ZDelimitrou CGavrilovska ATerry D(2024)SabreProceedings of the 18th USENIX Conference on Operating Systems Design and Implementation10.5555/3691938.3691939(1-18)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.5555/3691938.3691939
Wrona ZGanzha MPaprzycki MKrzyżanowski S(2024)Dynamic Knowledge Management in an Agent-Based Extended Green Cloud SimulatorEnergies10.3390/en1704078017:4(780)Online publication date: 6-Feb-2024
https://doi.org/10.3390/en17040780
Wang QLan TTang YSang BHuang ZDu YZhang HSha JLu HZhou YZhang KTang M(2024)DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the CloudProceedings of the VLDB Endowment10.14778/3685800.368583217:12(4130-4144)Online publication date: 8-Nov-2024
https://dl.acm.org/doi/10.14778/3685800.3685832
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten