research-article

Omega: flexible, scalable schedulers for large compute clusters

Authors:

Malte Schwarzkopf,

Andy Konwinski,

Michael Abd-El-Malek,

John WilkesAuthors Info & Claims

EuroSys '13: Proceedings of the 8th ACM European Conference on Computer Systems

Pages 351 - 364

https://doi.org/10.1145/2465351.2465386

Published: 15 April 2013 Publication History

Abstract

Increasing scale and the need for rapid response to changing requirements are hard to meet with current monolithic cluster scheduler architectures. This restricts the rate at which new features can be deployed, decreases efficiency and utilization, and will eventually limit cluster growth. We present a novel approach to address these needs using parallelism, shared state, and lock-free optimistic concurrency control.

We compare this approach to existing cluster scheduler designs, evaluate how much interference between schedulers occurs and how much it matters in practice, present some techniques to alleviate it, and finally discuss a use case highlighting the advantages of our approach -- all driven by real-life Google production workloads.

References

[1]

Adaptive Computing Enterprises Inc. Maui Scheduler Administrator's Guide, 3.2 ed. Provo, UT, 2011.

[2]

Adl-Tabatabai, A.-R., Lewis, B. T., Menon, V., Murphy, B. R., Saha, B., and Shpeisman, T. Compiler and runtime support for efficient software transactional memory. In Proceedings of PLDI (2006), pp. 26--37.

Digital Library

[3]

Ananthanarayanan, G., Douglas, C., Ramakrishnan, R., Rao, S., and Stoica, I. True elasticity in multitenant data-intensive compute clusters. In Proceedings of SoCC (2012), p. 24.

Digital Library

[4]

Apache. Hadoop On Demand. http://goo.gl/px8Yd, 2007. Accessed 20/06/2012.

[5]

Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., and Gruber, R. E. Bigtable: A Distributed Storage System for Structured Data. ACM Transactions on Computer Systems 26, 2 (June 2008), 4:1--4:26.

Digital Library

[6]

Chen, Y., Alspaugh, S., Borthakur, D., and Katz, R. Energy efficiency for large-scale MapReduce workloads with significant interactive analysis. In Proceedings of EuroSys (2012).

Digital Library

[7]

Chen, Y., Ganapathi, A. S., Griffith, R., and Katz, R. H. Design insights for MapReduce from diverse production workloads. Tech. Rep. UCB/EECS-2012-17, UC Berkeley, Jan. 2012.

[8]

Dean, J., and Ghemawat, S. MapReduce: Simplified data processing on large clusters. CACM 51, 1 (2008), 107--113.

Digital Library

[9]

Engler, D. R., Kaashoek, M. F., and O'Toole, Jr., J. Exokernel: an operating system architecture for application-level resource management. In Proceedings of SOSP (1995), pp. 251--266.

Digital Library

[10]

Ferguson, A. D., Bodik, P., Kandula, S., Boutin, E., and Fonseca, R. Jockey: guaranteed job latency in data parallel clusters. In Proceedings of EuroSys (2012), pp. 99--112.

Digital Library

[11]

Ghodsi, A., Zaharia, M., Hindman, B., Konwinski, A., Shenker, S., and Stoica, I. Dominant resource fairness: fair allocation of multiple resource types. In Proceedings of NSDI (2011), pp. 323--336.

Digital Library

[12]

Herodotou, H., Dong, F., and Babu, S. No one (cluster) size fits all: automatic cluster sizing for data-intensive analytics. In Proceedings of SoCC (2011).

Digital Library

[13]

Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A., Katz, R., Shenker, S., and Stoica, I. Mesos: a platform for fine-grained resource sharing in the data center. In Proceedings of NSDI (2011).

Digital Library

[14]

Iqbal, S., Gupta, R., and Fang, Y.-C. Planning considerations for job scheduling in HPC clusters. Dell Power Solutions (Feb. 2005).

[15]

Isard, M., Prabhakaran, V., Currey, J., Wieder, U., Talwar, K., and Goldberg, A. Quincy: fair scheduling for distributed computing clusters. In Proceedings of SOSP (2009).

Digital Library

[16]

Jackson, D. and Snell, Q. and Clement, M. Core algorithms of the Maui scheduler. In Job Scheduling Strategies for Parallel Processing. 2001, pp. 87--102.

Digital Library

[17]

Kavulya, S., Tan, J., Gandhi, R., and Narasimhan, P. An analysis of traces from a production MapReduce cluster. In Proceedings of CCGrid (2010), pp. 94--103.

Digital Library

[18]

Kung, H. T., and Robinson, J. T. On optimistic methods for concurrency control. ACM Transactions on Database Systems 6, 2 (June 1981), 213--226.

Digital Library

[19]

Malewicz, G., Austern, M., Bik, A., Dehnert, J., Horn, I., Leiser, N., and Czajkowski, G. Pregel: a system for large-scale graph processing. In Proceedings of SIGMOD (2010), pp. 135--146.

Digital Library

[20]

Mishra, A. K., Hellerstein, J. L., Cirne, W., and Das, C. R. Towards characterizing cloud backend workloads: insights from Google compute clusters. SIGMETRICS Performance Evaluation Review 37 (Mar. 2010), 34--41.

Digital Library

[21]

Murthy, A. C., Douglas, C., Konar, M., O'Malley, O., Radia, S., Agarwal, S., and K V, V. Architecture of next generation Apache Hadoop MapReduce framework. Tech. rep., Apache Hadoop, 2011.

[22]

Pan, H., Hindman, B., and Asanović, K. Lithe: enabling efficient composition of parallel libraries. In Proceedings of HotPar (2009).

Digital Library

[23]

Peng, D., and Dabek, F. Large-scale incremental processing using distributed transactions and notifications. In Proceedings of OSDI (2010).

Digital Library

[24]

Reiss, C., Tumanov, A., Ganger, G. R., Katz, R. H., and Kozuch, M. A. Heterogeneity and dynamicity of clouds at scale: Google trace analysis. In Proceedings of SoCC (2012).

Digital Library

[25]

Sharma, B., Chudnovsky, V., Hellerstein, J., Rifaat, R., and Das, C. Modeling and synthesizing task placement constraints in Google compute clusters. In Proceedings of SoCC (2011).

Digital Library

[26]

Verma, A., Cherkasova, L., and Campbell, R. SLO-driven right-sizing and resource provisioning of MapReduce jobs. In Proceedings of LADIS (2011).

[27]

Wilkes, J. More Google cluster data. Google research blog, Nov. 2011. Posted at http://goo.gl/9B7PA.

[28]

Zaharia, M., Borthakur, D., Sen Sarma, J., Elmeleegy, K., Shenker, S., and Stoica, I. Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In Proceedings of EuroSys (2010), pp. 265--278.

Digital Library

[29]

Zhang, Q., Hellerstein, J., and Boutaba, R. Characterizing task usage shapes in Google's compute clusters. In Proceedings of LADIS (2011).

Cited By

Reidys BZardoshti PGoiri ÍIrvene CBerger DMa HArya KCortez EStark TBak EIyigun MNovakovic SHsu LTrueba KPan ABansal CRajmohan SHuang JBianchini REeckhout LSmaragdakis GLiang KSampson AKim MRossbach C(2025)Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud PlatformsProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707226(164-181)Online publication date: 30-Mar-2025
https://dl.acm.org/doi/10.1145/3669940.3707226
Kosińska JSąkól KSzczurek M(2025)AMoCNA operator: a Kubernetes operator pattern that enhances cloud-native execution environments with autonomic featuresThe Journal of Supercomputing10.1007/s11227-024-06855-781:4Online publication date: 18-Feb-2025
https://doi.org/10.1007/s11227-024-06855-7
Santos HSimão JVeiga L(2025)FaaS-Utility: Tackling FaaS Cold Starts with User-Preference and QoS-Driven PricingEconomics of Grids, Clouds, Systems, and Services10.1007/978-3-031-81226-2_5(43-57)Online publication date: 6-Feb-2025
https://doi.org/10.1007/978-3-031-81226-2_5
Show More Cited By

Index Terms

Omega: flexible, scalable schedulers for large compute clusters
1. Social and professional topics
  1. Professional topics
    1. Management of computing and information systems
      1. System management
        Centralization / decentralization
2. Software and its engineering
  1. Software organization and properties
    1. Software system structures
      1. Distributed systems organizing principles

Recommendations

An extended fine-grained conflict detection method for shared-state scheduling in large scale cluster
ICIIP '16: Proceedings of the 1st International Conference on Intelligent Information Processing

Nowadays, shared-state scheduling architecturehas been paid more attention in the large-scale cluster scheduling area. As a crucial part of shared-state scheduling architecture, optimistic concurrency control (OCC) algorithm has been studied by database ...
Improving transaction abort rates without compromising throughput through judicious scheduling
SAC '13: Proceedings of the 28th Annual ACM Symposium on Applied Computing

Althought optimistic concurrency control protocols have increasingly been used in distributed database management systems, they imply a trade-off between the number of transactions that can be executed concurrently, hence, the peak throughput, and ...
Infinite Resources for Optimistic Concurrency Control
NetCompute '18: Proceedings of the 2018 Morning Workshop on In-Network Computing

Optimistic concurrency control (OCC) is inefficient for high-contention workloads. When concurrent transactions conflict, an OCC system wastes CPU resources verifying transactions, only to abort them. This paper presents a new system, called Network ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

EuroSys '13: Proceedings of the 8th ACM European Conference on Computer Systems

April 2013

401 pages

ISBN:9781450319942

DOI:10.1145/2465351

General Chairs:
Zdenek Hanzálek
Czech Technical University Prague
,
Hermann Härtig
Technische Universität Dresden
,
Program Chairs:
Miguel Castro
Microsoft Research Cambridge
,
M. Frans Kaashoek
MIT

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGOPS: ACM Special Interest Group on Operating Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 April 2013

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

EuroSys '13

Sponsor:

SIGOPS

EuroSys '13: Eighth Eurosys Conference 2013

April 15 - 17, 2013

Prague, Czech Republic

Acceptance Rates

EuroSys '13 Paper Acceptance Rate 28 of 143 submissions, 20%;

Overall Acceptance Rate 241 of 1,308 submissions, 18%

Upcoming Conference

EuroSys '25

Sponsor:
sigops

Twentieth European Conference on Computer Systems

March 30 - April 3, 2025

Rotterdam , Netherlands

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

508
Total Citations
View Citations
3,525
Total Downloads

Downloads (Last 12 months)199
Downloads (Last 6 weeks)30

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Reidys BZardoshti PGoiri ÍIrvene CBerger DMa HArya KCortez EStark TBak EIyigun MNovakovic SHsu LTrueba KPan ABansal CRajmohan SHuang JBianchini REeckhout LSmaragdakis GLiang KSampson AKim MRossbach C(2025)Coach: Exploiting Temporal Patterns for All-Resource Oversubscription in Cloud PlatformsProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 110.1145/3669940.3707226(164-181)Online publication date: 30-Mar-2025
https://dl.acm.org/doi/10.1145/3669940.3707226
Kosińska JSąkól KSzczurek M(2025)AMoCNA operator: a Kubernetes operator pattern that enhances cloud-native execution environments with autonomic featuresThe Journal of Supercomputing10.1007/s11227-024-06855-781:4Online publication date: 18-Feb-2025
https://doi.org/10.1007/s11227-024-06855-7
Santos HSimão JVeiga L(2025)FaaS-Utility: Tackling FaaS Cold Starts with User-Preference and QoS-Driven PricingEconomics of Grids, Clouds, Systems, and Services10.1007/978-3-031-81226-2_5(43-57)Online publication date: 6-Feb-2025
https://doi.org/10.1007/978-3-031-81226-2_5
Wang ZLi PLiang CWu FYan FVanbever LZhang I(2024)AutothrottleProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691834(149-165)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691834
Fried JChaudhry GSaurez EChoukse EGoiri ÍElnikety SFonseca RBelay AVanbever LZhang I(2024)Making kernel bypass practical for the cloud with junctionProceedings of the 21st USENIX Symposium on Networked Systems Design and Implementation10.5555/3691825.3691829(55-73)Online publication date: 16-Apr-2024
https://dl.acm.org/doi/10.5555/3691825.3691829
Lekkala C(2024)Leveraging Reinforcement Learning for Autonomous Data Pipeline Optimization and ManagementSSRN Electronic Journal10.2139/ssrn.4908414Online publication date: 2024
https://doi.org/10.2139/ssrn.4908414
Shin JArroyo DTantawi AWang CYoussef ANagi R(2024)Cloud-native Workflow Scheduling using a Hybrid Priority Rule, Dynamic Resource Allocation, and Dynamic Task PartitionProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698551(830-846)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698551
Gupta NNarayanan IHanda SChakraborti SThapar PShan BRao ALiu YWang PWu YGao QCheng CYou SHuang LFan JYu KLin KMu TMalani PWang HLu TZhang P(2024)Dynamic Idle Resource Leasing To Safely Oversubscribe Capacity At MetaProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698537(792-810)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698537
Vijayakumar SMadhavapeddy AKalyvianaki E(2024)Scheduling for Reduced Tail Task Latencies in Highly Utilized DatacentersProceedings of the 2024 ACM Symposium on Cloud Computing10.1145/3698038.3698522(302-321)Online publication date: 20-Nov-2024
https://dl.acm.org/doi/10.1145/3698038.3698522
Cvetković LCosta FDjokic MFriedman MKlimovic AWitchel EArpaci-Dusseau ARossbach CKeeton K(2024)Dirigent: Lightweight Serverless OrchestrationProceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles10.1145/3694715.3695966(369-384)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3694715.3695966
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten