Skip to main content
SpringerLink
Log in

Malleable scheduling for flows of jobs and applications to MapReduce

  • Published:
Journal of Scheduling Aims and scope Submit manuscript

Abstract

This paper provides a unified family of algorithms with performance guarantees for malleable scheduling problems on flows. A flow represents a set of jobs with precedence constraints. Each job has a speedup function that governs the rate at which work is done on the job as a function of the number of processors allocated to it. In our setting, each speedup function is linear up to some job-specific processor maximum. A key aspect of malleable scheduling is that the number of processors allocated to any job is allowed to vary with time. The overall objective is to minimize either the total cost (minisum) or the maximum cost (minimax) of the flows. Our approach handles a very general class of cost functions, and in particular provides the first constant-factor approximation algorithms for total and maximum weighted completion time. Our motivation for this work was scheduling in MapReduce, and we also provide experimental evaluations that show good practical performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Hadoop: https://hadoop.apache.org. Accessed 4 Aug 2018.

  2. IBM BigInsights: www-01.ibm.com/software/data/infosphere/biginsights/. Accessed 4 Aug 2018.

  3. Gridmix: https://hadoop.apache.org/docs/r1.2.1/gridmix.html. Accessed 4 Aug 2018.

  4. Facebook: https://code.facebook.com/posts/229861827208629/scaling-the-facebook-data-warehouse-to-300-pb/. Accessed 4 Aug 2018.

  5. The \(\tilde{O}\) notation ignores logarithmic factors.

  6. IBM BigInsights: www-01.ibm.com/software/data/infosphere/biginsights/. Accessed 4 Aug 2018.

  7. Gridmix: https://hadoop.apache.org/docs/r1.2.1/gridmix.html. Accessed 4 Aug 2018.

References

  • Baker, B., Coffman, E., & Rivest, R. (1980). Orthogonal packings in two dimensions. SIAM Journal on Computing, 9(4), 846–855.

    Article  Google Scholar 

  • Berlinska, J., & Drozdowski, M. (2011). Scheduling divisible MapReduce computations. Journal of Parallel and Distributed Computing, 71, 450–459.

    Article  Google Scholar 

  • Beyer, K., Ercegovac, V., Gemulla, R., Balmin, A., Eltabakh, M., Kanne, C.-C., et al. (2011). Jaql: A scripting language for large scale semistructured data analysis. Proceedings of VLDB, 4(12), 1272–1283.

    Google Scholar 

  • Chen, C.-Y., & Chu, C.-P. (2013). A 3.42-approximation algorithm for scheduling malleable tasks under precedence constraints. IEEE Transactions on Parallel and Distributed Systems, 24(8), 1479–1488.

    Article  Google Scholar 

  • Coffman, E., Garey, M., Johnson, D., & Tarjan, R. (1980). Performance bounds for level-oriented two-dimensional packing algorithms. SIAM Journal on Computing, 9(4), 808–826.

    Article  Google Scholar 

  • Daitch, S. I., & Spielman, D. A. (2008) Faster approximate lossy generalized flow via interior point algorithms. In Proceedings of the 40th annual ACM symposium on theory of computing (pp. 451–460).

  • Dean, J., & Ghemawat, S. (2008). Mapreduce: Simplified data processing on large clusters. ACM Transactions on Computer Systems, 51(1), 107–113.

    Google Scholar 

  • Drozdowski, M. (1996). Real-time scheduling of linear speedup parallel tasks. Information Processing Letters, 57(1), 35–40.

    Article  Google Scholar 

  • Drozdowski, M. (2001). New applications of the Munz and Coffman algorithm. Journal of Scheduling, 4(4), 209–223.

    Article  Google Scholar 

  • Drozdowski, M. (2009). Scheduling for parallel processing. Heidelberg: Springer.

    Book  Google Scholar 

  • Drozdowski, M., & Kubiak, W. (1999). Scheduling parallel tasks with sequential heads and tails. Annals of Operations Research, 90, 221–246.

    Article  Google Scholar 

  • Fleischer, L., & Wayne, K. (2002). Fast and simple approximation schemes for generalized flow. Mathematical Programming, 91(2), 215–238.

    Article  Google Scholar 

  • Fotakis, D., Milis, I., Papadigenopoulos, O., Vassalos, V. & Zois, G. (2016). Scheduling MapReduce jobs under multi-round precedences. In Euro-Par 2016: Parallel processing—22nd international conference on parallel and distributed computing (pp. 209–222).

  • Garey, M., & Graham, R. (1975). Bounds for multiprocessor scheduling with resource constraints. SIAM Journal on Computing, 4(2), 187–200.

    Article  Google Scholar 

  • Gates, A., Natkovich, O., Chopra, S., Kamath, P., Narayanamurthy, S., Olston, C., et al. (2009). Building a high-level dataflow system on top of MapReduce: The pig experience. Proceedings of VLDB, 2(2), 1414–1425.

    Article  Google Scholar 

  • Graham, R. L. (1966). Bounds for certain multiprocessing anomalies. Bell System Technical Journal, 45(9), 1563–1581.

    Article  Google Scholar 

  • Günther, E., König, F. G., & Megow, N. (2014). Scheduling and packing malleable and parallel tasks with precedence constraints of bounded width. Journal of Combinatorial Optimization, 27(1), 164–181.

    Article  Google Scholar 

  • Hochbaum, D. S., & Shmoys, D. B. (1986). A unified approach to approximation algorithms for bottleneck problems. Journal of the ACM, 33(3), 533–550.

    Article  Google Scholar 

  • Ibarra, O. H., & Kim, C. E. (1975). Fast approximation algorithms for the knapsack and sum of subset problems. Journal of the ACM, 22(4), 463–468.

    Article  Google Scholar 

  • Islam, M., Huang, A., Battisha, M., Chiang, M., Srinivasan, S., Peters, C., et al. (2012) Oozie: Towards a scalable workflow management system for Hadoop. In Proceedings of the ACM workshop on scalable workflow execution engines and technologies.

  • Jansen, K., & Zhang, H. (2006). An approximation algorithm for scheduling malleable tasks under general precedence constraints. ACM Transactions on Algorithms, 2(3), 416–434.

    Article  Google Scholar 

  • Jansen, K., & Zhang, H. (2012). Scheduling malleable tasks with precedence constraints. Journal of Computer and System Sciences, 78(1), 245–259.

    Article  Google Scholar 

  • Kalyanasundaram, B., & Pruhs, K. (2000). Speed is as powerful as clairvoyance. Journal of the ACM, 47(4), 617–643.

    Article  Google Scholar 

  • Karloff, H., Suri, S., & Vassilvitskii, S. (2010) A model of computation for MapReduce. In SODA (pp. 938–948).

  • Koutris, P., & Suciu, D. (2011) Parallel evaluation of conjunctive queries. In PODS (pp. 223–234).

  • Labetoulle, J., Lawler, E., Lentra, J., & Rinnoy Kan, A. (1984). Preemptive scheduling of uniform machines subject to release dates. In W. Pulleyblank (Ed.), Progress in combinatorial optimization (pp. 245–261). New York: Academic Press.

    Chapter  Google Scholar 

  • Lepére, R., Trystram, D., & Woeginger, G. J. (2001) Approximation algorithms for scheduling malleable tasks under precedence constraints. In ESA (pp. 146–157).

  • Leung, J. (2004). Handbook of scheduling. London: Chapman and Hall/CRC.

    Google Scholar 

  • Ludwig, W., & Tiwari, P. (1994). Scheduling malleable and nonmalleable parallel tasks. In Symposium on discrete algorithms, Arlington, VA (pp. 167–176).

  • McNaughton, R. (1959). Scheduling with deadlines and loss functions. Management Science, 6(1), 1–12.

    Article  Google Scholar 

  • Moseley, B., Dasgupta, A., Kumar, R., & Sarlós, T. (2011). On scheduling in map-reduce and flow-shops. In Symposium on parallel algorithms and architectures. San Jose, CA.

  • Nagarajan, V., Wolf, J., Balmin, A., & Hildrum, K. (2013) Flowflex: Malleable scheduling for flows of MapReduce jobs. In International middleware conference, Beijing, China (pp. 103–122).

  • Pinedo, M. (1995). Scheduling: Theory, algorithms and systems. London: Prentice Hall.

    Google Scholar 

  • Popescu, A., Ercegovac, V., Balmin, A., Branco, M., & Ailamaki, A. (2012) Same queries, different data: Can we predict runtime performance? In International conference in data engineering, Washington, DC (pp. 275–280).

  • Schuurman, P., & Woeginger, G. J. (2000). A polynomial time approximation scheme for the two-stage multiprocessor flow shop problem. Theoretical Computer Science, 237(1–2), 105–122.

    Article  Google Scholar 

  • Schwiegelshohn, U., Ludwig, W., Wolf, J., Turek, J., & Yu, P. (1999). Smart SMART bounds for weighted response time scheduling. SIAM Journal on Computing, 28(1), 237–253.

    Article  Google Scholar 

  • Sleator, D. (1980). A 2.5 times optimal algorithm for packing in two dimensions. Information Processing Letters, 10(1), 37–40.

    Article  Google Scholar 

  • Thusoo, A., Sarma, J., Jain, N., Shao, Z., Chakka, P., Zhang, N., Anthony, S., Liu, H., & Murthy, R. (2010) Hive—A petabyte scale data warehouse using Hadoop. In International conference on data engineering, Long Beach, CA (pp. 996–1005).

  • Turek, J., Wolf, J., & Yu, P. (1992) Approximate algorithms for scheduling parallelizable tasks. In Symposium on parallel algorithms and architectures, San Diego, CA (pp. 323–332).

  • Vizing, V. (1982). Minimization of the maximum delay in servicing systems with interruption. USSR Computational Mathematics and Methematical Physics, 22(3), 227–233.

    Article  Google Scholar 

  • Wolf, J., Rajan, D., Hildrum, K., Khandekar, R., Kumar, R., Parekh, S., et al. (2010) FLEX: A slot allocation scheduling optimizer for MapReduce workloads. In International middleware conference, Bangalore, India (pp. 1–20).

  • Wolf, J., Balmin, A., Rajan, D., Hildrum, K., Khandekar, R., Parekh, S., et al. (2012). On the optimization of schedules for MapReduce workloads in the presence of shared scans. VLDB Journal, 21(5), 589–609.

    Article  Google Scholar 

  • Zaharia, M., Borthakur, D., Sarma, J., Elmeleegy, K., Schenker, S., & Stoica, I. (2009) Job scheduling for multi-user MapReduce clusters. UC Berkeley technical report EECS-2009-55.

  • Zaharia, M., Borthakur, D., Sarma, J., Elmeleegy, K., Shenker, S., & Stoica, I. (2010) Delay scheduling: A simple technique for achieving locality and fairness in cluster scheduling. In European conference on computer systems, Paris (pp. 265–278).

Download references

Author information

Authors and Affiliations

  1. Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI, 48109, USA

    Viswanath Nagarajan

  2. IBM T.J. Watson Research Center, Yorktown Heights, NY, 10598, USA

    Joel Wolf & Kirsten Hildrum

  3. Platfora Inc., San Mateo, USA

    Andrey Balmin

Authors
  1. Viswanath Nagarajan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Joel Wolf
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrey Balmin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kirsten Hildrum
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Viswanath Nagarajan.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Nagarajan, V., Wolf, J., Balmin, A. et al. Malleable scheduling for flows of jobs and applications to MapReduce. J Sched 22, 393–411 (2019). https://doi.org/10.1007/s10951-018-0576-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10951-018-0576-y

Keywords

Search

Navigation