Skip to main content
SpringerLink
Log in

Middleware support for many-task computing

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

Many-task computing aims to bridge the gap between two computing paradigms, high throughput computing and high performance computing. Many-task computing denotes high-performance computations comprising multiple distinct activities, coupled via file system operations. The aggregate number of tasks, quantity of computing, and volumes of data may be extremely large. Traditional techniques found in production systems in the scientific community to support many-task computing do not scale to today’s largest systems, due to issues in local resource manager scalability and granularity, efficient utilization of the raw hardware, long wait queue times, and shared/parallel file system contention and scalability. To address these limitations, we adopted a “top-down” approach to building a middleware called Falkon, to support the most demanding many-task computing applications at the largest scales. Falkon (Fast and Light-weight tasK executiON framework) integrates (1) multi-level scheduling to enable dynamic resource provisioning and minimize wait queue times, (2) a streamlined task dispatcher able to achieve orders-of-magnitude higher task dispatch rates than conventional schedulers, and (3) data diffusion which performs data caching and uses a data-aware scheduler to co-locate computational and storage resources. Micro-benchmarks have shown Falkon to achieve over 15K+ tasks/s throughputs, scale to hundreds of thousands of processors and to millions of queued tasks, and execute billions of tasks per day. Data diffusion has also shown to improve applications scalability and performance, with its ability to achieve hundreds of Gb/s I/O rates on modest sized clusters, with Tb/s I/O rates on the horizon. Falkon has shown orders of magnitude improvements in performance and scalability than traditional approaches to resource management across many diverse workloads and applications at scales of billions of tasks on hundreds of thousands of processors across clusters, specialized systems, Grids, and supercomputers. Falkon’s performance and scalability have enabled a new class of applications called Many-Task Computing to operate at previously so-believed impossible scales with high efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Gara, A., et al.: Overview of the Blue Gene/L system architecture, IBM J. Res. Develop. 49(2/3) (2005)

  2. IBM BlueGene/P, http://www.research.ibm.com/bluegene/ (2008)

  3. Ousterhout, J.: Scripting: higher level programming for the 21st century. Computer 31, 23–30 (1998)

    Article  Google Scholar 

  4. Zhao, Y., Raicu, I., Foster, I.: Scientific workflow systems for 21st century e-science, new bottle or new wine? In: IEEE Workshop on Scientific Workflows (2008)

  5. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: USENIX OSDI04 (2004)

  6. Zhao, Y., Hategan, M., Clifford, B., Foster, I., von Laszewski, G., Raicu, I., Stef-Praun, T., Wilde, M.: Swift: fast, reliable, loosely coupled parallel computation. In: IEEE Workshop on Scientific Workflows (2007)

  7. Raicu, I., Zhao, Y., Dumitrescu, C., Foster, I., Wilde, M.: Falkon: a Fast and Light-weight tasK executiON framework. In: IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC07 (2007)

  8. Deelman, E., et al.: Pegasus: a framework for mapping complex scientific workflows onto distributed systems. Sci. Program. J. 13(3), 219–237 (2005)

    Google Scholar 

  9. Raicu, I., Foster, I., Zhao, Y.: Many-task computing for grids and supercomputers. In: IEEE Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS08 (2008)

  10. Raicu, I., Zhao, Y., Foster, I., Szalay, A.: Accelerating large-scale data exploration through data diffusion. In: ACM International Workshop on Data-Aware Distributed Computing (2008)

  11. Isard, M., Budiu, M., Yu, Y., Birrell, A., Fetterly, D.: Dryad: distributed data-parallel programs from sequential building blocks. In: Eur. Conf. Comput. Syst. (2007)

  12. Pike, R., Dorward, S., Griesemer, R., Quinlan, S.: Interpreting the data: parallel analysis with Sawzall. Sci. Program. J. 13(4), 227–298 (2005). Special Issue on Grids and Worldwide Computing Programming Models and Infrastructure

    Google Scholar 

  13. Livny, M., Basney, J., Raman, R., Tannenbaum, T.: Mechanisms for high throughput computing, SPEEDUP J. 1(1) (1997)

  14. Foster, I., Kesselman, C. (eds.): The Grid: Blueprint for a Future Computing Infrastructure, Chapter 2: Computational Grids. Morgan Kaufmann, San Mateo (1999)

    Google Scholar 

  15. Foster, I., Kesselman, C., Tuecke, S.: The anatomy of the grid. Int. J. Supercomput. Appl. 15, 200–222 (2001)

    Article  Google Scholar 

  16. Hey, T., Trefethen, A.: The data deluge: an e-science perspective. In: Gid Computing: Making the Global Infrastructure a Reality (2003)

  17. Catlett, C., et al.: TeraGrid: analysis of organization, system architecture, and middleware Enabling New Types of Applications. In: HPC (2006)

  18. Open Science Grid (OSG). http://www.opensciencegrid.org/ (2008)

  19. Szalay, A., Bunn, A., Gray, J., Foster, I., Raicu, I.: The importance of data locality in distributed computing applications. In: NSF Workflow Workshop (2006)

  20. Gray, J.: Distributed computing economics. Technical Report MSR-TR-2003-24, Microsoft Research, Microsoft Corp. (2003)

  21. Raicu, I., Zhang, Z., Wilde, M., Foster, I., Beckman, P., Iskra, K., Clifford, B.: Towards loosely-coupled programming on petascale systems. In: IEEE/ACM International Conference for High Performance Computing, Networking, Storage and Analysis, SuperComputing/SC08 (2008)

  22. SiCortex. http://www.sicortex.com/ (2008)

  23. IBM Blue Gene team: overview of the IBM Blue Gene/P project. IBM J. Res. Develop. 52(1/2), 199–220 (2008)

    Google Scholar 

  24. Raicu, I., Foster, I., Zhao, Y., Little, P., Moretti, C., Chaudhary, A., Thain, D.: The quest for scalable support of data intensive workloads in distributed systems. In: ACM HPDC09 (2009)

  25. Raicu, I., Zhao, Y., Foster, I., Szalay, A.: A data diffusion approach to large-scale scientific exploration. In: Microsoft eScience Workshop at RENCI (2007)

  26. Raicu, I.: Harnessing grid resources with data-centric task farms. Technical Report, University of Chicago (2007)

  27. Zhang, Z., Espinosa, A., Iskra, K., Raicu, I., Foster, I., Wilde, M.: Design and evaluation of a collective I/O model for loosely-coupled petascale programming. In: IEEE Workshop on Many-Task Computing on Grids and Supercomputers, MTAGS08 (2008)

  28. Raicu, I., Foster, I., Zhao, Y., Szalay, A., Little, P., Moretti, C., Chaudhary, A., Thain, D.: Towards data intensive many-task computing. In: Data Intensive Distributed Computing: Challenges and Solutions for Large-Scale Information Management. IGI Global, Hershey (2009)

    Google Scholar 

  29. Thain, D., Tannenbaum, T., Livny, M.: Distributed computing in practice: the condor experience. Concurr. Comput. Pract. Exp. 17(2–4), 323–356 (2005)

    Article  Google Scholar 

  30. Robinson, E., DeWitt, D.J.: Turning cluster management into data management: a system overview. In: Conference on Innovative Data Systems Research (2007)

  31. Bode, B., Halstead, D.M., Kendall, R., Lei, Z., Hall, W., Jackson, D.: The portable batch scheduler and the Maui scheduler on Linux clusters. In: Usenix, Linux Showcase & Conference (2000)

  32. Zhou, S.: LSF: Load sharing in large-scale heterogeneous distributed systems. In: Workshop on Cluster Computing (1992)

  33. Gentzsch, W.: Sun grid engine: towards creating a compute power grid. In: 1st International Symposium on Cluster Computing and the Grid (2001)

  34. Bialecki, A., Cafarella, M., Cutting, D., O’Malley, O.: Hadoop: a framework for running applications on large clusters built of commodity hardware. http://lucene.apache.org/hadoop/ (2005)

  35. Anderson, D.P.: BOINC: a system for public-resource computing and storage. In: IEEE/ACM International Workshop on Grid Computing (2004)

  36. Frey, J., Tannenbaum, T., Foster, I., Frey, M., Tuecke, S.: Condor-G: a computation management agent for multi-institutional grids. Cluster Comput 5, 237–246 (2002)

    Article  Google Scholar 

  37. Banga, G., Druschel, P., Mogul, J.C.: Resource containers: a new facility for resource management in server systems. In: Symposium on Operating Systems Design and Implementation (1999)

  38. Stankovic, J.A., Ramamritham, K., Niehaus, D., Humphrey, M., Wallace, G.: The spring system: integrated support for complex real-time systems. Real-Time Syst. 16(2/3), 97–125 (1999)

    Article  Google Scholar 

  39. Mehta, G., Kesselman, C., Deelman, E.: Dynamic deployment of VO-specific schedulers on managed resources. Technical Report, USC ISI (2006)

  40. Walker, E., Gardner, J.P., Litvin, V., Turner, E.L.: Creating personal adaptive clusters for managing scientific tasks in a distributed computing environment. In: Workshop on Challenges of Large Applications in Distributed Environments (2006)

  41. Singh, G., Kesselman, C., Deelman, E.: Performance impact of resource provisioning on workflows. Technical Report, USC ISI (2006)

  42. Anderson, D.P., Korpela, E., Walton, R.: High-performance task distribution for volunteer computing. In: IEEE International Conference on e-Science and Grid Technologies (2005)

  43. Cope, J., et al.: High throughput grid computing with an IBM Blue Gene/L. In: Cluster (2007)

  44. Peters, A., King, A., Budnik, T., McCarthy, P., Michaud, P., Mundy, M., Sexton, J., Stewart, G.: Asynchronous task dispatch for high throughput computing for the eServer IBM Blue Gene® Supercomputer. In: Parallel and Distributed Processing, IPDPS, (2008)

  45. IBM coorporation: High-throughput computing (HTC) paradigm. In: IBM System Blue Gene Solution: Blue Gene/P Application Development, IBM RedBooks (2008)

  46. Desai, N.: Cobalt: an open source platform for HPC system software research. In: Edinburgh BG/L System Software Workshop (2005)

  47. Zhao, Y., Raicu, I., Foster, I., Hategan, M., Nefedova, V., Wilde, M.: Realizing fast, scalable and reliable scientific computations in grid environments. In: Grid Computing Research Progress. Nova Publisher, New York (2008)

    Google Scholar 

  48. Swift Workflow System. www.ci.uchicago.edu/swift (2008)

  49. Laszewski, G.v., Hategan, M., Kodeboyina, D.: Java CoG kit workflow. In: Taylor, I.J., Deelman, E., Gannon, D.B., Shields, M. (eds.): Workflows for eScience, pp. 340–356. Springer, Berlin (2007)

    Chapter  Google Scholar 

  50. Foster, I.: Globus toolkit version 4: software for service-oriented systems. In: Conference on Network and Parallel Computing (2005)

  51. The Globus Security Team. Globus toolkit version 4: grid security infrastructure: a standards perspective. Technical Report, Argonne National Laboratory, MCS (2005)

  52. Feller, M., Foster, I., Martin, S.: GT4 GRAM: a functionality and performance study. In: TeraGrid Conference (2007)

  53. Podlipnig, S., Böszörmenyi, L.: A survey of Web cache replacement strategies. ACM Comput. Surv. 35(4), 374–398 (2003)

    Article  Google Scholar 

  54. Allcock, W., Bresnahan, J., Kettimuthu, R., Link, M., Dumitrescu, C., Raicu, I., Foster, I.: The globus striped GridFTP framework and server. In: ACM/IEEE SC05 (2005)

  55. GKrellM. http://members.dslextreme.com/users/billw/gkrellm/gkrellm.html (2008)

  56. Walker, E., Earl, D.J., Deem, M.W.: How to run a million jobs in six months on the NSF teraGrid. In: TeraGrid Conference (2007)

  57. Raicu, I., Dumitrescu, C., Foster, I.: Dynamic resource provisioning in grid environments. In: TeraGrid Conference (2007)

  58. ANL/UC TeraGrid site details. http://www.uc.teragrid.org/tg-docs/tg-tech-sum.html (2007)

  59. Schmuck, F., Haskin, R.: GPFS: a shared-disk file system for large computing clusters. In: FAST (2002)

  60. Moretti, C., Bulosan, J., Thain, D., Flynn, P.: All-pairs: an abstraction for data-intensive cloud computing. In: IPDPS (2008)

  61. Thain, D., Moretti, C., Hemmes, J.: Chirp: a practical global file system for cluster and grid computing. J. Grid Comput 7, 51–72 (2008)

    Article  Google Scholar 

  62. The Functional Magnetic Resonance Imaging Data Center. http://www.fmridc.org/ (2007)

  63. NIST Chemistry WebBook Database. http://webbook.nist.gov/chemistry/ (2008)

  64. Moustakas, D.T., et al.: Development and validation of a modular, extensible docking program: DOCK 5. J. Comput. Aided Mol. Des. 20, 601–619 (2006)

    Article  Google Scholar 

  65. KEGG’s Ligand Database. http://www.genome.ad.jp/kegg/ligand.html (2008)

  66. Hanson, D.: Enhancing technology representations within the stanford energy modeling forum (EMF) climate economic models, energy and economic policy models: a reexamination of fundamentals (2006)

  67. Raicu, I., Foster, I., Szalay, A., Turcu, G.: AstroPortal: a science gateway for large-scale astronomy data analysis. In: TeraGrid Conference (2006)

  68. Raicu, I., Foster, I., Szalay, A.: Harnessing grid resources to enable the dynamic analysis of large astronomy datasets. In: IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC06 (2006)

  69. SDSS: Sloan Digital Sky Survey. http://www.sdss.org/ (2008)

  70. Jacob, J.C., et al.: The montage architecture for grid-enabled science processing of large, distributed datasets. In: Earth Science Technology Conference (2004)

  71. Katz, D., Berriman, G., Deelman, E., Good, J., Jacob, J., Kesselman, C., Laity, A., Prince, T., Singh, G., Su, M.: A comparison of two methods for building astronomical image mosaics on a grid. In: Proceedings of the 7th Workshop on High Performance Scientific and Engineering Computing, HPSEC-05 (2005)

  72. Pham, Q.T., Balkir, A.S., Tie, J., Foster, I., Wilde, M., Raicu, I.: Data intensive scalable computing on TeraGrid: A comparison of MapReduce and Swift. In: TeraGrid Conference, TG08 (2008)

  73. Stevens, R.: The LLNL/ANL/IBM collaboration to develop BG/P and BG/Q. DOE ASCAC Report (2006)

Download references

Author information

Authors and Affiliations

  1. Northwestern University, Evanston, IL, USA

    Ioan Raicu & Alok Choudhary

  2. University of Chicago, Chicago, IL, USA

    Ian Foster, Mike Wilde, Zhao Zhang, Kamil Iskra & Peter Beckman

  3. Argonne National Laboratory, Argonne, IL, USA

    Ian Foster, Mike Wilde, Kamil Iskra & Peter Beckman

  4. Microsoft, Redmond, WA, USA

    Yong Zhao

  5. John Hopkins University, Baltimore, MD, USA

    Alex Szalay

  6. University of Notre Dame, Notre Dame, IN, USA

    Philip Little, Christopher Moretti, Amitabh Chaudhary & Douglas Thain

Authors
  1. Ioan Raicu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ian Foster
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mike Wilde
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Kamil Iskra
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Peter Beckman
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yong Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Alex Szalay
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Alok Choudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Philip Little
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Christopher Moretti
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Amitabh Chaudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Douglas Thain
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ioan Raicu.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Raicu, I., Foster, I., Wilde, M. et al. Middleware support for many-task computing. Cluster Comput 13, 291–314 (2010). https://doi.org/10.1007/s10586-010-0132-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-010-0132-9

Keywords

Search

Navigation