Skip to main content
SpringerLink
Log in

Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family

  • Chapter
  • First Online:
Cloud Computing

Part of the book series: Computer Communications and Networks ((CCN))

Abstract

The Nimrod tool family facilitates high-throughput science by allowing researchers to explore complex design spaces using computational models. Users are able to describe large experiments in which models are executed across changing input parameters. Different members of the tool family support complete and partial parameter sweeps, numerical search by non-linear optimisation and even workflows. In order to provide timely results and to enable large-scale experiments, distributed computational resources are aggregated to form a logically single high-throughput engine. To date, we have leveraged grid middleware standards to spawn computations on remote machines. Recently, we added an interface to Amazon’s Elastic Compute Cloud (EC2), allowing users to mix conventional grid resources and clouds. A range of schedulers, from round-robin queues to those based on economic budgets, allow Nimrod to mix and match resources. This provides a powerful platform for computational researchers, because they can use a mix of university-level infrastructure and commercial clouds. In particular, the system allows a user to pay money to increase the quality of the research outcomes and to decide exactly how much they want to pay to achieve a given return. In this chapter, we will describe Nimrod and its architecture, and show how this naturally scales to incorporate clouds. We will illustrate the power of the system using a case study and will demonstrate that cloud computing has the potential to enable high-throughput science.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The recently released EC2 Spot Instance pricing (http://aws.amazon.com/ec2/spot-instances/) – a supply-demand-driven auctioning of excess EC2 data-centre capacity – is an early example of a scheme to bridge this gap.

References

  1. Abramson D, Giddy J, Kotler L (2000) High performance parametric modeling with Nimrod/G: killer application for the global grid? In the 14th international parallel and distributed processing symposium (IPDPS 2000), pp 520–528

    Book  Google Scholar 

  2. Abramson D, Buyya R, Giddy J (Oct 2002) A computational economy for grid computing and its implementation in the Nimrod-G resource broker. Future Gen Comput Syst 18:1061–1074

    Article  MATH  Google Scholar 

  3. Litzkow M, Livny M, Mutka M (1988) Condor – a hunter of idle workstations. In the proceedings of the 8th international conference of distributed computing systems. IEEE Press, June 1988, pp 104–111

    Google Scholar 

  4. Foster I, Kesselman C (1997) Globus: a metacomputing infrastructure toolkit. Int J Supercomput Appl 11:115–128

    Article  Google Scholar 

  5. Erwin DW (2002) UNICORE – a grid computing environment. Concurr Comput Prac Exp 14:1395–1410

    Article  MATH  Google Scholar 

  6. Laure E, Fisher SM, Frohner A, Grandi C, Kunszt P, Krenek A, Mulmo O, Pacini F, Prelz F, White J, Barroso M, Buncic P, Hemmer F, Di Meglio A, Edlund A (2006) Programming the Grid with gLite. Comput Method Sci Tech 12:33–46

    Google Scholar 

  7. Bethwaite B, Abramson D, Buckle A (2008) Grid interoperability: an experiment in bridging grid islands. In the IEEE Fourth International Conference on eScience 2008, pp 590–596

    Google Scholar 

  8. Riedel M (2009) Interoperation of world-wide production e-Science infrastructures. Concurr Comput Pract Exp 21:961–990

    Article  Google Scholar 

  9. Goscinski W, Abramson D (2008) An infrastructure for the deployment of e-science applications. In: Grandinetti L (ed) High performance computing (HPC) and grids in action. IOS Press, Amsterdam, Netherlands, pp 131–148

    Google Scholar 

  10. Schmidberger J, Bethwaite B, Enticott C, Bate M, Androulakis S, Faux N, Reboul C, Phan J, Whisstock J, Goscinski W, Garic S, Abramson D, Buckle A (2009) High-throughput protein structure determination using grid computing. In the IEEE International Symposium on Parallel & Distributed Processing (IPDPS 2009), pp 1–8

    Google Scholar 

  11. Sher A, Abramson D, Enticott C, Garic S, Gavaghan D, Noble D, Noble P, Peachey T (2008) Incorporating local Ca2 + dynamics into single cell ventricular models. In: Proceedings of the 8th international conference on computational science, Part I, Springer-Verlag, Krakow, Poland, pp 66–75

    Google Scholar 

  12. Baldridge KK, Sudholt W, Greenberg JP, Amoreira C, Potier Y, Altintas I, Birnbaum A, Abramson D, Enticott C, Garic S (2006) Cluster and grid infrastructure for computational chemistry and biochemistry. In: Zomaya AY (ed) Parallel computing for bioinformatics and computational biology, Wiley Interscience, New York, pp 531–550

    Google Scholar 

  13. Lynch AH, Abramson D, Görgen K, Beringer J, Uotila P (Oct 2007) Influence of savanna fire on Australian monsoon season precipitation and circulation as simulated using a distributed computing environment, Geophysical Research Letters, 34(20):L20801

    Article  Google Scholar 

  14. Abramson D, Lewis A, Peachey T, Fletcher C (2001) An automatic design optimization tool and its application to computational fluid dynamics. In: Proceedings of the 2001 ACM/IEEE conference on supercomputing (CDROM), ACM, Denver, Colorado, pp 25–25

    Google Scholar 

  15. Peachey T, Diamond N, Abramson D, Sudholt W, Michailova A, Amirriazi S (Jan 2008) Fractional factorial design for parameter sweep experiments using Nimrod/E. Sci Program 16:217–230

    Google Scholar 

  16. Abramson D, Enticott C, Altinas I (2008) Nimrod/K: towards massively parallel dynamic grid workflows. In: Proceedings of the 2008 ACM/IEEE conference on supercomputing, IEEE Press, Austin, Texas, pp 1–11

    Google Scholar 

  17. Buyya R, Abramson D, Giddy J, Stockinger H (2002) Economic models for resource management and scheduling in Grid computing. Concurr Comput Prac Exp 14:1507–1542

    Article  MATH  Google Scholar 

  18. Zhang Y, Mandal A, Koelbel C, Cooper K (2009) Combined fault tolerance and scheduling techniques for workflow applications on computational grids. In: The proceedings of the 2009 9th IEEE/ACM international symposium on cluster computing and the grid – volume 00, IEEE Computer Society, pp 244–251

    Google Scholar 

  19. Nurmi D, Brevik J, Wolski R (2008) QBETS: Queue Bounds Estimation from Time Series. Job Scheduling Strategies for Parallel Processing, pp 76–101

    Google Scholar 

  20. Buyya R, Giddy J, Abramson D (2000) An evaluation of economy-based resource trading and scheduling on computational power grids for parameter sweep applications. Active middleware services: from the proceedings of the 2nd annual workshop on active middleware services, p 221

    Google Scholar 

  21. Nurmi D, Wolski R, Grzegorczyk C, Obertelli G, Soman S, Youseff L, Zagorodnov D (2009) The eucalyptus open-source cloud-computing system. In: The IEEE international symposium on cluster computing and the grid, IEEE Press, Shanghai, China, p 131, 124

    Google Scholar 

  22. Sotomayor B, Montero RS, Llorente IM, Foster I (2008) Capacity leasing in cloud systems using the OpenNebula engine. Chicago, IL

    Google Scholar 

  23. Raicu I, Zhao Y, Dumitrescu C, Foster I, Wilde M (2007) Falkon: a fast and light-weight tasK executiON framework. In the proceedings of the 2007 ACM/IEEE conference on supercomputing – volume 00, ACM Press, Reno, Nevada, pp 1–12

    Google Scholar 

  24. Troger P, Rajic H, Haas A, Domagalski P (2007) Standardization of an API for distributed resource management systems. In the proceedings of the seventh IEEE international symposium on cluster computing and the grid, IEEE Computer Society, pp 619–626

    Google Scholar 

  25. Resnick P, Varian HR (1997) Recommender systems. Commun ACM 40:56–58

    Article  Google Scholar 

  26. Bohnert F, Schmidt DF, Zukerman I (2009) Spatial processes for recommender systems. In: The 21st international joint conference on artificial intelligence (IJCAI-09), Pasadena, CA, pp 2022–2027

    Google Scholar 

  27. Bohnert F, Zukerman I, Berkovsky S, Baldwin T, Sonenberg L (2008) Using interest and transition models to predict visitor locations in museums. AI Commun 21:195–202

    MathSciNet  MATH  Google Scholar 

  28. Banerjee S, Carlin BP, Gelfand AE (2004) Hierarchical modeling and analysis for spatial data. CRC Press, Boca Raton, FL

    MATH  Google Scholar 

  29. Neal RM (2003) Slice sampling. Annal Stat 31:705–767

    Article  MathSciNet  MATH  Google Scholar 

  30. Bohnert F, Zukerman I, Schmidt DF (2009) Using Gaussian spatial processes to model and predict interests in museum exhibits. In the seventh workshop on intelligent techniques for web personalization and recommender systems (ITWP-09), pp 13–19

    Google Scholar 

Download references

Acknowledgements

This work has been supported by the Australian Research Council under the Discovery grant scheme. We thank the Australian Academy of Technological Sciences and Engineering (ATSE) Working Group on Cloud Computing for discussions that were used as input to Section 1. We thank Ian Foster for his helpful discussions about the role of high-throughput science and for his contribution to Section 2.

We acknowledge the work of Benjamin Dobell, Aidan Steele, Ashley Taylor and David Warner, Monash University Faculty of I.T. students who worked on the initial Nimrod EC2 actuator prototype. We also thank Neil Soman for assistance in using the Eucalyptus Public Cloud.

Author information

Authors and Affiliations

  1. Faculty of Information Technology, Monash University, Clayton, Australia

    Blair Bethwaite

Authors
  1. Blair Bethwaite
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Abramson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fabian Bohnert
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Slavisa Garic
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Colin Enticott
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tom Peachey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Blair Bethwaite .

Editor information

Editors and Affiliations

  1. , School of Computing, University Derby, Kedleston Road, Derby, DE22 1GB, United Kingdom

    Nick Antonopoulos

  2. Dept. Computing, University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom

    Lee Gillam

Rights and permissions

Reprints and permissions

Copyright information

© 2010 Springer London

About this chapter

Cite this chapter

Bethwaite, B., Abramson, D., Bohnert, F., Garic, S., Enticott, C., Peachey, T. (2010). Mixing Grids and Clouds: High-Throughput Science Using the Nimrod Tool Family. In: Antonopoulos, N., Gillam, L. (eds) Cloud Computing. Computer Communications and Networks. Springer, London. https://doi.org/10.1007/978-1-84996-241-4_13

Download citation

  • DOI: https://doi.org/10.1007/978-1-84996-241-4_13

  • Published:

  • Publisher Name: Springer, London

  • Print ISBN: 978-1-84996-240-7

  • Online ISBN: 978-1-84996-241-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation