Skip to main content
SpringerLink
Log in

Language and Runtime Support for Automatic Configuration and Deployment of Scientific Computing Software over Cloud Fabrics

  • Published:
Journal of Grid Computing Aims and scope Submit manuscript

Abstract

In this paper, we present the design and implementation of Neptune, a simple, domain-specific language based on the Ruby programming language. Neptune automates the configuration and deployment of scientific software frameworks over disparate cloud computing systems. Neptune integrates support for MPI, MapReduce, UPC, X10, StochKit, and others. We implement Neptune as a software overlay for the AppScale cloud platform and extend AppScale with support for elasticity and hybrid execution for scientific computing applications. Neptune imposes no overhead on application execution, yet significantly simplifies the application deployment process, enables portability across cloud systems, and promotes lock-in avoidance by specific cloud vendors.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Amazon Simple Storage Service (Amazon S3): http://aws.amazon.com/s3/. Last accessed 31 December 2011

  2. Armstrong, J., Virding, R., Wikström, C., Williams, M.: Concurrent Programming in ERLANG (1993)

  3. Boto: http://code.google.com/p/boto/. Last accessed 31 December 2011

  4. Bunch, C., Chohan, N., Krintz, C., Chohan, J., Kupferman, J., Lakhina, P., Li, Y., Nomura, Y.: An evaluation of distributed datastores using the AppScale Cloud Platform. In: IEEE International Conference on Cloud Computing (2010)

  5. Bunch, C., Chohan, N., Krintz, C., Shams, K.: Neptune: a domain specific language for deploying HPC software on cloud platforms. In: ACM Workshop on Scientific Cloud Computing (2011)

  6. Cassandra Operations: http://wiki.apache.org/cassandra/Operations

  7. Charles, P., Grothoff, C., Saraswat, V., Donawa, C., Kielstra, A., Ebcioglu, K., von Praun, C., Sarkar, V.: X10: an object-oriented approach to non-uniform cluster computing. SIGPLAN Not. 40, 519–538 (2005)

    Article  Google Scholar 

  8. Chohan, N., Bunch, C., Krintz, C., Nomura, Y.: Database-agnostic transaction support for cloud infrastructures. In: IEEE International Conference on Cloud Computing (2011)

  9. Chohan, N., Bunch, C., Pang, S., Krintz, C., Mostafa, N., Soman, S., Wolski, R.: AppScale: scalable and open AppEngine application development and deployment. In: ICST International Conference on Cloud Computing (2009)

  10. Daigle, B.J., Roh, M.K., Gillespie, D.T., Petzold, L.R.: Automated estimation of rare event probabilities in biochemical systems. J. Phys. Chem. 134, 044110 (2011)

    Article  Google Scholar 

  11. Dean, J., Ghemawat, S.: MapReduce: simplified data processing on large clusters. In: Proceedings of 6th Symposium on Operating System Design and Implementation (OSDI), pp. 137–150 (2004)

  12. Drawert, B., Lawson, M.J., Petzold, L., Khammash, M.: The diffusive finite state projection algorithm for effficient simulation of the stochastic reaction-diffusion master equation. J. Phys. Chem. 132(7), 074101-1 (2010)

    Article  Google Scholar 

  13. El-Ghazawi, T., Cantonnet, F.: UPC performance and potential: a NPB experimental study. In: Proceedings of the 2002 ACM/IEEE Conference on Supercomputing. Supercomputing ’02, pp. 1–26. IEEE Computer Society Press, Los Alamitos, CA, USA (2002)

  14. El-Samad, H., Kurata, H., Doyle, J.C., Gross, C.A., Khammash, M.: Surviving heat shock: control strategies for robustness and performance. Proc. Natl. Acad. Sci. USA 102(8), 2736–2741 (2005)

    Article  Google Scholar 

  15. Engaging the Missing Middle: http://www.hpcinthecloud.com/features/Engaging-the-Missing-Middle-in-HPC-95750644.html. Last accessed 31 December 2011

  16. Gropp, W., Lusk, E., Doss, N., Skjellum, A.: A high-performance, portable implementation of the MPI message passing interface standard. Parallel Comput. 22(6), 789–828 (1996)

    Article  MATH  Google Scholar 

  17. Hadoop Distributed File System: http://hadoop.apache.org. Last accessed 31 December 2011

  18. Heroku Learns from Amazon EC2 Outage: http://searchcloudcomputing.techtarget.com/news/1378426/Heroku-learns-from-Amazon-EC2-outage. Last accessed 31 December 2011

  19. Hindman, B., Konwinski, A., Zaharia, M., Ghodsi, A., Joseph, A., Katz, R., Shenker, S., Stoica, I.: Mesos: a platform for fine-grained resource sharing in the data center. In: Networked Systems Design and Implementation (2011)

  20. Hucka, M., Finney, A., Sauro, H.M., Bolouri, H., Doyle, J.C., Kitano, H., the rest of the SBML Forum, Arkin, A.P., Bornstein, B.J., Bray, D., Cornish-Bowden, A., Cuellar, A.A., Dronov, S., Gilles, E.D., Ginkel, M., Gor, V., Goryanin, I.I., Hedley, W.J., Hodgman, T.C., Hofmeyr, J.-H., Hunter, P.J., Juty, N.S., Kasberger, J.L., Kremling, A., Kummer, U., Le, N., NovÃĺre, Loew, L.M., Lucio, D., Mendes, P., Minch, E., Mjolsness, E.D., Nakayama, Y., Nelson, M.R., Nielsen, P.F., Sakurada, T., Schaff, J.C., Shapiro, B.E., Shimizu, T.S., Spence, H.D., Stelling, J., Takahashi, K., Tomita, M., Wagner, J., Wang, J.: The systems biology markup language (SBML): a medium for representation and exchange of biochemical network models. Bioinformatics 19(4), 524–531 (2003)

    Article  Google Scholar 

  21. Kaiser, H., Merzky, A., Hirmer, S., Allen, G., Seidel, E.: The SAGA C++ reference implementation: a milestone toward new high-level Grid applications. In: Proceedings of the 2006 ACM/IEEE Conference on Supercomputing, SC ’06. ACM, New York, NY, USA (2006)

  22. Keahey, K., Freeman, T.: Nimbus or an open source cloud platform or the best open source EC2 no money can buy. In: Supercomputing (2008)

  23. Koslovski, G., Huu, T.T., Montagnat, J., Vicat-Blanc, P.: Executing distributed applications on virtualized infrastructures specified with the VXDL language and managed by the HIPerNET framework. In: ICST International Conference on Cloud Computing (2009)

  24. Krintz, C., Bunch, C., Chohan, N.: AppScale: Open-Source Platform-A s-A-Service. Technical Report 2011-01, University of California, Santa Barbara (2011)

  25. Lustre: http://www.lustre.org/. Last accessed 31 December 2011

  26. Nagel, W.E., Arnold, A., Weber, M., Hoppe, H.-Ch., Solchenbach, K.: VAMPIR: visualization and analysis of MPI resources. Supercomputer 12, 69–80 (1996)

    Google Scholar 

  27. Nurmi, D., Wolski, R., Grzegorczyk, C., Obertelli, G., Soman, S., Youseff, L., Zagorodnov, D.: The eucalyptus open-source cloud-computing system. In: IEEE International Symposium on Cluster Computing and the Grid. http://open.eucalyptus.com/documents/ccgrid2009.pdf (2009). Last accessed 31 December 2011

  28. Pbspro home page: http://www.altair.com/software/pbspro.htm. Last accessed 31 December 2011

  29. RightScale: RightScale Gems http://rightaws.rubyforge.org/. Last accessed 31 December 2011

  30. Rolfe, T.J.: A specimen MPI application: N-Queens in parallel. Inroads (bulletin of the ACM SIG on Computer Science Education), vol. 40(4) (2008)

  31. Ruby language: http://www.ruby-lang.org. Last accessed 31 December 2011

  32. Ruby on Rails: http://www.rubyonrails.org. Last accessed 31 December 2011

  33. Sanft, K.R., Wu, S., Roh, M., Fu, J., Lim, R.K., Petzold, L.R.: StochKit2: software for discrete stochastic simulation of biochemical systems with events. Bioinformatics (2011)

  34. Shen, K., Tang, H., Yang, T.: Adaptive two-level thread Management for fast MPI execution on shared memory machines. In: Proceedings of ACM/IEEE SuperComputing ’99 (1999)

  35. StochKit: http://www.cs.ucsb.edu/cse/StochKit/. Last accessed 31 December 2011

Download references

Author information

Authors and Affiliations

  1. Computer Science Department, University of California, Santa Barbara, CA, USA

    Chris Bunch, Brian Drawert, Navraj Chohan, Chandra Krintz & Linda Petzold

  2. Jet Propulsion Laboratory, California Institute of Technology, Pasadena, CA, USA

    Khawaja Shams

Authors
  1. Chris Bunch
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Brian Drawert
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Navraj Chohan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chandra Krintz
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Linda Petzold
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Khawaja Shams
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Chris Bunch.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Bunch, C., Drawert, B., Chohan, N. et al. Language and Runtime Support for Automatic Configuration and Deployment of Scientific Computing Software over Cloud Fabrics. J Grid Computing 10, 23–46 (2012). https://doi.org/10.1007/s10723-012-9213-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10723-012-9213-8

Keywords

Mathematics Subject Classifications (2010)

Search

Navigation