skip to main content
10.1145/2756594.2756597acmconferencesArticle/Chapter ViewAbstractPublication PageshpdcConference Proceedingsconference-collections
research-article

A Framework for Realizing Software-Defined Federations for Scientific Workflows

Published: 16 June 2015 Publication History

Abstract

Federated computing has been shown to be an effective model for harnessing the capabilities and capacities of geographically- distributed resources in order to solve large science and en- gineering problems. However, traditional High Performance Computing (HPC) based federation models can be restrictive as they present users with a pre-defined set of resources and do not allow federations to evolve in response to changing resources or application needs. As emerging application workflows and the underlying resources become increasingly dynamic and exhibit changing requirements and constraints, they cannot be easily supported by such federation models. Instead, new federation models that are capable of dynamically adapting to these emerging needs are required. In this paper, we present a programmable dynamic federation model that uses software-defined environment concepts to drive the federation process and seamlessly adapt resource compositions at runtime. The resulting software-defined federation adapts to meet both requirements and constraints set by the user, application, and/or resource providers. In this paper we present the design and prototype implementation of such software-defined federation model, and demonstrate its operation and performance through a use case where heterogeneous, geographically distributed resources are federated based on user specifications, and the federation evolves over time following the requirements and constraints defined by the user.

References

[1]
https://www-304.ibm.com/partnerworld/wps/servlet/ContentHandler/pw_com_sol_software-defined-environment.
[2]
http://aws.amazon.com/ec2/purchasing-options/spot-instances.
[3]
https://www.chameleoncloud.org.
[4]
M. AbdelBaky, M. Parashar, K. E. Jordan, H. Jamjoom, et al. Enabling high-performance computing as a service. Computer, (10):72--80, 2012.
[5]
G. Allen and D. Katz. Computational science, infrastructure and interdisciplinary research on university campuses: Experiences and lessons from the center for computation & technology. Technical Report CCT-TR-2010-1, LSU.
[6]
F. Berman, G. Fox, and A. J. Hey. Grid Computing: Making the Global Infrastructure a Reality. John Wiley & Sons, 2003.
[7]
L. F. Bittencourt, C. R. Senna, and E. R. Madeira. Enabling execution of service workflows in grid/cloud hybrid systems. In IEEE/IFIP Network Operations and Management Symp. Wksps., pages 343--349, 2010.
[8]
N. Bobroff, L. Fong, S. Kalayci, Y. Liu, J. Martinez, I. Rodero, S. Sadjadi, and D. Villegas. Enabling interoperability among meta-schedulers. In CCGrid, pages 306--315, 2008.
[9]
G. Breiter, M. Behrendt, M. Gupta, et al. Software defined environments based on tosca in ibm cloud implementations. IBM Journal of Research and Development, 58(2):1--10, 2014.
[10]
A. Celesti, F. Tusa, M. Villari, and A. Puliafito. How to enhance cloud architectures to enable cross-federation. In CLOUD, pages 337--345, 2010.
[11]
D. Das. Integrating cloud service deployment automation with software-defined environments. 2014.
[12]
M. D. De Assunçao, A. Di Costanzo, and R. Buyya. Evaluating the cost-benefit of using cloud computing to extend the capacity of clusters. In Proceedings of the 18th ACM international symposium on High performance distributed computing, pages 141--150. ACM, 2009.
[13]
M. Dias de Assuncao, R. Buyya, and S. Venugopal. InterGrid: a case for internetworking islands of grids. Concurrency Computat. Pract. Exper., 20(8):997--1024, 2008.
[14]
J. Diaz-Montes, M. AbdelBaky, M. Zou, and M. Parashar. Cometcloud: Enabling software-defined federations for end-to-end application workflows. IEEE Internet Computing, 19(1):69--73, 2015.
[15]
J. Diaz-Montes, Y. Xie, I. Rodero, J. Zola, B. Ganapathysubramanian, and M. Parashar. Exploring the use of elastic resource federations for enabling large-scale scientific workflows. In MTAGS workshop, pages 1--10, 2013.
[16]
J. Diaz-Montes, M. Zou, R. Singh, S. Tao, and M. Parashar. Data-driven workflows in multi-cloud marketplaces. In IEEE Cloud 2014, 2014.
[17]
E. E. Huedo, R. Montero, and I. Llorente. A recursive architecture for hierarchical grid resource management. Future Generation Computer Systems, 25:401--405, 2009.
[18]
E. Elmroth and J. Tordsson. A standards-based grid resource brokering service supporting advance reservations, coallocation, and cross-grid interoperability. Concurrency and Computation: Practice and Experience, 21(18):2298--2335, 2009.
[19]
R. Fichera, D. Washburn, and E. Chi. The software-defined data center is the future of infrastructure architecture. Forrester Research, 2012.
[20]
R. T. Fielding and R. N. Taylor. Principled design of the modern web architecture. ACM Transactions on Internet Technology (TOIT), 2(2):115--150, 2002.
[21]
G. Garzoglio, T. Levshina, M. Rynge, et al. Supporting shared resource usage for a diverse user community: the OSG experience and lessons learned. J. of Physics: Conf. Series, 396, 2012.
[22]
I. Gorton, Y. Liu, and J. Yin. Exploring architecture options for a federated, cloud-based system biology knowledgebase. In Cloud Computing Technology and Science (CloudCom), 2010 IEEE 2nd Intl. Conf. on, pages 218--225, 2010.
[23]
J.-M. Kang, H. Bannazadeh, H. Rahimi, T. Lin, M. Faraji, and A. Leon-Garcia. Software-defined infrastructure and the future central office. In Communications Workshops (ICC), 2013 IEEE Intl. Conf. on, pages 225--229. IEEE, 2013.
[24]
A. Kertész and P. Kacsuk. Grid meta-broker architecture: Towards an interoperable grid resource brokering service. In Euro-Par 2006, pages 112--115. Springer, 2007.
[25]
A. Kertész, G. Kecskemeti, M. Oriol, et al. Enhancing federated cloud management with an integrated service monitoring approach. Journal of grid computing, 11(4):699--720, 2013.
[26]
H. Kim and M. Parashar. Cometcloud: An autonomic cloud engine. Cloud Computing: Principles and Paradigms, pages 275--297, 2011.
[27]
C. Li, B. Brech, S. Crowder, D. Dias, H. Franke, M. Hogstrom, D. Lindquist, G. Pacifici, S. Pappe, B. Rajaraman, et al. Software defined environments: An introduction. IBM Journal of Research and Development, 58(2):1--11, 2014.
[28]
Z. Li and M. Parashar. A computational infrastructure for grid-based asynchronous parallel applications. In HPDC, pages 229--230, 2007.
[29]
T. Lin, J.-M. Kang, H. Bannazadeh, and A. Leon-Garcia. Enabling sdn applications on software-defined infrastructure. In Network Operations and Management Symposium (NOMS), 2014 IEEE, pages 1--7. IEEE, 2014.
[30]
H. Mohamed and D. Epema. KOALA: a co-allocating grid scheduler. Concurrency Computat. Pract. Exper., 20:1851--1876, 2008.
[31]
S. Ostermann, R. Prodan, and T. Fahringer. Extending grids with cloud resource management for scientific computing. In Intl. Conf. on Grid Computing (GRID), pages 42--49, 2009.
[32]
M. Parashar, M. AbdelBaky, I. Rodero, and A. Devarakonda. Cloud paradigms and practices for computational and data-enabled science and engineering. Computing in Science & Engineering, 15(4):10--18, 2013.
[33]
M. Parashar and C. Lee. Special issue on grid computing. Proceedings of the IEEE, 93(3), 2005.
[34]
P. Riteau, M. Tsugawa, A. Matsunaga, et al. Large-scale cloud computing research: Sky computing on FutureGrid and Grid'5000. In ERCIM News, 2010.
[35]
B. Rochwerger, D. Breitgand, E. Levy, et al. The reservoir model and architecture for open federated cloud computing. IBM J. of Research and Development, 53, 2009.
[36]
C. Vazquez, E. Huedo, R. Montero, and I. Llorente. Dynamic provision of computing resources from grid infrastructures and cloud providers. In Intl. Conf. on Grid and Pervasive Computing (GPC), pages 113--120, 2009.
[37]
D. Villegas, N. Bobroff, I. Rodero, J. Delgado, Y. Liu, A. Devarakonda, L. Fong, S. M. Sadjadi, and M. Parashar. Cloud federation in a layered service model. Journal of Computer and System Sciences, 78(5):1330--1344, 2012.
[38]
J.-S. Vöckler, G. Juve, E. Deelman, M. Rynge, and B. Berriman. Experiences using cloud computing for a scientific workflow application. In Workshop on Scientific Cloud Computing, pages 402--412, 2011.
[39]
P. Wieder, J. Seidel, O. Wäldrich, et al. Using sla for resource management and scheduling-a survey. In Grid Middleware and Services, pages 335--347. Springer, 2008.

Cited By

View all
  • (2018)Software-defined environments for science and engineeringInternational Journal of High Performance Computing Applications10.5555/3195474.319547532:1(104-122)Online publication date: 1-Jan-2018
  • (2017)Software-defined environments for science and engineeringThe International Journal of High Performance Computing Applications10.1177/109434201771070632:1(104-122)Online publication date: 15-Jun-2017
  • (2017)Computing in the Continuum: Combining Pervasive Devices and Services to Support Data-Driven Applications2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2017.323(1815-1824)Online publication date: Jun-2017
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
BigSystem '15: Proceedings of the 2nd International Workshop on Software-Defined Ecosystems
June 2015
48 pages
ISBN:9781450335683
DOI:10.1145/2756594
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 June 2015

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. autonomic computing
  2. dynamic resource provision
  3. dynamic workflows
  4. federated computing
  5. software-defined environments
  6. software-defined federation

Qualifiers

  • Research-article

Funding Sources

Conference

HPDC'15
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6 of 8 submissions, 75%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)1
  • Downloads (Last 6 weeks)0
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2018)Software-defined environments for science and engineeringInternational Journal of High Performance Computing Applications10.5555/3195474.319547532:1(104-122)Online publication date: 1-Jan-2018
  • (2017)Software-defined environments for science and engineeringThe International Journal of High Performance Computing Applications10.1177/109434201771070632:1(104-122)Online publication date: 15-Jun-2017
  • (2017)Computing in the Continuum: Combining Pervasive Devices and Services to Support Data-Driven Applications2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS)10.1109/ICDCS.2017.323(1815-1824)Online publication date: Jun-2017
  • (2017)Towards Distributed Software-Defined EnvironmentsProceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing10.1109/CCGRID.2017.30(703-706)Online publication date: 14-May-2017
  • (2017)Enabling Distributed Software-Defined Environments Using Dynamic Infrastructure Service CompositionProceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing10.1109/CCGRID.2017.104(274-283)Online publication date: 14-May-2017
  • (2015)Docker containers across multiple clouds and data centersProceedings of the 8th International Conference on Utility and Cloud Computing10.5555/3233397.3233457(368-371)Online publication date: 7-Dec-2015
  • (2015)Docker Containers across Multiple Clouds and Data Centers2015 IEEE/ACM 8th International Conference on Utility and Cloud Computing (UCC)10.1109/UCC.2015.58(368-371)Online publication date: Dec-2015

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media