The InstaGENI initiative: An architecture for distributed systems and advanced programmable networks
Introduction
The National Science Foundation’s Global Environment for Network Innovations (GENI) [10], [30], managed through the GENI Project Office (GPO) is an effort to build an environment for large-scale networking experimental research. Experimental projects involve novel network services, architecture and technologies, long-running distributed systems and cloud services.
Deploying an environment to support these research activities requires a platform that is deeply programmable, widely deployed, highly instrumented and directly accessible to end-users at many layers. These concepts formed the principal design criteria of GENI, now entering its fifth year of development.
The GENI platform has been developed by multiple US network research communities over the past four years into a coherent set of control frameworks and aggregate managers, underlying resource infrastructure, and an overlay wide-area national network testbed, with several international extensions. This distributed environment has already been used to support multiple major network research experiments. The next stage of the GENI project is the Mesoscale deployment, which will allow advanced GENI services to be deployed deep within all parts of the environment at national and global scale in concert with related projects in the United States and around the world [14], [31], [39], [46], [64], [67].
The GENI Mesoscale deployment envisions a network of distributed clusters, where each cluster features an internal network that runs the OpenFlow [47], [54], [65] framework natively, and operates one or more GENI control frameworks and aggregate managers. The GENI Project Office selected two designs for the Mesoscale clusters: the ExoGENI [5] design based on the ORCA control framework [4], [7], from the Renaissance Computing Institute of North Carolina and Duke University, and InstaGENI, a collaborative partnership effort of HP, Northwestern, Princeton, the University of Utah, and the Open Networking Foundation. InstaGENI features a lightweight, expandable cluster design featuring integration with the FlowVisor OpenFlow Aggregate Manager (FOAM) [26], ProtoGENI [61] and PlanetLab [59] Aggregate Managers, and an advanced network architecture implemented nationally to exploit the capabilities of advanced research networks.
The challenges involved in building a distributed network, using various host facilities, for experimental purposes are well-known [57]. The primary challenge is that each node in the network is hosted by a separate individual donor institution, which offers bandwidth and maintenance services as a gift to the community. Under these circumstances, the node and the experiments running on it must behave as a pilot guest; it should not unduly consume resources, it should require hands-on maintenance very infrequently, and it should not do harmful things to the institution, or to third parties who will interpret the damage as emanating from the institution. This means every node in the network must have two distinct administrators: the central authority, and the host institution. It further means that the node and each experiment hosted on it must be heavily instrumented, and either the central authority or the local authority must be able to shut it down immediately in the event of abuse or excessive resource consumption.
All of this is well known from PlanetLab; GENI merely deepens the requirement, since it introduces programmable networks and heterogenous computing environments (VMs, containers, and bare-metal machines running a variety of operating systems; PlanetLab permitted only Linux containers under VServers, which simplified administration significantly).
InstaGENI takes its name and its inspiration from its commitment and its plan to build a working, usable, live, highly distributed experimental facility immediately, capable of running any existing GENI research experiment on an intercontinental scale. InstaGENI architectural concepts are cognizant of the requirement for easy adoptability. The original Internet grew exponentially in the 1990s because its core software and protocols could be deployed on almost any computer and its messages could be carried by and across a very wide variety of existing communications systems. In this spirit, InstaGENI placed a premium on developing a working facility immediately, using as far as possible only existing software modules and commodity, off-the-shelf (COTS) hardware components. Expensive, gold-plated designs are difficult to fund, deploy, and maintain. The base InstaGENI design is engineered for affordability; it is easy to expand a simple, inexpensive base installation. Our design goal in each component was to choose a minimal, expandable configuration in place of a comprehensive, expensive configuration. COTS components were an important consideration: it was vital that now and in the future, InstaGENI sites be able to upgrade and expand their installation using components available from a very wide range of suppliers.
InstaGENI is designed for easy and painless integration into an existing campus environment, with minimal change to the existing network. The value of a notional specialized facility is hard to quantify and justify, especially because it must, in part, depend on existing production infrastructure, yet be sufficiently distanced from that infrastructure to avoid limitations to its goal of supporting advanced experimental research. Further, production network service organizations are much more comfortable supporting generalized common communication services for large communities than specialized facilities for small communities. Too often in the past such specialized facilities have been funded and implemented at great expense only to be abandoned after the end of a project. InstaGENI was developed to address these challenges – designing, establishing, and operating a next generation environment for the network researchers as well as providing appropriate procedures and communications to provide for a sustainable, persistent, cost effective resource. These activities are undertaken in partnership with key stakeholder organizations to advance the goals of the GENI project including the National Science Foundation, the GPO, communities associated with national networks, regional networks, campus networks, international networks, emerging communities that wish to create next generation applications based on GENI, affiliated regional testbeds [48], and others.
A further goal of InstaGENI is creating extensions beyond its initial deployment in the United States. Excellent advanced network research is being undertaken at universities and private laboratories in countries throughout the world. From its initial conceptualization, the InstaGENI design was intended to be globally deployable, using only free and open-source software and globally available hardware. A key goal is to establish and interconnect advanced network research sites around the world.
One important component to our strategy in addressing these challenges is making progress as soon as possible in creating the facility itself – bring it up, get it used, then build it out. The value of an extant facility is easy to see, and it is often greater than the most optimistic estimates of its proponents. Establishing this facility among multiple sites provides the most important means of communicating its value and purpose.
InstaGENI is designed to be up and running out of the box, and instantly connected through appropriate communication services to GENI. To make InstaGENI racks turnkey simple, we concentrated on three strategies:
- 1.
Homogenous hardware: diverse hardware from multiple manufacturers proved to be a support issue for earlier distributed environments. Even nominally standard hardware –x86-based servers running a standard Linux image – showed enough variation, specifically in drivers for diverse components – to be a significant uptime issue. InstaGENI racks all feature HP ProLiant servers and standard HP E Series switches.
- 2.
Remote management: A distributed cluster must be easily maintainable without hands-on, on-site administration. InstaGENI features HP Integrated Lights-Out Management (iLO) [36] on each server, which offers in situ remote monitoring and remote management, including reboot and diagnosis.
- 3.
Prebuilt configurations: The first two InstaGENI Racks were shipped to the University of Utah and the GENI Project Office, and the ProtoGENI and FOAM control frameworks were ported to the InstaGENI Racks. PlanetLab node images were pre-loaded on the racks, and the InstaGENI-PlanetLab Central service is established and running. Preconfigured flash drives are shipped with the racks, and the boss node of the rack will boot GENI software directly. Racks arrive pre-wired on campus.
The efficacy of the GENI Infrastructure for network and Cloud experimentation has been amply demonstrated over the last few years. Readers should see http://groups.geni.net/geni/wiki/GENIBibliography for a full list of experiments performed and underway on the existing GENI facility. In particular, GENI has been used for network measurement and experimentation [1], [12], [32], [33], distributed Clouds [9], network-based manufacturing [11], smart power systems [16], network virtualization [18], federated testbeds [21], security experiments, and networked large-scale science workloads [43], among many others. We therefore considered the architecture to have met its major design goals: our task was to distribute the GENI architecture across the wide area.
The rest of this paper is organized as follows. In Section 2, we discuss the background technologies of the InstaGENI rack; specifically, the key components of the GENI architecture. In Section 3, we review the InstaGENI software architecture, with its triple pillars of isolated OpenFlow networks, ProtoGENI for Hardware-as-a-Service, and PlanetLab for Containers-as-a-Service. In Section 4, we briefly review the InstaGENI Hardware Architecture, driven by the requirements of the InstaGENI software stack. In Section 5, we discuss the GENI Network architecture and deployment. In Section 6 we discuss operational and maintenance issues. In Section 7, we discuss a user’s view of the InstaGENI Network. In Section 8, we conclude and describe opportunities for future design enhancements, deployments, and interconnections.
Section snippets
Background: Slivers and slices
Above all, InstaGENI is designed to meet the primary goals of the GENI project, which are directed at creating a highly customizable environment for innovative research, without restrictions and pre-conditions and with complete direct control over all resource elements. Consequently, InstaGENI is a deployment platform for GENI control frameworks, which enable researchers to discover, integrate, and experiment with GENI resources. Fundamentally, GENI is a platform for the deployment of virtual
The InstaGENI software architecture
The InstaGENI software architecture is designed to provide deeply configurable and deeply-programmable Infrastructure-as-a-Service and customizable OpenFlow networks as a service. A critical design consideration was user familiarity: an InstaGENI rack should behave exactly as a small Emulab, with an embedded OpenFlow switch to permit the construction of virtual networks. Further, a collection of InstaGENI racks should behave exactly as a distributed Emulab. While new capabilities and functions
The InstaGENI hardware architecture
The InstaGENI hardware design was driven by three principal considerations. First, the goal was to support the software architecture described above; InstaGENI is fundamentally characterized by code, not boxes. Second, commodity off-the-shelf hardware was to be used, for reasons of maintenance and operations. When something broke, it had to be easy to fix or replace. Finally, a large collection of inexpensive racks is preferred to a smaller collection of more capable racks. It is relatively
The InstaGENI network architecture
The InstaGENI Network Architecture was driven by two principal considerations: the need to offer layer-2 services across the wide area and the need to permit deep programmability and end-to-end OpenFlow capability across the entire Mesoscale.
The InstaGENI design required close consideration of three major classes of WAN connectivity. One class of WAN resources consists of those that constitute core foundation infrastructure, including those that support management planes, control planes and
Operations and maintenance considerations
Software installation for the ProtoGENI control nodes is accomplished through virtual machine images. The Xen instance on the control node first has a basic configuration (such as its IP address) set by local administrators. Generic control nodes images, to run inside the Xen VM, are provided by the ProtoGENI team and are customized by local administrators. In particular, ProtoGENI has developed software that allows the local administrators to fill in a configuration file describing the local
User interface
Insofar as possible, the goal of the InstaGENI design team was seamless interoperation with other GENI Aggregates, and support for seamless federation. This required that InstaGENI nodes and networks be allocated through the same tool used for other aggregate managers. Ideally, GENI users should be as little aware as possible whether the resources they are requesting are on an InstaGENI rack, a ProtoGENI cluster, or an ExoGENI rack.
This goal has some obvious limits. For one thing, bare-metal
Conclusions and status
In this paper we have described InstaGENI, a complete architecture for the GENI Mesoscale deployment and beyond, capable of supporting an extremely wide range of advanced network research experiments. InstaGENI is designed to be out-of-the-box GENI-capable, easy to maintain and deploy, cost effective, and capable of running GENI experiments instantly. We are scheduled to deploy 32 InstaGENI racks by the end of 2013, and integrate those racks with other Mesoscale resources. As of this writing
Acknowledgements
This work is supported by the GENI Project Office under contract to the National Science Foundation. The authors wish to thank the anonymous reviewers for thoughtful and insightful comments on the manuscript, the issue editor, Prof. James P.G. Sterbenz, for help in shepherding this paper through, to Harry Rudin for editorial comments, to the staff at the GENI Project Office for assistance in deploying and testing the racks, to the 32 InstaGENI rack sites for their help in deploying the
Nick Bastin is the founder of Barnstormer Softworks. He was formerly a member of the technical staff at BigSwitch Networks, the OpenFlow Engineering Manager at Stanford University, and a software architect at OPNET technologies, among other jobs building, operating, modeling, and analyzing networks since 1996. He was the principal architect of the OPNET rules-based network analysis platform, is the author and maintainer of the FOAM resource management tool, and is an active participant in the
References (63)
- Jay Aikat, Shaddi Hasan, Kevin Jeffay, F. Donelson Smith, Discrete-approximation of measured round trip time...
- Thomas Anderson, Michael K. Reiter, GENI: Global Environment for Network Innovations Distributed Services Working...
- Ilia Baldine, Yufeng Xin, Anirban Mandal, Chris Heermann, Jeff Chase, Varun Marupadi, Aydan Yumerefendi, David Irwin,...
- Ilia Baldine, Yufeng Xin, Anirban Mandal, Paul Ruth, Aydan Yumerefendi, Jeff Chase, ExoGENI: a multi-domain...
- et al.
Xen and the art of virtualization
SOSP
(2003) - A. Bavier, M. Bowman, B. Chun, D. Culler, S. Karlin, S. Muir, L. Peterson, T. Roscoe, T. Spalink, M. Wawrzoniak,...
- Andy Bavier, Yvonne Coady, Tony Mack, Chris Matthews, Joe Mambretti, Rick McGeer, Paul Mueller, Alex Snoeren, Marco...
- Mark Berman, Jeffrey S. Chase, Lawrence Landweber, Akihiro Nakao, Max Ott, Dipankar Raychaudhuri, Robert Ricci, Ivan...
- Alex Berryman, Prasad Calyam, Prasad, Joe Cecil, George B. Adams, George B. Douglas Comer, Advanced manufacturing use...
- Ethan Blanton, Sarbajit Chatterjee, Sriharsha Gangam, Sumit Kala, Deepti Sharma, Sonya Fahmy, Puneet Sharma. Design and...
Designing a federated testbed as a distributed system
TRIDENTCOM 2012
The design of an instrumentation system for federated and virtualized network testbeds
IEEE NOMS
Cited by (23)
NASOR: A network slicing approach for multiple Autonomous Systems
2021, Computer CommunicationsCitation Excerpt :Similarly, approaches using Virtual LAN (VLAN), Generic Routing Encapsulation (GRE), Virtual Extensible LAN (VxLAN), Network Service Header (NSH) claimed network slicing realizations [6,7]. However, to realize network slicing requirements, all these solutions lack some form of network programmability, isolation, management of control and data planes, and a distributed mechanism [7,8] able to cope with multiple domains. To the best knowledge of the authors, there are no well-known solutions capable of establishing network slice connectivity inter-ASs, except in particular domains such as data centers [9] or mobile network [10].
An instrumentation framework for the critical task of measurement collection in the future Internet
2014, Computer NetworksCitation Excerpt :While it was originally an integral part of OMF [35,36], which it was developed to support measurement for, it is a stand-alone instrumentation suite which can be used in other contexts. It is currently used within GENI [37,38] and various EU projects such as Fed4FIRE [39], OpenLab [40] or SmartSantander [41], amongst others. The version of OML at the time of this writing (2.11) only implements a subset of the proposed framework.
A university testbed for large-scale interconnection experiments on distributed applications
2014, Computer NetworksCitation Excerpt :Among the goals of this project are the provisioning of an open environment on which the networking research community can run experiments and providing programmable optical infrastructure to their members. Another project in this group, InstaGENI racks [14], will design, document, build, deploy, and support GENI racks at sites that are interested in experimentation and integration with GENI. Our testbed focuses on testing network management systems, while K-GENI addresses large management systems.
Designing and orchestrating reproducible experiments on federated networking testbeds
2014, Computer NetworksCitation Excerpt :These case-studies demonstrate the operation and ease-of- use of OMF-F in real-life. Our future plans include completing the authorisation scheme based on assertion sets, improving the performance of OMF-F’s messaging and event-handing systems, polishing the current implementation and documentation to attract more users (not only experimenters, but also developers contributing supports for new resources, such as the GENI Racks [24]), and extending the existing deployment base. We are collaborating with other teams within both the FIRE and GENI initiatives to provide a seamless OMF-F interface with various existing contributions, and with measurement analysis, management and curation systems.
GENI: A federated testbed for innovative network experiments
2014, Computer NetworksCitation Excerpt :The ExoGENI design, built by a team lead by RENCI (Renaissance Computing Institute), is emphasizing performance, building on IBM hardware and an OpenStack software platform [5]. The InstaGENI rack, built by a HP Labs-led team, is emphasizing affordability and building on HP hardware and a combined ProtoGENI and PlanetLab software base [29,30]. In addition to the SDN capability found in GENI racks, GENI-enabled campuses also deploy within the campus network a number of switches supporting SDNs.
Creating environments for innovation: Designing and implementing advanced experimental network research testbeds based on the Global Lambda Integrated Facility and the StarLight Exchange
2014, Computer NetworksCitation Excerpt :GENI has also implemented a general aggregate manager API to integrate these frameworks [50]. Through its Meso-Scale project, particularly the InstaGENI [51] and ExoGENI initiatives, the GENI research environment will be expanded to over 50 additional campus research sites. Eventually, as many as 200 sites may participate.
Nick Bastin is the founder of Barnstormer Softworks. He was formerly a member of the technical staff at BigSwitch Networks, the OpenFlow Engineering Manager at Stanford University, and a software architect at OPNET technologies, among other jobs building, operating, modeling, and analyzing networks since 1996. He was the principal architect of the OPNET rules-based network analysis platform, is the author and maintainer of the FOAM resource management tool, and is an active participant in the NSF GENI project.
Andy Bavier is a Research Scholar at Princeton University. He has been building testbeds for the networking research community since 2002. He is an architect and designer of the PlanetLab, VINI, and VICCI testbeds, and is an active participant in the NSF GENI project.
Ms. Blaine is a Project Manager at HP Labs. She received a BachelorÕs of Science Degree in Business Administration with a concentration in Marketing from San Jose State University in 2009.
Jim Chen, Associate Director, International Center for Advanced Internet Research (iCAIR) at Northwestern University, is responsible for the center’s research infrastructure design and engineering. He leads multiple projects focus on the design and development of high performance platforms for advanced network systems and advanced network applications. Recently his focus areas include 100G network exchanges and data movement, high performance digital media networks, high resolution 2D/3D digital media streaming over networks, international collaboration virtual environments for science, programmable network testbeds, science cloud networks, and virtual science environments. Before joining iCAIR, Jim was a coordinator for Technology Testbeds at Northwestern.
Narayan Krishnan is an Infrastructure Manager at Hewlett-Packard Laboratories (Labs). The infrastructure team delivers a focused portfolio of core services to facilitate research productivity, including the management of the central computing environment at Labs, and providing research computing facilities and services for researchers at all Labs sites worldwide.
Joe Mambretti is the Director of the International Center for Advanced Internet Research at Northwestern University (iCAIR, www.icair.org), the Director of the Metropolitan Research and Education Network (MREN, www.mren.org), co-Director of the StarLight international exchange (www.starlight.net), member of the Executive Committee for I-WIRE, principal investigator for iGENI and for multiple network research testbeds. iCAIR accelerates leading edge innovation and enhanced digital communications through advanced Internet technologies, in partnership with the international community. iCAIR accomplishes its mission by undertaking large-scale (e.g., global, national, regional, metro) projects focused on high performance resource intensive applications, advanced communications middleware, and optical and photonic networking. He is co-editor of “Grid Networks: Enabling Grids With Advanced Communications Technology,” published by Wiley.
Rick McGeer received his Ph. D. in Computer Science from the University of California at Berkeley in 1989. He was an Assistant Professor in the Computer Science Department at the University of British Columbia, until returning to UC-Berkeley as a Research Engineer in 1991. In 1993, he co-founded the Cadence Berkeley Laboratories, the research arm of Cadence Design Systems. In 1998, he co-founded Softface, Inc., where he remained as Chief Scientist until 2003. In 2003 he joined Hewlett-Packard Laboratories. He is currently a Distinguished Technologist in HP Enterprise Systems. He is author of over 90 papers and one book in the fields of Computer-Aided Design, circuit theory, programming languages, distributed systems, networking, and information system design. His research interests include logic synthesis, timing analysis, formal verification, circuit simulation, programming languages, networking, wide-area distributed systems, and cloud systems. He is an adjunct professor of Computer Science at the University of Victoria, Victoria, BC, Canada. He serves on the PlanetLab Steering Committee and as Chief Scientist of the US Ignite Foundation.
Robert Ricci is a Research Assistant Professor in the University of Utah’s School of Computing. He has been been building network testbeds for over a decade, beginning with the Emulab facility and continuing with the National Science Foundation’s GENI and PRObE facilities. He has done research on a variety of topics regarding testbed design, construction, and use, including resource mapping, control system design, and emulation of realistic network conditions.
Nicki Watts is Director and acting Managing Practice Principal for Federal DoD and DOE Consulting Services at Hewlett-Packard Company, managing a multimillion dollar delivery organization to include the product and program management of internet software, enterprise network management frameworks, base operating systems, and software development tools. She joined HP in 1999 as a Program Manager and is a certified PMP. Before joining HP, Nicki served for 24 years in the U.S. Air Force attaining the rank of full Colonel.