Abstract
In this work we investigate the coupling of Flex-E and X-Ethernet technologies as a means to realize interface slicing and fast switching respectively. Flex-E is investigated as a key technology that is able to split a physical interface into isolated sub-channels, decouple MAC rate from PHY rate and achieve hard bandwidth isolation. X-Ethernet technology introduces Ethernet PCS switching, eliminates table lookup and buffer queuing and will be investigated as a fast switching mechanism that is exploiting the PCS layer relay. We present evaluation results from implementation activities that showcase the ability of an integrated Flex-E/X-Ethernet solution to enable network slicing features in transport networks. Furthermore we present a technical approach on the way the control plane of Flex-E/X-Ethernet network can be integrated with an orchestration and management solution.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
Keywords
1 Introduction
In order to couple with the strain raised and satisfy the strict requirements of 5G systems, significant changes have been undergoing in transport networking. Under the umbrella of 3GPP-based mobile cellular networks, transport networks support connectivity services between the various Disaggregated Radio Access Networks (DA-RAN) network components (3GPP TR38.816, TR38.801).
With respect to the terminology used by 3GPP (3GPP TR23.799, TR38.803) as transport network we consider the Fronthaul (FH), Midhaul (MH) and Backhaul (BH) type of communication networks that are used to interconnect Network Functions (NFs). Note that NFs can be either Physical (PNFs) or Virtual Network Functions (VNFs). In the case where the DA-RAN paradigm is adopted, NFs reside in the Radio Unit (RU), Centralized Unit (CU), Distributed Unit (DU) and in the Core Network (CN) [1]. Furthermore, in 3GPP TS23.501 where the 5G Architecture is described, future network designs also consider network operations in the light of Network Slicing [8].
In this paper we investigate Flex-E and X-Ethernet technologies as a means to realize transport network slicing with the support of fast switching. Flex-E will be investigated as a key technology that is able to split a physical interface into isolated sub-channels, decouple MAC rate from PHY rate and achieve “hard” bandwidth isolation. X-Ethernet technology introduces Ethernet PCS switching, eliminates table lookup and buffer queuing and will be investigated as a fast switching mechanism that is exploiting PCS layer relay.
Traditional deployments where relying on fiber technologies and PDH T1/E1 connections, ATM, SONET/SDH, like also MPLS to support transport network connectivity services. However, the last decade we witness a spiraling adoption of Ethernet technology as a means to implement transport networking for the mobile network. A flat Ethernet solution simplifies provisioning and planning, while also helps to radically reduce OPEX. Solutions like Ethernet over MPLS, Ethernet over SONET/SDH, Packet over SONET (PoS), Ethernet over DWDM Ethernet over OTN and so on, are inexorably driving the Ethernet technology as the de facto standard to support mobile network connectivity. Open Internet working Forum (OIF), a global industry forum for advanced inter-operable optical networking solutions, devised Flex-E in OIF-FLEXE-01.0 − Flex Ethernet Implementation Agreement (IA). The key features of Flex-E are the ability to create larger links out of multiple slower links, multiplex lower speed traffic from different clients in higher speed links, while also utilize a specific portion of some PHY link. As we will analyze in the following Flex-E can serve as an ideal technology to realize the concept of Network Slicing in transport networks and offer guaranteed performance and services per tenant even in scenarios of extreme flow multiplexing.
The main contributions of this paper are the following:
-
We present the primitives of operation of Flex-E and X-Ethernet technologies as key enabling technologies towards slicing the transport network.
-
We present evaluation results from real testbed experimentation. As we will analyze Flex-E is able to provide strict throughput guarantees, while X-Ethernet is able to bring switching time cost up to level of few nanoseconds.
-
Our demonstrator considers the case of channelization where several lower-speed links are created for different ethernet clients.
-
We provide insight information on the way a SDN solution could be incorporated in an integrated platform and how an overall orchestration and management solution could be designed.
This paper is organized as follows. In Sect. 2 we present the related work. In Sect. 3, we present the primitives of operation of Flex-E and X-Ethernet technologies. In Sect. 4, we present evaluation results. In Sect. 5 we elaborate on the way programmability can be enabled in Flex-E, X-Ethernet networks and the way they can be integrated with orchestration and management systems. We conclude our study and present future research directions in Sect. 6.
2 Related Work
Flex-E enables Ethernet-based services to be mapped over a next-generation optical transport network with the most efficient utilization of capacity possible. It was originally conceived to meet the challenges of Internet Content Providers (ICPs) for higher capacities and dynamicity, and as a new mechanism for the data center interconnect (DCI). It was originally proposed by OIF in Interoperability Agreement (IA) OIF-FLEXE-01.0 on March 2016. A detailed description of the technology can be found by OIF in IA OIF-FLEXE-02.0 (2018), IA OIF-FLEXE-01.1 (2017), IA OIF-FLEXE-01.0 (2016) and [7]. The forthcoming Flex-E 2.1 project is about prototyping Flex-E over 50 Gbps PHY applications and extending the recently released Flex-E 2.0. Flex-E 2.1 will further specify a new Flex-E frame and multiplexing format and address applications with lower bandwidth needs. Flex-E neighbor discovery is discussed in IA OIF-FLEXE-ND-01.0 (2018). Potential use cases were considered within other organizations such as the Ethernet Alliance and described by certain industrial players like Google in [3]. Proprietary deployments considering an integrated approach of the control and management plane of Flex-E over OTN were also contributed by Huawei [12]. Regarding Flex-E implementation landscape Huawei incorporates Flex-E in PTN990 router series, IXIA presented a demo in OFC 2016 with Altera/Intel, with Flex-E 2x100GbE and Ciena provides the Flex-E Liquid Spectrum solution. A 400 Gbps of Flex-E traffic was sent over four bonded 100 GbE interfaces on network demonstration in OFC 2018 by OIF and Ethernet Alliance.
In [11] a survey on backhaul network technologies is presented with an analysis on the relevant synchronization issues. A detailed analysis on the topic of transport network slicing is presented in [6]. Virtualization techniques for different types of optical networks and technologies where granularities may vary, according to the node and link characteristics. For example, in a wavelength switched network with optical cross connects (OXCs) and Reconfigurable Optical Add-Drop Multiplexing (ROADMs) virtualization approaches are different from networks with sub-wavelength granularity of switching and control [10]. In [5] hardware and spectrum resource virtualization techniques for the optical network are investigated. An adaptive transponder design is analyzed for WXC/WDM, while also multiflow transponder mechanisms for right-sized superchannels with the necessary number of subtransponders. Flex-O is described in ITU-T G.709.1/Y.1331.1 recommendation. Flex-O provides OTN interfaces with comparable functionality as to what was introduced in Flex-E for Ethernet interfaces. It provides an interoperable system interface for OTUCn transport signals; while it enables higher capacity ODUflex and OTUCn, by means of bonding m standard-rate interfaces (see ITU-T G.709.1/Y.1331).
3 Transport Network Slicing with Flex-E and X-Ethernet
3.1 Flex-E Basic Operation and Concepts
In principle, in the OSI stack in the Data Link Layer Logical Link Control (LLC) is performing multiplexing network protocols over the same Media Access Control Sublayer (MAC), which is used for addressing and channel access control mechanisms. Reconciliation Sublayer (RS) processes PHY local/remote fault messages, while in the PHY Layer Physical Coding Sublayer (PCS) performs auto-negotiation and coding.
Flex-E technology is introduced as a thin layer, known as Flex-Shim, and being able to support data rates out of the conventional range offered by current Ethernet standards. In more detail, the Flex-E Shim layer (Fig. 1) is responsible for the mapping of Flex-E clients (Ethernet flows) to groups of PHYs. From a layering perspective Flex-E Shim is introduced between the Ethernet MAC and the PCS sublayers. The main idea behind Flex-E is to decouple the actual PHY layer speed from the MAC layer speed of a client. Flex-E is based on a time-division multiplexing mechanism that is able to drive the asynchronous Ethernet flows over a synchronous schedule over multiple PHY layers. The main operational components of Flex-E are depicted in Table 1.
Currently there are three operational scenarios supported by Flex-E, which describe different ways the MAC layer speed is related with the corresponding PHY speed, allowing a distinct manner for multiplexing clients in time:
-
Bonding: allows a MAC layer speed higher than a single PHY by grouping multiple PHYs to serve a flow (e.g. support a 200G MAC over two bonded 100GBASE-R PHYs).
-
Sub-rating: MAC layer speed is less than the actual PHY. Allows the MAC layer to use a portion of a PHY to serve a flow (e.g. support 50 Gbps MAC over a 100GBASE-R PHY).
-
Channelization: enables multiple Flex-E clients over a shared single PHY or bounded PHY via the means of time division multiplexing in the Flex-Shim (e.g. support 150 Gbps and a 50 Gbps MAC over two bonded 100GBASE-R).
Hybrids of these scenarios are also possible, for instance a sub-rate of a bonded PHY supporting 250 Gbps MAC over three bonded 100GBASE-R PHYs. These options allow increased resource flexibility for 5G and fine-tuning the offered rate depending on the usage.
3.2 X-Ethernet Technology
X-Ethernet is a Huawei proprietary technology, where X stands for extended distance, expanded granularity and extremely low latency. X-Ethernet introduces Ethernet PCS switching based on the interface offered by Flexible-Ethernet. The basic switching unit is FlexE Client. The switch device will redirect FlexE Clients (64B/66B block streams) from its inbound port to its outbound port without waiting for the arrival of the whole Ethernet frame for FCS checksum and forwarding decision with table lookup. Therefore, all the time consuming procedure, such as encapsulation/decapsulation, queuing and table lookup, can be removed. We give another name to the PCS switching that is PCS Non-Stop Switch (NSS). What’s more, the remaining procedure processing time is predictable, which results in deterministic device latency. Idle insertion or deletion according to IEEE 802.3 may be performed to rate-adapt FlexE Client to the Flex Group. Details of the technology can be found in [9].
4 Performance Evaluation
4.1 Flex-E Testbed, Experiment Description and Evaluation Results
We demonstrate and evaluate Flex-E technology, while verifying the theoretical framework proposed by OIF. The system demonstrator under test, is depicted in Fig. 2 where for the implementation of the testbed two Huawei Optix PTN 990 are used. A software patch on VRP V100R008C10, supports the necessary Flex-E functionality. PTN devices are primarily used on bearer networks that carry various services, such as mobile communication and enterprise users’ services. It transports packet services on the network and converges them to an IP/MPLS backbone network.
In principle channelisation enables multiple Flex-E clients over a shared single PHY or bounded PHY via the means of time division multiplexing in the Flex-Shim (e.g. support 150G and a 50 MAC over two bonded 100GBASE-R PHYs). In this demonstrator one 100G link between the two routers is “splitted” using the Flex-E technology. The splitting is made by creating Flex-E channels. The Flex-E clients are identified using VLAN technology and each VLAN is mapped to a specific Flex-E channel.
As presented in Fig. 3 for the demonstrator we allocate different time slots for two Flex-E channels, namely Channel 0 and Channel 2. The mapping of channels in Flex-E flows is made on a VLAN basis where for the current implementation the lower two bits of the channel ID and VLAN ID correspond to each other (e.g., VLAN 4 maps to channel 0 and VLAN 5 to channel 1 and VLAN 6 in channel 2 and so on). In Fig. 3 a sample channel configuration is presented for router 1 (R1 prompt) while a similar configuration also exists for router 2. In the example depicted, slots 5 to 9 are allocated in channel 2, while all the other slots (1–4 and 6–20) are allocated in channel 0. With this allocation channel 0 receives 15/20 = 3/4 capacity (translated to scheduling opportunities) from the 100G interface, channel 2 receives 1/4 from the link capacity, while channels 1 and 3 are blocked. Because of the specific flow mapping implementation, channel 0 will support not only VLAN 4 but also all VLAN-ids where the ending bits are 00 (like 0b000, which is VLAN 8) and so on. Different Flex-E client mapping mechanisms are under consideration and development.
To create multiple Ethernet flows with the necessary VLAN identification, traffic has been generated by using a Huawei 100GE traffic generator, which was controlled using Tesegine 2.0 V300R006C10B410 software. Using the Tesegine 2.0 traffic generator software, we were able to generate multiple concurrent flows with different configuration options of all the frames/packets fields (like src/dst MAC address, VLAN-id, src/dst IP address etc.). As a baseline experiment, we created two flows: Flow-1 (VLAN-id 4) that could be critical traffic and Flow-2 (VLAN-id 6) being the background traffic. The goal of the demonstrator was to showcase the ability of Flex-E channelisation to provide precise capacity shares between the two competing channels/flows. Channel statistics were obtained using interface counters that were updated to report per channel reports.
From all the experiments performed, perfect isolation was achieved; not only the capacity ratios were respected but also channels 1 and 3 that could block the traffic are not mapped to “open” Flex-E channels and thus had no effect on performance. Even though we only present here a representative subset of results, our conclusions apply to a wide range of experiments with the adjustment of different network parameters. In Figs. 4 and 5 we present the evaluation results of the experiment described above with two Flex-e channels serving to Ethernet flows that are identified using their VLAN-id. As we can see, the 100G link is precisely sliced exploiting time scheduling on the Flex-Calendar. Flex-E channel 0 throughput is 75 Gb/s on average while channel 2 throughput is 25 Gb/s on average (5 slots out of 25) Fig. 4. An important observation however is that although Flex-e technology is able to provide precise throughput guarantees per flow using very low level slicing, is not able to differentiate delay and jitter per channel for each Flex-E client. This phenomenon was however expected and is depicted in Fig. 5(a) and (b) where, as we can observe, both flows experience similar delay and jitter performance.
4.2 X-Ethernet Testbed, Experiment Description and Evaluation Results
A X-Ethernet testbed was used to carry demonstration activities, where for each X-Ethernet switch the solution is based on a FPGA board, six 100G CFP2 optical module slots, two 10G SFP+ optical module slots, one Ethernet interface slot. The FPGA board is mainly composed by one Virtex UltraScale chip, one ZYNQ chip, two DDR3 SDRAM chips, one DDR4 SODIMM, two Quad-SPI flashes and one Micro SD.
In the following, we provide experimentation results from the execution of three test scenarios that were carried out and showcase the ability of X-Ethernet technology to satisfy challenging switching requirements for a Flex-E network. Three X-Ethernet prototypes are connected to each other and formed a network. A controller (PC) configures each of the devices via the RS232 port on each device. A CPRI tester is used to generate CPRI option 7 traffic. A network performance tester is used to generate Ethernet traffic with 100 Gb/s maximum bit rate. CPRI is injected into the X-Ethernet node device 1 (XE1) and transport to XE2, then loop back to the CPRI tester. The Ethernet traffic pass through three XE devices one by one and loop back to the network performance tester. It should be noted that the connection between XE devices is though 100G links. Particularly, CPRI and Ethernet traffic share the same 100G link between XE No.1 and XE2. Moreover, the connection between Network performance tester and XE devices are also 100G link. Network performance tester generate an Ethernet traffic that has an effective data rate ranges from 0 to 100 Gb/s.
Experiment T1: Setup the test environment as shown in Fig. 6(a). We configured end-to-end Ethernet traffic flow between XE nodes in two scenarios: Flex-E tunnel allocated 2 and 10 slots respectively. The TestCenter generated and send packets of length 128 bytes.
Experiment T2: Setup the test environment as shown in Fig. 6(b). We configured end-to-end Ethernet traffic flows between three XE nodes. The TestCenter generate and send packets of length 128 bytes. In both experiments we modified traffic packet length to 1518 bytes in sequence, and repeated the above operation, while the metric of interest is traffic latency. The experiment results are depicted in the following Fig. 7:
For the packet streams of different packet length at different rates, X-Ethernet exhibits its ultra-low latency forwarding capability at around 0.5 use. Compared to the classic router/switch performance (30 \(\upmu \)s to 200 ms), X-Ethernet has a huge advantage in carrying latency sensitive services. Thanks to its L1.5 Non Stop Switching mechanism, all the time consuming procedures like table lookup, queuing, buffering, etc., are removed. What’s more, due to its TDM like mechanism, it provides end-to-end hard isolation pipe, which guarantees a nanosecond level jitter in the transmission.
5 Enabling Programmability on Flex-E, X-Ethernet Networks an Integration with 5G-OS
Software Defined Networking (SDN) is a technology paradigm that drives innovation on the network segment but mainly inside the data center, it has just recently applied also in the wireless access but also in the transport network.
Although there is a proposal to adopt GMPLS as the Flex-E control plane, is not yet standardized [4]. To enable programmability on the Flex-E/X-Ethernet solution a NETCONF protocol-based solution with out-of-band signaling could be also adopted. In the case of NETCONF protocol, a NETCONF server should operate on the switching device to interpret messages send from a remote SDN controller implementing the NETCONF client. NETCONF is connected oriented using TCP while messages are encoded in XML and encrypted by SSH. An SDN controller like ODL or RYU could be used to implement this functionality and be directly connected to an orchestration and management solution like the 5G Operating System (5G-OS) proposed and designed in the context of the 5G-PICTURE project [2]. A 5G-OS instance can be used to manage services and slices on top of the infrastructure provided by multiple Infrastructure Provider(s). A 5G-OS is composed of different orchestration, management and control components that are tightly integrated to control the operational and business aspects of multiple technological and administrative domains. A simplified version of the 5G-OS integrated with a NETCONF based solution for the programmability of the X-Ethernet solution is depicted in Fig. 8. In 5G-OS a controller within a single domain is referred to as a Domain Controller (DC), similarly an orchestrator within a single domain is called a Domain Orchestrator (DO). An orchestrator that spans multiple DOsand is responsible for full service instantiation is called a Multi-Domain Orchestrator (MDO). Depending on the NETCONF approach with the appropriate YANG models RESTCONF API can also be exploited. RESTCONF is a REST like protocol running over HTTP for accessing data defined in YANG using datastores defined in NETCONF.
In both GMPLS and NETCONF cases, new data models need to be devised that expose the Flex-E information and functionalities to the control plane. Although the design of YANG models is possible over RSVP, new YANG models are expected to emerge for Flex-E. As in all control plane models, the design primitives for the Flex-E control plane are security, scalability and fast convergence. To on-board on 5G-OS a possible service descriptor should include:
-
Flex-E Group provisioning, configuration and instantiation operations: Routers must advertise the type of Flex-E support that they offer, the current calendar allocation and information like link delay and node delay. Regarding capabilities exposure auto-negotiation procedures also need to be defined.
-
Flex-E calendar scheduling: The control plane must be able to provide an efficient mechanism for the optimal assignment of PHYs to a specific group, while also consider for the optimal slot allocation in the group calendar for each Flex-E client.
-
Establishment of Flex-E multi-hop paths: Existing solutions consider a pre-configured Command-Line Interface (CLI) based Flex-E group configuration and client assignment. Note that the most important functionality in order to have a functional Flex-E setup is that for each PHY the mux and demux share the same sub-calendar. Otherwise, it would be impossible to decode the slot information to a specific Flex-E client. In a multi-hop setup this information sharing can be challenging.
-
Dynamic calendar switching configurations: Control plane must support dynamic switching between calendar configurations (A or B) and allow modifying the configuration of Flex-E clients into calendar slots, based on SLAs and performance criteria.
Regarding X-Ethernet, note that it is a Huawei proprietary experimental solution and the focus in the current development phase is on the dataplane operations. No YANG models are currently available or under development. However, these are planned to be implemented the following period, exploiting NETCONF protocol and enabling integration with the 5G-OS. For an orchestration and management system like 5G-OS a possible X-Ethernet service descriptor should include: VLAN ID, Flex-E client bandwidth, Flex-E group ID, PHY ID, Flex-E calendar slot number, performance monitoring information, system-type. data mode, client signal type.
6 Conclusions and Future Work
In this work we investigated the coupling of Flex-E and X-Ethernet technologies as a means to realize interface slicing and fast switching respectively. Flex-E is a key technology that is able to split a physical interface into isolated sub-channels, decouple MAC rate from PHY rate and achieve “hard” bandwidth isolation. X-Ethernet technology introduces Ethernet PCS switching, eliminates table lookup and buffer queuing and will be investigated as a fast switching mechanism that is exploiting PCS layer relay. We presented evaluation results from implementation activities that showcase the ability of an integrated Flex-E/X-Ethernet solution to enable programmability and network slicing features for future transport networks. As Flex-E is a living standard, we are following all the recent amendments that we are willing to incorporate into our testing infrastructure. Furthermore, we are also investigating solutions where on the top of the end-to-end pipe that Flex-E is able to construct, virtual networks (like VLAN or enhanced VPN solutions based) can be constructed and operate depending on network slice requirements.
References
Chang, C., et al.: Slice orchestration for multi-service disaggregated ultra-dense rans. IEEE Commun. Mag. 56(8), 70–77 (2018)
Dräxler, S., et al.: 5G OS: control and orchestration of services on multi-domain heterogeneous 5G infrastructures. In: 2018 European Conference on Networks and Communications (EuCNC), pp. 1–9, June 2018
Hofmeister, T., Vusirikala, V., Koley, B.: How can flexibility on the line side best be exploited on the client side? In: Optical Fiber Communication Conference, p. W4G–4. Optical Society of America (2016)
Hussain, I., Valiveti, R., Wang, Q., Andersson, L., Chen, M., Zheng, H.: GMPLS routing and signaling framework for flexible ethernet (FlexE). draft-izh-camp-fiexe-fwk-05. IETF-Draft (2018)
Jinno, M., Takara, H., Yonenaga, K., Hirano, A.: Virtualization in optical networks from network level to hardware level. J. Opt. Commun. Netw. 5(10), A46–A56 (2013)
Kaloxylos, A., Mannweiler, C., et al.: Network Slicing. 5G System Design: Architectural and Functional Considerations and Long Term Research, pp. 181–205. Wiley, Hoboken (2018)
Katsalis, K., Gatzikis, L., Samdanis, K.: Towards slicing for transport networks: the case of flex-ethernet in 5G. In: 2018 IEEE Conference on Standards for Communications and Networking (CSCN), pp. 1–7, October 2018
Katsalis, K., Nikaein, N., Schiller, E., Ksentini, A., Braun, T.: Network slices toward 5G communications: slicing the LTE network. IEEE Commun. Mag. 55(8), 146–154 (2017)
Li, R., Wang, R., et al.: X-Ethemet: enabling integrated fronthaul/backhaul architecture in 5G networks. In: 2017 IEEE Conference on Standards for Communications and Networking (CSCN), pp. 121–125. IEEE (2017)
Nejabati, R., Escalona, E., Peng, S., Simeonidou, D.: Optical network virtualization. In: 15th International Conference on Optical Network Design and Modeling-ONDM 2011, pp. 1–5. IEEE (2011)
Tipmongkolsilp, O., Zaghloul, S., Jukan, A.: The evolution of cellular backhaul technologies: current issues and future trends. IEEE Commun. Surv. Tutor. 13(1), 97–113 (2011)
Vilalta, R., et al.: Network slicing using dynamic flex ethernet over transport networks. In: ECOC, pp. 1–3, September 2017
Acknowledgements
This work has received funding from the EU H2020 5G-PICTURE project under grant agreement no 762057.
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 IFIP International Federation for Information Processing
About this paper
Cite this paper
Katsalis, K., Li, R. (2020). Programmable Flex-E and X-Ethernet Networks for Traffic Isolation in Multi-tenant Environments. In: Tzanakaki, A., et al. Optical Network Design and Modeling. ONDM 2019. Lecture Notes in Computer Science(), vol 11616. Springer, Cham. https://doi.org/10.1007/978-3-030-38085-4_52
Download citation
DOI: https://doi.org/10.1007/978-3-030-38085-4_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-38084-7
Online ISBN: 978-3-030-38085-4
eBook Packages: Computer ScienceComputer Science (R0)