Priority-awareness VNF migration method based on deep reinforcement learning

Abstract

By decoupling the software function on hardware devices, Network Function Virtualization (NFV) provides a new service architecture named Service Function Chain (SFC), which combines multiple Virtual Network Functions (VNFs) in a specific order. In order to improve the reliability and Quality of Service (QoS) of network services, VNF migration provides an effective solution for this requirement. However, traditional migration methods lack an appropriate VNF selection mechanism and consider a single network state. Moreover, how to determine the accurate migration location dynamically with machine learning is challenging. This paper proposed a novel PAVM algorithm for reliable and optimal VNF migration, which distinguishes two kinds of VNF migration schemes for network congestion and node failure states respectively, and a node and VNF priority-awareness mechanism is designed, which can select appropriate VNF on suitable node to migrate. Moreover, PAVM utilizes deep reinforcement learning algorithm to choose target node location of VNF migration, which jointly utilizes delay, load balancing of network as feedback factors to optimize the QoS. The experimental results indicate that compared with other three benchmark algorithms, PAVM can effectively reduce the transmission delay and improve node and link load balancing after the VNF migration.

Introduction

In the traditional network, the realization of network function relies on proprietary hardware. The network functions on different hardware are combined into certain network services, however, with the rapid increase of the number and diversity of network service requests, the traditional deployment of dedicated hardware by operators cannot meet the current complex and changeable network requirements. In order to solve this problem, Network Function Virtualization (NFV) technology uses general hardware such as x86 and virtualization technology to carry many functions of software processing [1]. NFV can decouple the hardware and software and abstract the functions, so that the functions of network devices no longer depend on the dedicated hardware, and the resources can be fully and flexibly shared to realize the rapid development and deployment of new services. NFV can also carry out automatic deployment, elastic scaling, fault isolation and self-healing based on the actual business needs [2]. In NFV, several Virtual Network Functions (VNF) are deployed on server nodes in a certain order to form a network function sequence that can handle specific network services, namely Service Function Chain (SFC), which uses Software Defined Network (SDN) function to create a service chain (such as firewall, network address translation, intrusion protection) connecting network services and connect it to the virtual chain [3]. Network operators can use this function to set the suite or directory of connection services, so that many services with different characteristics can use a single network connection [4].

In the 5 G network scenario, the demand for network services has increased significantly, which leads to network congestion and reduces the Quality of Service (QoS). On the other hand, the instability of the network environment is also easily to cause network failure, which results in a large number of SFCs unable to provide services normally. To this end, the emergence of VNF migration strategy solves these problems [5]. VNF migration, that is, in the network with NFV characteristics, VNF will be removed from the traditional physical machine and redeployed on better nodes, which can solve the problem of load imbalance in NFV and SDN deployment. However, there are some shortcomings that need to be addressed. Firstly, most of the methods only consider a single network situation, and do not analyze the VNF migration schemes under different network failure situations. Secondly, the current researches lack a suitable priority mechanism, which leads to the irregular migration order of VNF, and the important SFC is difficult to be optimized or restored. Last but not least, traditional VNF migration mechanism can only deal with simple network scenarios, which is difficult to adapt to the network environment with complex network state and migration strategy.

In order to solve the above shortcomings in VNF migration problem, some literatures have carried out some research to a certain extent. Previously, some SFC deployment methods can achieve reliability results like VNF migration [6], [7], [8], [9], [10], [11], [12], [13]. However, the change of network state is not considered in SFC deployment method, which may lead to SFC failure and unable to recover. So that several VNF migration methods optimize the related QoS indexes to a certain extent to solve this problem [14], [15], [16], [17]. However, they ignore the network failure, and the static method is difficult to adapt to the large-scale dynamic VNF migration needs, and due to the dependency between VNFs in SFC, the current VNF decision will affect the next VNF. For that reason, more advanced and powerful reinforcement learning models have significantly achieved performance gains in SFC deployment and VNF migration problem [18,19], which can solve sequential decision problem well. However, reinforcement learning, a classical algorithm of reinforcement learning, needs to maintain a quite large Q-table because of large state and action set, so that the computing power of the algorithm will be affected with wasting of CPU and memory. On the other hand, the search processing of reinforcement learning is very time-consuming. To this end, deep learning is proposed to simulate the Q value, and the deep reinforcement learning is proposed to apply on VNF migration problem [20,21]. Specifically, the deep reinforcement learning can choose a more suitable migration location for VNF with learning process, and it can make accurate judgements about unknown situations in the sample.

In order to effectively distinguish the importance of network node and SFC, comprehensively consider the network state, and solve the problem of dynamic VNF migration in complex networks, this paper proposes node and VNF priority-awareness strategies firstly, then designs VNF migration schemes for network congestion and failure respectively, and designs a VNF migration algorithm based on deep reinforcement learning for network congestion and node failure states finally. The main contributions of this paper are as follows.

• Firstly, in order to effectively distinguish the importance of network node and SFC, we propose node and VNF priority-awareness strategies respectively. In view of the network failure, we measure the VNF priority according to the SFC priority, which is calculated by the CPU and bandwidth resource requirements, and the proportion of node and link resources affected by node failure. In view of the network congestion, we first calculate the node priority according to the node and link resource occupation, and then calculate the VNF priority of the highest priority node. It is the first time that determining the node and VNF priority in our work.

• Secondly, in order to comprehensively consider the network state, we design two VNF migration schemes for network congestion and failure respectively. The former scheme selects the appropriate VNF to migrate to the appropriate node according to the node priority and VNF priority, and the latter scheme migrates VNF to the appropriate node in order according to the VNF priority.

• Lastly, in order to solve the problem of dynamic VNF migration in complex networks, we apply deep reinforcement learning algorithm on the process of VNF migration, then we propose PAVM model to migrate VNFs with QoS requirement. Specifically, we design the feedback function with three factors including transmission delay, node balancing, and link balancing. Then, we evaluate the proposed PAVM model on the Internet2 network topology, and the extensive experimental results demonstrate that PAVM can effectively migrate VNFs to appropriate nodes and achieve QoS performance compared with other three models including two widely used VNF migration methods and random priority scheme of PAVM.

The rest of this paper is organized as follows Section 2. surveys the related works. In Section 3, the priority awareness framework used for determining migration order is described and then the proposed PAVM algorithm based on deep reinforcement learning to solve VNF migration is discussed in details. In Section 4, the experimental designs and simulation results are shown. Finally, the conclusion and discussion of future work are given in Section 5.