Skip to main content
SpringerLink
Log in

Availability-aware and energy-aware dynamic SFC placement using reinforcement learning

  • Published:
The Journal of Supercomputing Aims and scope Submit manuscript

Abstract

Software-defined networking and network functions virtualisation are making networks programmable and consequently much more flexible and agile. To meet service-level agreements, achieve greater utilisation of legacy networks, faster service deployment, and reduce expenditure, telecommunications operators are deploying increasingly complex service function chains (SFCs). Notwithstanding the benefits of SFCs, increasing heterogeneity and dynamism from the cloud to the edge introduces significant SFC placement challenges, not least adding or removing network functions while maintaining availability, quality of service, and minimising cost. In this paper, an availability- and energy-aware solution based on reinforcement learning (RL) is proposed for dynamic SFC placement. Two policy-aware RL algorithms, Advantage Actor-Critic (A2C) and Proximal Policy Optimisation (PPO), are compared using simulations of a ground truth network topology based on the Rede Nacional de Ensino e Pesquisa Network, Brazil’s National Teaching and Research Network backbone. The simulation results show that PPO generally outperformed A2C and a greedy approach in terms of both acceptance rate and energy consumption. The biggest difference in the PPO when compared to the other algorithms relates to the SFC availability requirement of 99.965%; the PPO algorithm median acceptance rate is 67.34% better than the A2C algorithm. A2C outperforms PPO only in the scenario where network servers had a greater number of computing resources. In this case, the A2C is 1% better than the PPO.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Notes

  1. https://stable-baselines.readthedocs.io/en/master/index.html.

  2. https://github.com/GutoL/SFC_RL.

  3. https://stable-baselines.readthedocs.io/en/master/modules/a2c.html.

  4. https://stable-baselines.readthedocs.io/en/master/modules/ppo2.html.

References

  1. Ali HMM, Lawey AQ, El-Gorashi TE, Elmirghani JM (2015) Energy efficient disaggregated servers for future data centers. In: 2015 20th European Conference on Networks and Optical Communications-(NOC), IEEE pp. 1–6.

  2. Andrade E, Nogueira B, Matos R, Callou G, Maciel P (2017) Availability modeling and analysis of a disaster-recovery-as-a-service solution. Computing 99(10):929–954

    Article  MathSciNet  Google Scholar 

  3. Araujo J, Maciel P, Andrade E, Callou G, Alves V, Cunha P (2018) Decision making in cloud environments: an approach based on multiple-criteria decision analysis and stochastic models. J Cloud Comput 7(1):7

    Article  Google Scholar 

  4. Araujo J, Maciel P, Torquato M, Callou G, Andrade E (2014) Availability evaluation of digital library cloud services. In: Dependable Systems and Networks (DSN), 2014 44th Annual IEEE/IFIP International Conference on, IEEE pp. 666–671

  5. Arulkumaran K, Deisenroth MP, Brundage M, Bharath AA (2017) A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866

  6. Bhamare D, Jain R, Samaka M, Erbad A (2016) A survey on service function chaining. J Netw Comput Appl 75:138–155

    Article  Google Scholar 

  7. Cai J, Huang Z, Luo J, Liu Y, Zhao H, Liao L (2020) Composing and deploying parallelized service function chains. J Netw Comput Appl 163:102637

    Article  Google Scholar 

  8. Chai H, Zhang J, Wang Z, Shi J, Huang T (2019) A parallel placement approach for service function chain using deep reinforcement learning. In: 2019 IEEE 5th International Conference on Computer and Communications (ICCC), IEEE pp. 2123–2128

  9. Costa I, Araujo J, Dantas J, Campos E, Silva FA, Maciel P (2016) Availability evaluation and sensitivity analysis of a mobile backend-as-a-service platform. Qual Reliab Eng Int 32(7):2191–2205

    Article  Google Scholar 

  10. Dâmaso A, Rosa N, Maciel P (2014) Reliability of wireless sensor networks. Sensors 14(9):15760–15785

    Article  Google Scholar 

  11. Dâmaso A, Rosa N, Maciel P (2017) Integrated evaluation of reliability and power consumption of wireless sensor networks. Sensors 17(11):2547

    Article  Google Scholar 

  12. Fan J, Guan C, Zhao Y, Qiao C (2017) Availability-aware mapping of service function chains. In: IEEE INFOCOM 2017-IEEE Conference on Computer Communications, IEEE pp. 1–9

  13. Farshin A, Sharifian S (2019) A modified knowledge-based ant colony algorithm for virtual machine placement and simultaneous routing of nfv in distributed cloud architecture. J Supercomput 75(8):5520–5550

    Article  Google Scholar 

  14. Gens F (2013) The 3rd platform: enabling digital transformation, vol 209. IDC, USA

    Google Scholar 

  15. Gissler B, Shrivastava P (2015) A system for design decisions based on reliability block diagrams. In: 2015 Annual Reliability and Maintainability Symposium (RAMS), pp. 1–6. IEEE

  16. Gomez-Rodriguez MA, Sosa-Sosa VJ, Carretero J, Gonzalez JL (2020) Cloudbench: an integrated evaluation of vm placement algorithms in clouds. J Supercomput :1–34

  17. Goodfellow I, Bengio Y, Courville A, Bengio Y (2016) Deep learning, vol 1. MIT press, Cambridge

    MATH  Google Scholar 

  18. ISG, NFV. Network functions virtualisation (nfv)-network operator perspectives on industry progress. ETSI GS NFV-SEC 001 V1.1.1. 2013. Available at https://www.etsi.org/deliver/etsi_gs/nfv-sec/001_099/001/01.01.01_60/gs_nfv-sec001v010101p.pdf. Accessed Apr 2021

  19. Guo S, Dai Y, Xu S, Qiu X, Qi F (2019) Trusted cloud-edge network resource management: Drl-driven service function chain orchestration for iot. IEEE Internet Things J 7(7):6010–6022

    Article  Google Scholar 

  20. Hasselt HV (2010) Double q-learning. Adv Neural Inf Process Syst 23:2613–2621

    Google Scholar 

  21. He W, Chen X, Qiu X, Guo S, Yu P (2019) Asco: an availability-aware service chain orchestration. In: 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), IEEE pp. 590–593

  22. Høyland A, Rausand M (2009) System reliability theory: models and statistical methods, vol 420. Wiley, New Jersey

    MATH  Google Scholar 

  23. Jain R (1991) The art of computer systems performance analysis. Wiley, New Jersey

    MATH  Google Scholar 

  24. Jim M (2015) Nfv applications - key considerations for profitability. https://web.dialogic.com/making-nfv-profitable

  25. Kaur K, Mangat V, Kumar K (2020) A comprehensive survey of service function chain provisioning approaches in sdn and nfv architecture. Comput Sci Rev 38:100298

    Article  Google Scholar 

  26. Khezri HR, Moghadam PA, Farshbafan MK, Shah-Mansouri V, Kebriaei H, Niyato D (2019) Deep reinforcement learning for dynamic reliability aware nfv-based service provisioning. In: 2019 IEEE Global Communications Conference (GLOBECOM), IEEE pp. 1–6

  27. Kouah R, Alleg A, Laraba A, Ahmed T (2018) Energy-aware placement for iot-service function chain. In: 2018 IEEE 23rd International Workshop on Computer Aided Modeling and Design of Communication Links and Networks (CAMAD), IEEE pp. 1–7

  28. Kumar A, Pant S, Ram M (2017) System reliability optimization using gray wolf optimizer algorithm. Qual Reliab Eng Int 33(7):1327–1335

    Article  Google Scholar 

  29. Leong YC, Radulescu A, Daniel R, DeWoskin V, Niv Y (2017) Dynamic interaction between reinforcement learning and attention in multidimensional environments. Neuron 93(2):451–463

    Article  Google Scholar 

  30. Lesort T, Díaz-Rodríguez N, Goudou JF, Filliat D (2018) State representation learning for control: an overview. Neural Netw 108:379–392

    Article  Google Scholar 

  31. Li G, Zhou H, Feng B, Li G (2018) Context-aware service function chaining and its cost-effective orchestration in multi-domain networks. IEEE Access 6:34976–34991

    Article  Google Scholar 

  32. Li G, Zhou H, Feng B, Zhang Y, Yu S (2019) Efficient provision of service function chains in overlay networks using reinforcement learning. IEEE Trans Cloud Comput

  33. Lima PA, Neto ASB, Maciel P (2020) Data centers’ services restoration based on the decision-making of distributed agents. Telecommun Syst :1–12

  34. Luo Z, Wu C, Li Z, Zhou W (2019) Scaling geo-distributed network function chains: a prediction and learning framework. IEEE J Sel Areas Commun 37(8):1838–1850

    Article  Google Scholar 

  35. Lynn T, Gourinovitch A, Svorobeh S, Endo PT (2018) Software defined networking and network functions virtualization - market briefing. https://recap-project.eu/media/market-briefings/

  36. Mann ZÁ (2015) Allocation of virtual machines in cloud data centers-a survey of problem models and optimization algorithms. ACM Comput Surv (CSUR) 48(1):1–34

    Article  Google Scholar 

  37. Matos R, Dantas J, Araujo J, Trivedi KS, Maciel P (2017) Redundant eucalyptus private clouds: availability modeling and sensitivity analysis. J Grid Comput 15(1):1–22

    Article  Google Scholar 

  38. Mirjalily G, Zhiquan L (2018) Optimal network function virtualization and service function chaining: a survey. Chin J Electron 27(4):704–717

    Article  Google Scholar 

  39. Mnih V, Badia AP, Mirza M, Graves A, Lillicrap T, Harley T, Silver D, Kavukcuoglu K (2016) Asynchronous methods for deep reinforcement learning. In: International Conference on Machine Learning, pp. 1928–1937

  40. Moualla G, Turletti T, Saucez D (2018) An availability-aware sfc placement algorithm for fat-tree data centers. In: 2018 IEEE 7th International Conference on Cloud Networking (CloudNet), IEEE pp. 1–4

  41. Mousavi SS, Schukat M, Howley E (2017) Traffic light control using deep policy-gradient and value-function-based reinforcement learning. IET Intell Transp Syst 11(7):417–423

    Article  Google Scholar 

  42. Mundie C, de Vries P, Haynes P, Corwine M (2002) Trustworthy computing. Tech. rep, Technical report, p 10

  43. Osband I, Blundell C, Pritzel A, Van Roy B (2016) Deep exploration via bootstrapped dqn. arXiv preprint http://arxiv.org/abs/1602.04621

  44. Palhares A, Santos M, Endo P, Vitalino J, Rodrigues M, Gonçalves G, Sadok D, Sefidcon A, Wuhib F (2014) Joint allocation of nodes and links with load balancing in network virtualization. In: 2014 IEEE 28th International Conference on Advanced Information Networking and Applications, pp. 148–155. IEEE

  45. Pan J, Wang X, Cheng Y, Yu Q (2018) Multisource transfer double dqn based on actor learning. IEEE Trans Neural Netw Learn Syst 29(6):2227–2238

    Article  Google Scholar 

  46. Peng B, Li X, Gao J, Liu J, Chen YN, Wong KF (2018) Adversarial advantage actor-critic model for task-completion dialogue policy learning. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6149–6153. IEEE

  47. Qiu Z, Zhang J, Ning P, Wen X (2017) Reliability modeling and analysis of sic mosfet power modules. In: IECON 2017-43rd Annual Conference of the IEEE Industrial Electronics Society, IEEE pp. 1459–1463

  48. Ravichandiran S (2018) Hands-on reinforcement learning with Python: master reinforcement and deep reinforcement learning using OpenAI gym and tensorFlow. Packt Publishing Ltd, England

    Google Scholar 

  49. Santos GL, Endo PT, da Silva Lisboa MFF, da Silva LGF, Sadok D, Kelner J, Lynn T et al (2018) Analyzing the availability and performance of an e-health system integrated with edge, fog and cloud infrastructures. J Cloud Comput 7(1):16

    Article  Google Scholar 

  50. Sayadnavard MH, Haghighat AT, Rahmani AM (2019) A reliable energy-aware approach for dynamic virtual machine consolidation in cloud data centers. J Supercomput 75(4):2126–2147

    Article  Google Scholar 

  51. Schulman J, Wolski F, Dhariwal P, Radford A, Klimov O (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

  52. de Sousa NFS, Perez DAL, Rosa RV, Santos MA, Rothenberg CE (2019) Network service orchestration: a survey. Comput Commun 142:69–94

    Article  Google Scholar 

  53. Sun G, Li Y, Yu H, Vasilakos AV, Du X, Guizani M (2019) Energy-efficient and traffic-aware service function chaining orchestration in multi-domain networks. Future -Gener Comput Syst 91:347–360

    Article  Google Scholar 

  54. Sun P, Lan J, Li J, Guo Z, Hu Y (2020) Combining deep reinforcement learning with graph neural networks for optimal vnf placement. IEEE Commun Lett

  55. Sutton RS, Barto AG (2018) Reinforcement learning: an introduction. MIT press, Cambridge

    MATH  Google Scholar 

  56. Tavakoli-Someh S, Rezvani MH (2019) Multi-objective virtual network function placement using nsga-ii meta-heuristic approach. J Supercomput 75(10):6451–6487

    Article  Google Scholar 

  57. Torquato M, Torquato L, Maciel P, Vieira M (2019) Iaas cloud availability planning using models and genetic algorithms. In: 2019 9th Latin-American Symposium on Dependable Computing (LADC), IEEE pp. 1–10

  58. Troia S, Alvizu R, Maier G (2019) Reinforcement learning for service function chain reconfiguration in nfv-sdn metro-core optical networks. IEEE Access 7:167944–167957

    Article  Google Scholar 

  59. Xiao Y, Zhang Q, Liu F, Wang J, Zhao M, Zhang Z, Zhang J (2019) Nfvdeep: Adaptive online service function chain deployment with deep reinforcement learning. In: Proceedings of the International Symposium on Quality of Service, pp. 1–10

  60. Xu Z, Zhang X, Yu S, Zhang J (2018) Energy-efficient virtual network function placement in telecom networks. In: 2018 IEEE International Conference on Communications (ICC), IEEE pp. 1–7

  61. Zhang J, Wang Z, Ma N, Huang T, Liu Y (2018) Enabling efficient service function chaining by integrating nfv and sdn: architecture, challenges and opportunities. IEEE Netw 32(6):152–159

    Article  Google Scholar 

  62. Zhang X, Xu Z, Fan L, Yu S, Qu Y (2019) Near-optimal energy-efficient algorithm for virtual network function placement. IEEE Trans Cloud Comput

  63. Zheng Y, Li X, Xu L (2020) Balance control for the first-order inverted pendulum based on the advantage actor-critic algorithm. Int J Control Autom Syst 18(12):3093–3100

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Centro de Informática, Universidade Federal de Pernambuco (UFPE), Recife, Brazil

    Guto Leoni Santos & Judith Kelner

  2. Business School, Dublin City University (DCU), Dublin, Ireland

    Theo Lynn

  3. Universidade de Pernambuco (UPE), Recife, Brazil

    Patricia Takako Endo

Authors
  1. Guto Leoni Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theo Lynn
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Judith Kelner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patricia Takako Endo
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Guto Leoni Santos.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Santos, G.L., Lynn, T., Kelner, J. et al. Availability-aware and energy-aware dynamic SFC placement using reinforcement learning. J Supercomput 77, 12711–12740 (2021). https://doi.org/10.1007/s11227-021-03784-7

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11227-021-03784-7

Keywords

Search

Navigation