Skip to main content
SpringerLink
Log in

Actor-critic architecture based probabilistic meta-reinforcement learning for load balancing of controllers in software defined networks

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

The exponential growth in the complexity of network architecture to accommodate the enormous amount of data has motivated the emergence of software-defined networks (SDN). However, the scaling of network size and related services seriously affects the controller resource utilization in SDN. An optimal load balancing strategy is required to accommodate the dynamic network traffic and load distribution among different controllers. Although reinforcement learning has been used for load balancing in SDN by modelling it as the linear optimization problem, it could not perform well for the real-time load in SDN due to the uncertain and dynamic correlation between the resources. This paper presents a deep meta-reinforcement learning (meta-RL) technique to derive an intelligent optimization framework for load balancing in SDN using actor-critic architecture. Meta-RL is the modified version of conventional reinforcement learning which utilizes a smaller amount of training data comparatively to enable the agent to learn the policies. The proposed technique separates the task's inference and control to deal with the uncertainty while adapting to the new tasks in the sparse reward problem. It utilizes the probabilistic interpretation of task variables to solve the new task through sparse experience space. The simulation analysis supports the theoretical analysis of SDN's optimal load balancing technique. A real-time database has been utilized in this work to evaluate the efficiency and effectiveness of the proposed work. It has been found that the proposed technique showed an average of 10.4%, 19.1%, and 6.04% improvement in load balancing rate as compared to DC-LB, MMO-LB, and CRL-LB techniques. It also showed an improvement of 15.7%, 12.18%, and 5.18% in processing delay compared to the same methods, respectively. The standard deviation is also improved by 23.72% compared to the scenario of no-load balancing.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Belgaum, M.R., Musa, S., Alam, M.M., Su’ud, M.M.: A systematic review of load balancing techniques in software-defined networking. IEEE Access 8, 98612–98636 (2020)

    Article  Google Scholar 

  • Benzekki, K., El Fergougui, A., Elalaoui, A.E.: Software-defined networking (SDN): a survey. Secur. Commun. Netw. 9(18), 5803–5833 (2016)

    Article  Google Scholar 

  • Cimorelli, F., Priscoli, F.D., Pietrabissa, A., Celsi, L.R., Suraci, V., Zuccaro, L. (2016): A distributed load balancing algorithm for the control plane in software defined networking. In: Proc. 24th Medit. Conf. Control Autom. (MED), pp. 1033–1040 (2016)

  • Hu, T., Zhang, J., Kong, W., Yang, S., Cao, L.: Switch competing migration algorithm based on process optimization in SDN. J. Commun. 8, 213–222 (2017)

    Google Scholar 

  • iPref: https://iperf.fr/

  • Kang, B., Choo, H.: An SDN-enhanced load-balancing technique in the cloud system. J. Supercomput. 3, 1–24 (2016)

    Google Scholar 

  • Lemeshko, O., Yeremenko, O.: Enhanced method of fast re-routing with load balancing in software-defined networks. J. Electr. Eng. 68(6), 444–454 (2017)

    Google Scholar 

  • Li, Z., Zhou, Z., Gao, J., Qin, Y.: SDN controller load balancing based on reinforcement learning. In: Proceedings of the 2018 IEEE 9th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, 23–25 November 2018

  • Li, Z., Zhou, X., Gao, J., Qin, Y.: SDN controller load balancing based on reinforcement learning (2021). arXiv:2103.06579v1

  • McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., Turner, J.: Openflow: enabling innovation in campus networks. SIGCOMM Comput. Commun. Rev. 38(2), 69–74 (2008)

    Article  Google Scholar 

  • Mininet: http://mininet.org/

  • Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K.: Asynchronous methods for deep reinforcement learning. In: Proceedings of the 33rd International Conference on Machine Learning, vol. 48, pp. 1928–1937 (2016)

  • Mu, T., Al-Fuqaha, A., Shuaib, K., Sallabi, F.M., Qadir, J.: SDN flow entry management using reinforcement learning. ACM Trans. Auton. Adapt. Syst. 13, 1–23 (2018)

    Article  Google Scholar 

  • Nagabandi, A., Clavera, I., Liu, S., Fearing, R.S., Abbeel, P., Levine, S., Finn, C.: Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In: International Conference on Learning Representations (ICLR) (2019)

  • Open Networking Foundation: https://www.opennetworking.org/

  • Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Desmaison, A.: Pytorch: an imperative style, high-performance deep learning library. Adv. Neural Inf. Process. Syst. 32, 8026–8037 (2019)

    Google Scholar 

  • Rakelly, K., Zhou, A., Quillen, D., Finn, C., Levine, S.: Efficient off-policy meta-reinforcement learning via probabilistic context variables (2019). arXiv:1903.08254v1

  • Rusu, A.A., Rao, D., Sygnowski, J., Vinyals, O., Pascanu, R., Osindero, S., Hadsell, R.: Meta-learning with latent embedding optimization. In: International Conference on Learning Representations (2019)

  • Ryu SDN Framework: http://osrg.github.io/ryu/

  • Sahoo, K.S., Puthal, D., Tiwary, M., Usman, M., Sahoo, B., Wen, Z., Sahoo, B.P.S., Ranjan, R.: ESMLB: efficient switch migration-based load balancing for multi-controller SDN in IoT. IEEE Internet Things J. 7, 5852–5860 (2019)

    Article  Google Scholar 

  • Schweighofera, N., Doya, K.: Meta-learning in reinforcement learning. Neural Netw. 16(1), 5–9 (2003)

    Article  Google Scholar 

  • Stadie, B.C., Yang, G., Houthooft, R., Chen, X., Duan, Y., Wu, Y., Abbeel, P., Sutskever, I.: Some considerations on learning to explore via meta-reinforcement learning (2018). arXiv preprint arXiv:1803.01118

  • Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction, vol. 1, 2nd edn. MIT Press, Cambridge (2017)

    MATH  Google Scholar 

  • Symondson, S., Hofmann, K., Deisenroth, M.: Meta reinforcement learning with latent variable gaussian processes. In: Conference on Uncertainty in Artificial Intelligence (UAI) (2018)

  • Tosounidis, V., Pavlidis, G., Sakellaiou, I.: Deep Q-learning for load balancing traffic in SDN networks. In: Proceedings of the SETN: Hellenic Conference on Artificial Intelligence, Athens, Greece, 2–4 September 2020

  • Wang, Y., Zhang, Y., Chen, J.: SDNPS: a load-balanced topic-based Publish/Subscribe system in software-defined networking. Appl. Sci. 6(4), 91 (2016)

    Article  Google Scholar 

  • Wu, Y., Zhou, S., Wei, Y., Leng, S.: Deep reinforcement learning for controller placement in software defined network. In: Proceedings of the IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), Toronto, ON, Canada, 6–9 July 2020

  • Ye, X., Cheng, G., Luo, X.: Maximizing SDN control resource utilization via switch migration. Comput. Netw. 126(24), 69–80 (2017)

    Article  Google Scholar 

  • Zhang, Y., Cui, L., Wang, W., Zhang, Y.: A survey on software defined networking with multiple controllers. J. Netw. Comput. Appl. 103, 101–118 (2018)

    Article  Google Scholar 

  • Zhang, S., Lan, J., Sun, P., Jiang, Y.: Online load balancing for distributed control plane in software-defined data center network. IEEE Access 6, 18184–18191 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Govt. Women’s Polytechnic College Indore, Indore, M.P., India

    Ashish Sharma

  2. IET DAVV Indore, Indore, M.P., India

    Sanjiv Tokekar

  3. SGSITS Indore, Indore, M.P., India

    Sunita Varma

Authors
  1. Ashish Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanjiv Tokekar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sunita Varma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ashish Sharma.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sharma, A., Tokekar, S. & Varma, S. Actor-critic architecture based probabilistic meta-reinforcement learning for load balancing of controllers in software defined networks. Autom Softw Eng 29, 59 (2022). https://doi.org/10.1007/s10515-022-00362-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-022-00362-w

Keywords

Search

Navigation