Skip to main content
SpringerLink
Log in

Improved Exploration Strategy for Q-Learning Based Multipath Routing in SDN Networks

  • Published:
Journal of Network and Systems Management Aims and scope Submit manuscript

Abstract

Software-Defined Networking (SDN) is characterized by a high level of programmability and offers a rich set of capabilities for network management operations. Network intelligence is centralized in the controller, which is responsible for updating the routing policies according to the applications’ requirements. To further enhance such capabilities, the controller has to be endowed with intelligence by integrating Artificial Intelligence (AI) tools in order to provide the controller the ability to autonomously reconfigure the network in a timely way. In this paper, we address the deployment of a Q-learning algorithm for the routing optimization problem in terms of latency minimization. Using a direct modeling approach of the multi-path flow-routing problem, we delve deeper into the impact of the exploration-exploitation strategies on the algorithm’s performance. Furthermore, we propose a couple of improvements to the Q-Learning algorithm to enhance its performance within the considered environment. On the one hand, we integrate a congestion-avoidance mechanism in the exploration phase, which leads to effective improvements in the algorithm’s performance with regard to average latency, convergence time, and computation time. On the other hand, we propose to implement a novel strategy based on the Max-Boltzman Exploration method (MBE), which is a combination of the traditional \(\varepsilon\)- greedy and softmax strategies. The results show that, for an appropriate tuning of the hyperparameters, the MBE strategy combined with the congestion-avoidance mechanism performs better than the \(\varepsilon\)-greedy, \(\varepsilon\)-decay, and Softmax strategies in terms of average latency, convergence time, and computation time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Amin, R., Rojas, E., Aqdus, A., Ramzan, S., Casillas-Perez, D., Arco, J.M.: A survey on machine learning techniques for routing optimization in sdn. IEEE Access (2021)

  2. Kamboj, P., Pal, S., Bera, S., Misra, S.: Qos-aware multipath routing in software-defined networks. IEEE Trans. Network Sci. Eng. 10(2), 723–732 (2022)

    Article  MathSciNet  Google Scholar 

  3. Fu, Q., Sun, E., Meng, K., Li, M., Zhang, Y.: Deep q-learning for routing schemes in sdn-based data center networks. IEEE Access 8, 103491–103499 (2020)

    Article  Google Scholar 

  4. Rischke, J., Sossalla, P., Salah, H., Fitzek, F.H., Reisslein, M.: Qr-sdn: towards reinforcement learning states, actions, and rewards for direct flow routing in software-defined networks. IEEE Access 8, 174773–174791 (2020)

    Article  Google Scholar 

  5. Hassen, H., Meherzi, S., Jemaa, Z.B.: \(\varepsilon\)-qlmr: \(\varepsilon\)-greedy based q-learning algorithm for multipath routing in sdn networks. In: 2023 International Wireless Communications and Mobile Computing (IWCMC), pp. 234–239 (2023). IEEE

  6. Explorations in efficient reinforcement learning. PhD thesis, University of Amsterdam (1999)

  7. Masoudi, R., Ghaffari, A.: Software defined networks: a survey. J. Network Comput. Appl.67, 1–25 (2016)

    Article  Google Scholar 

  8. Ahmad, S., Mir, A.H.: Scalability, consistency, reliability and security in sdn controllers: a survey of diverse sdn controllers. J. Network Syst. Manage. 29(1), 1–59 (2021)

    Article  Google Scholar 

  9. Hussain, M., Shah, N., Amin, R., Alshamrani, S.S., Alotaibi, A., Raza, S.M.: Software-defined networking: categories, analysis, and future directions. Sensors 22(15), 5551 (2022)

    Article  Google Scholar 

  10. Latif, Z., Sharif, K., Li, F., Karim, M.M., Biswas, S., Wang, Y.: A comprehensive survey of interface protocols for software defined networks. J. Network Comput. Appl. 156, 102563 (2020)

    Article  Google Scholar 

  11. Braun, W., Menth, M.: Software-defined networking using openflow: protocols, applications and architectural design choices. Future Internet 6(2), 302–336 (2014)

    Article  Google Scholar 

  12. Gosavi, A.: Reinforcement learning: a tutorial survey and recent advances. INFORMS J. Comput. 21(2), 178–192 (2009)

    Article  MathSciNet  Google Scholar 

  13. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: A brief survey of deep reinforcement learning. arXiv preprint arXiv:1708.05866 (2017)

  14. Mammeri, Z.: Reinforcement learning based routing in networks: review and classification of approaches. Ieee Access 7, 55916–55950 (2019)

    Article  Google Scholar 

  15. Zhao, Y., Li, Y., Zhang, X., Geng, G., Zhang, W., Sun, Y.: A survey of networking applications applying the software defined networking concept based on machine learning. IEEE Access 7, 95397–95417 (2019)

    Article  Google Scholar 

  16. Ke, C.-H., Tu, Y.-H., Ma, Y.-W.: A reinforcement learning approach for widest path routing in software-defined networks. ICT Exp. 9(5), 882–889 (2022)

    Article  Google Scholar 

  17. Le, D.-H., Tran, H.-A., Souihi, S.: A reinforcement learning-based solution for intra-domain egress selection. In: 2021 IEEE 22nd International Conference on High Performance Switching and Routing (HPSR), pp. 1–6 (2021). IEEE

  18. Casas-Velasco, D.M., Rendon, O.M.C., Fonseca, N.L.: Intelligent routing based on reinforcement learning for software-defined networking. IEEE Trans. Network Service Manage. 18(1), 870–881 (2020)

    Article  Google Scholar 

  19. Casas-Velasco, D.M., Rendon, O.M.C., Fonseca, N.L.: Drsir: a deep reinforcement learning approach for routing in software-defined networking. IEEE Trans. Network Service Manage. 19, 4807–4820 (2021)

    Article  Google Scholar 

  20. Yu, C., Lan, J., Guo, Z., Hu, Y.: Drom: optimizing the routing in software-defined networks with deep reinforcement learning. IEEE Access 6, 64533–64539 (2018)

    Article  Google Scholar 

  21. Chen, Y.-R., Rezapour, A., Tzeng, W.-G., Tsai, S.-C.: Rl-routing: an sdn routing algorithm based on deep reinforcement learning. IEEE Trans. Network Sci. Eng. 7(4), 3185–3199 (2020)

    Article  Google Scholar 

  22. Zhang, J., Ye, M., Guo, Z., Yen, C.-Y., Chao, H.J.: Cfr-rl: Traffic engineering with reinforcement learning in sdn. IEEE J. Selected Areas Commun. 38(10), 2249–2259 (2020)

    Article  Google Scholar 

  23. Chen, C., Xue, F., Lu, Z., Tang, Z., Li, C., et al.: Rlmr: reinforcement learning based multipath routing for sdn. Wireless Commun. Mobile Comput. (2022). https://doi.org/10.1155/2022/5124960

    Article  Google Scholar 

  24. Lu, Y., Chen, Y., Xu, X., Fu, Q., Chen, J., Liu, L.: A sub-flow adaptive multipath routing algorithm for data centre network. Int. J. Comput. Intell. Syst. 16(1), 25 (2023)

    Article  Google Scholar 

  25. Chiu, K.-C., Liu, C.-C., Chou, L.-D., et al.: Reinforcement learning-based service-oriented dynamic multipath routing in sdn. Wireless Commun. Mobile Comput. (2022). https://doi.org/10.1155/2022/5124960

    Article  Google Scholar 

  26. Huang, H., Zhu, X., Bi, J., Cao, W., Zhang, X.: Machine learning for broad-sensed internet congestion control and avoidance: a comprehensive survey. IEEE Access 9, 31525–31545 (2021)

    Article  Google Scholar 

  27. Isravel, D.P., Silas, S., Rajsingh, E.B.: Centrality based congestion detection using reinforcement learning approach for traffic engineering in hybrid sdn. J. Network Syst. Manage. 30(1), 2 (2022)

    Article  Google Scholar 

  28. Amin, S., Gomrokchi, M., Satija, H., Hoof, H., Precup, D.: A survey of exploration methods in reinforcement learning. arXiv preprint arXiv:2109.00157 (2021)

  29. Zhao, J., Tong, M., Qu, H., Zhao, J.: An intelligent congestion control method in software defined networks. In: 2019 IEEE 11th International Conference on Communication Software and Networks (ICCSN), pp. 51–56 (2019). IEEE

  30. Bouzidi, E.H., Outtagarts, A., Langar, R.: Deep reinforcement learning application for network latency management in software defined networks. In: 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1–6 (2019). IEEE

  31. Wang, C., Cao, W., Hu, Y., Liu, J.: Data center traffic scheduling strategy for minimization congestion and quality of service guaranteeing. Comput. Mater. Continua (2023). https://doi.org/10.32604/cmc.2023.037625

    Article  Google Scholar 

  32. Phemius, K., Bouet, M.: Monitoring latency with openflow. In: Proceedings of the 9th International Conference on Network and Service Management (CNSM 2013), pp. 122–125 (2013). IEEE

  33. Jang, B., Kim, M., Harerimana, G., Kim, J.W.: Q-learning algorithms: a comprehensive classification and applications. IEEE Access 7, 133653–133667 (2019)

    Article  Google Scholar 

  34. Tokic, M., Palm, G.: Value-difference based exploration: adaptive control between epsilon-greedy and softmax. In: Annual Conference on Artificial Intelligence, pp. 335–346. Springer, Berlin (2011)

    Google Scholar 

  35. Dholakiya, D., Kshirsagar, T., Nayak, A.: Survey of mininet challenges, opportunities, and application in software-defined network (sdn). In: International Conference on Information and Communication Technology for Intelligent Systems, pp. 213–221. Springer, Berlin (2020)

    Google Scholar 

  36. “Mininet”: [Online] Available At. http://mininet.org

  37. “RYU”: [Online].Available At. https://ryu-sdn.org/

Download references

Author information

Authors and Affiliations

  1. RISC Laboratory, National Engineering School of Tunis, University of Tunis El Manar, 1002, Tunis, Tunisia

    Houda Hassen, Soumaya Meherzi & Zouhair Ben Jemaa

Authors
  1. Houda Hassen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Soumaya Meherzi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zouhair Ben Jemaa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Houda Hassen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hassen, H., Meherzi, S. & Jemaa, Z.B. Improved Exploration Strategy for Q-Learning Based Multipath Routing in SDN Networks. J Netw Syst Manage 32, 25 (2024). https://doi.org/10.1007/s10922-024-09804-0

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10922-024-09804-0

Keywords

Search

Navigation