Skip to main content

Deep Reinforcement Learning for Cost-Effective Controller Placement in Software-Defined Multihop Wireless Networking

  • Conference paper
  • First Online:
Machine Learning for Networking (MLN 2021)

Abstract

One of the key features in software-defined networking (SDN) with a multi-controller environment is the controller placement problem (CPP) that aims to find the number of controllers, controller placements and controller assignment. Solving the CPP in software-defined multihop wireless networking (SDMWN) has a significant impact on the generated control overhead. In SDMWN, devices use unreliable and shared multihop wireless communications without the help of any infrastructure, such as a base station or an access point. Various algorithms have been proposed to find near-optimal solutions for the CPP. Deep reinforcement learning (DRL) becomes popular in various fields and a few studies have also investigated using DRL for the CPP in wired networks and infrastructure-based wireless networks. However, DRL has not been researched for the CPP in SDMWN. Hence, in this paper, the potential of using DRL to the CPP in SDMWN for a given number of controllers is investigated to minimize the generated control overhead referred to as the network cost. The results show that the adapted DRL is able to find controller placements and assign the controllers to devices such that the obtained network cost and the average number of hops among devices and their assigned controllers, as well as the average number of hops among different controllers in the network, are close to those obtained from the optimal solutions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Heller, B., Sherwood, R., McKeown, N.: The controller placement problem. In: Proceedings of the 1st Workshop on Hot topics in Software Defined Networks, pp. 7–12 (2012)

    Google Scholar 

  2. Das, T., Sridharan, V., Gurusamy, M.: A survey on controller placement in SDN. IEEE Commun. Surv. Tutor. 22(1), 472–503 (2020)

    Article  Google Scholar 

  3. Isong, B., Molose, R.R.S., Abu-Mahfouz, A.M., Dladlu, N.: Comprehensive review of SDN controller placement strategies. IEEE Access 8, 170070–170092 (2020)

    Article  Google Scholar 

  4. Yao, G., Bi, J., Li, Y., Guo, L.: On the capacitated controller placement problem in software defined networks. IEEE Commun. Lett. 18,(8), 1339–1342 (2014)

    Google Scholar 

  5. Zahmatkesh, A., Kunz, T., Lung, C.-H.: Cost-effective controller placement problem for software defined multihop wireless networks. In: Foschini, L., El Kamili, M. (eds.) ADHOCNETS 2020. LNICSSITE, vol. 345, pp. 130–146. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-67369-7_10

    Chapter  Google Scholar 

  6. Sallahi, A., St-Hilaire, M.: Expansion model for the controller placement problem in software defined networks. IEEE Commun. Lett. 21(2), 274–277 (2017)

    Article  Google Scholar 

  7. Zahmatkesh, A., Kunz, T.: Software defined multihop wireless networks: promises and challenges. J. Commun. Netw. 19(6), 546–554 (2017)

    Article  Google Scholar 

  8. Kumari, A., Sairam, A.S.: Controller placement problem in software-defined networking: a survey. Networks, 1–29 (2021)

    Google Scholar 

  9. Roscher, R., Bohn, B., Duarte, M.F., Garcke, J.: Explainable machine learning for scientific insights and discoveries. IEEE Access 8, 42200–42216 (2020)

    Article  Google Scholar 

  10. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A.: Deep reinforcement learning: a brief survey. IEEE Signal Process. Mag. 34, 26–38 (2017)

    Article  Google Scholar 

  11. Fran cois-Lavet, V., Henderson, P., Islam, R., Bellemare, M.G., Pineau, J.: An Introduction to Deep Reinforcement Learning. Now Publishing Inc., Boston (2018)

    Google Scholar 

  12. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press, Cambridge (2016)

    Google Scholar 

  13. Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press, Cambridge (1998)

    Google Scholar 

  14. Mostafaei, H., Menth, M., Obaidat, M.S.: A learning automaton-based controller placement algorithm for software-defined networks. In: 2018 IEEE Global Communications Conference (GLOBECOM), pp. 1–6 (2018)

    Google Scholar 

  15. Wu, Y., Zhou, S., Wei, Y., Leng, S.: Deep reinforcement learning for controller placement in software defined network. In: IEEE INFOCOM 2020 - IEEE Conference on Computer Communications Workshops (INFOCOM WKSHPS), pp. 1254–1259 (2020)

    Google Scholar 

  16. Yuan, T., Neto, W.d.R., Rothenberg, C.E., Obraczka, K., Barakat, C., Turletti, T.: Dynamic controller assignment in software defined internet of vehicles through multi-agent deep reinforcement learning. In: IEEE Transactions on Network and Service Management, vol. 18, no. 1, pp. 585–596 (2021)

    Google Scholar 

  17. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep reinforcement learning. Nature 518, 529–533 (2015)

    Article  Google Scholar 

  18. Wang, G., Zhao, Y., Huang, J., Duan, Q., Li, J.: A K-means-based network partition algorithm for controller placement in software defined network. In: 2016 IEEE International Conference on Communications (ICC), pp. 1–6 (2016)

    Google Scholar 

  19. Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I.: Multi-agentactor-critic for mixed cooperative-competitive environments. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS 2017, Red Hook, NY, USA, pp. 6382–6393, Curran Associates Inc. (2017)

    Google Scholar 

  20. Fourer, R., Gay, D.M., Kernighan, B.: Algorithms and model formulations in mathematical programming. AMPL: A Mathematical Programming Language, pp. 150–151. Springer, Heidelberg (1989). https://doi.org/10.1007/978-3-642-83724-1

  21. Gropp, W., More, J.J.: Optimization environments and the NEOS server. In: Buhmann, M.D., Iserles, A. (eds.) Approximation Theory and Optimization, pp. 167–182. Cambridge University Press, Cambridge (1997)

    Google Scholar 

  22. CPLEX: IBM’s Linear Programming Solver. http://www.ilog.com/product/cplex/. Accessed May 2021

  23. NEOS Server for CPLEX/AMPL. https://neos-server.org/neos/solvers/lp:CPLEX/AMPL.html. Accessed May 2021

Download references

Acknowledgment

The authors acknowledge the support from the Natural Sciences and Engineering Research Council of Canada (NSERC) through the Discovery Grant program.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Chung-Horng Lung .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Zahmatkesh, A., Lung, CH. (2022). Deep Reinforcement Learning for Cost-Effective Controller Placement in Software-Defined Multihop Wireless Networking. In: Renault, É., Boumerdassi, S., Mühlethaler, P. (eds) Machine Learning for Networking. MLN 2021. Lecture Notes in Computer Science, vol 13175. Springer, Cham. https://doi.org/10.1007/978-3-030-98978-1_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-98978-1_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-98977-4

  • Online ISBN: 978-3-030-98978-1

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics