Skip to main content
SpringerLink
Log in

Reinforcement learning based routing in wireless mesh networks

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

This paper addresses the problem of efficient routing in backbone wireless mesh networks (WMNs) where each mesh router is equipped with multiple radio interfaces and a subset of nodes serve as gateways to the Internet. Most routing schemes have been designed to reduce routing costs by optimizing one metric, e.g., hop count and interference ratio. However, when considering these metrics together, the complexity of the routing problem increases drastically. Thus, an efficient and adaptive routing scheme that takes into account several metrics simultaneously and considers traffic congestion around the gateways is needed. In this paper, we propose an adaptive scheme for routing traffic in WMNs, called Reinforcement Learning-based Distributed Routing (RLBDR), that (1) considers the critical areas around the gateways where mesh routers are much more likely to become congested and (2) adaptively learns an optimal routing policy taking into account multiple metrics, such as loss ratio, interference ratio, load at the gateways and end-to end delay. Simulation results show that RLBDR can significantly improve the overall network performance compared to schemes using either Metric of Interference and Channel switching, Best Path to Best Gateway, Expected Transmission count, nearest gateway (i.e., shortest path to gateway) or load at gateways as a metric for path selection.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Akyildiz, I. F., Wang, X., & Wang, W. (2005). Wireless mesh networks: A survey. Computer Networks, 47(4), 445–487.

    Article  MATH  Google Scholar 

  2. Benyamina, D., Hafid, A., & Gendreau, M. (2012). Design of scalable and efficient multi-radio wireless networks. Wireless Networks, 18(1), 75–94.

    Article  Google Scholar 

  3. Jun, J., & Sichitiu, M. L. (2003). The nominal capacity of wireless mesh networks. IEEE Wireless Communications, 10(5), 8–14.

    Article  Google Scholar 

  4. De Couto, D. S., Aguayo, D., Bicket, J., & Morris, R. (2005). A high-throughput path metric for multi-hop wireless routing. Wireless Networks, 11(4), 419–434.

    Article  Google Scholar 

  5. Draves, R., Padhye, J., & Zill, B. (2004). Routing in multi-radio, multi-hop wireless mesh networks. In Proceedings of the 10th Annual International Conference on Mobile Computing and Networking, pp. 114–128.

  6. Campista, M. E. M., Esposito, P. M., Moraes, I. M., Costa, L. H. M., Duarte, O. C. M., Passos, D. G., et al. (2008). Routing metrics and protocols for wireless mesh networks. IEEE Network, 22(1), 6–12.

    Article  Google Scholar 

  7. Boukerche, A. (2004). Performance evaluation of routing protocols for ad hoc wireless networks. Mobile Networks and Applications, 9(4), 333–342.

    Article  Google Scholar 

  8. Fantacci, R., Tarchi, D., & Tassi, A. (2010). A novel routing algorithm for mobile pervasive computing. In Proceedings of Global Telecommunications Conference, pp. 1–5.

  9. Fantacci, R., Tarchi, D., & Tassi, A. (2011). Wireless communication protocols for distributed computing environments. In M. Khatib (Ed.), Advanced trends in wireless communications. Rijeka: InTech.

    Google Scholar 

  10. Clausen, T., & Jacquet, P. (2003). Optimized link state routing protocol (OLSR). RFC3626. IETF internet draft, http://www.ietf.org/rfc/rfc3626.txt.

  11. Perkins, C. E., & Royer, E. M. (1999). Ad-hoc on-demand distance vector routing. In Proceedings of the Second Workshop on Mobile Computing Systems and Applications, pp. 90–100.

  12. Park, V. D., & Corson, M. S. (1997). A highly adaptive distributed routing algorithm for mobile wireless networks. In Proceedings the Sixteenth Annual Joint Conference of the IEEE Computer and Communications Societies, pp. 1405–1413.

  13. Yang, Y., Wang, J., & Kravets, R. (2005). Interference-aware load balancing for multihop wireless networks, Technical report. University of Illinois at Urbana-Champaign, 361702.

  14. Boushaba, M., & Hafid, A. (2011). Best path to best gateway scheme for multichannel multi-interface wireless mesh networks. In Proceedings of Wireless Communications and Networking Conference, pp. 689–694.

  15. Das, S., Wu, Y., Chandra, R., & Hu, Y. C. (2008). Context-based routing: Technique, applications, and experience. In Proceedings of the 5th USENIX Symposium on Networked Systems Design and Implementation, pp. 379–392.

  16. Ma, L., & Denko, M. K. (2007, May). A routing metric for load-balancing in wireless mesh networks. In Proceedings of the 21st International Conference on Advanced Information Networking and Applications Workshops, pp. 409–414.

  17. Shah, K., Di Francesco, M., & Kumar, M. (2012). Distributed resource management in wireless sensor networks using reinforcement learning. Wireless Networks, 1–20. doi:10.1007/s11276-012-0496-2.

  18. Boyan, J. A., & Littman, M. L. (1994). Packet routing in dynamically changing networks: A reinforcement learning approach. In Proceedings Advances in Neural Information Processing Systems, pp. 671–67.

  19. Belbekkouche, A., Hafid, A., & Gendreau, M. (2009). Novel reinforcement learning-based approaches to reduce loss probability in buffer-less OBS networks. Computer Networks, 53(12), 2091–2105.

    Article  MATH  Google Scholar 

  20. Celimuge, W. U., & Kumekawa, K. (2010). Distributed reinforcement learning approach for vehicular ad hoc networks. IEICE Transactions on Communications, 93(6), 1431–1442.

    Google Scholar 

  21. Beyens, P., Peeters, M., Steenhaut, K., & Nowe, A. (2005). Routing with compression in wireless sensor networks: A Q-learning approach. In Proceedings of the Fifth European Workshop on Adaptive Agents and Multi-Agent Systems.

  22. Lee, M., Marconett, D., Ye, X., & Yoo, S. B. (2007). Cognitive network management with reinforcement learning for wireless mesh networks. In Proceedings of IP Operations and Management, pp. 168–179.

  23. Atlasis, A. F., Loukas, N. H., & Vasilakos, A. V. (2000). The use of learning algorithms in ATM networks call admission control problem: A methodology. Computer Networks, 34(3), 341–353.

    Article  Google Scholar 

  24. Vasilakos, A., Saltouros, M. P., Atlassis, A. F., & Pedrycz, W. (2003). Optimizing QoS routing in hierarchical ATM networks using computational intelligence techniques. IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, 33(3), 297–312.

    Article  Google Scholar 

  25. Vasilakos, A., Ricudis, C., Anagnostakis, K., Pedryca, W., & Pitsillides, A. (1998). Evolutionary-fuzzy prediction for strategic QoS routing in broadband networks. In Proceedings of the IEEE World Congress on Computational Intelligence, pp. 1488–1493.

  26. Zeng, Y., Xiang, K., Li, D., & Vasilakos, A. V. (2013). Directional routing and scheduling for green vehicular delay tolerant networks. Wireless Networks, 19(2), 161–173.

    Article  Google Scholar 

  27. Gambardella, L. M., & Dorigo, M. (1995). Ant-Q: A reinforcement learning approach to the traveling salesman problem. In Proceedings of the 12th International Machine Learning Conference, pp. 252–260.

  28. Dorigo, M., Caro, G. D., & Gambardella, L. M. (1999). Ant algorithms for discrete optimization. Artificial life, 5(2), 137–172.

    Article  Google Scholar 

  29. Subramanian, D., Druschel, P., & Chen, J. (1997). Ants and reinforcement learning: A case study in routing in dynamic networks. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence, pp. 832–839.

  30. Iima, H., Kuroe, Y., & Matsuda, S. (2010, October). Swarm reinforcement learning method based on ant colony optimization. In Proceedings of International Conference on Systems Man and Cybernetics, pp. 1726–1733.

  31. Forster, A. (2007). Machine learning techniques applied to wireless ad-hoc networks: Guide and survey. In Proceedings of the 3rd International Conference on Intelligent Sensors, Sensor Networks and Information Processing, pp. 365–370.

  32. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285.

    Google Scholar 

  33. Sim, K. M., & Sun, W. H. (2003). Ant colony optimization for routing and load-balancing: Survey and new directions. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 33(5), 560–572.

    Article  Google Scholar 

  34. Shen, C. C., Huang, Z., & Jaikaeo, C. (2005). Ant-based distributed topology control algorithms for mobile ad hoc networks. Wireless Networks, 11(3), 299–317.

    Article  Google Scholar 

  35. Boushaba, M., Hafid, A., & Belbekkouche, A. (2011). Reinforcement learning-based best path to best gateway scheme for wireless mesh networks. In Proceedings of the 7th International Conference on Wireless and Mobile Computing Networking and Communications, pp. 373–379.

  36. Xu, J., Liu, W., Yang, Z., Chen, J., & Chen, X. (2010). A delay-aware routing metric for multi-radio multi-channel wireless mesh networks. In Proceedings of the 6th Internatinoal Conference on Wireless Communications Networking and Mobile Computing, pp. 1–4.

  37. Subramanian, A. P., Buddhikot, M. M., & Miller, S. (2006). Interference aware routing in multi-radio wireless mesh networks. In Proceedings of IEEE Workshop on Wireless Mesh Networks, pp. 55–63.

  38. Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine Learning, 8(3–4), 279–292.

    MATH  Google Scholar 

  39. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction, 1(1). Cambridge, MA: MIT Press.

    Google Scholar 

  40. Calvo, R.A., Campo, J.P. (2007). Adding Multiple Interface Support in NS-2, http://personales.unican.es/aguerocr/files/ucMultiIfacesSupport.pdf.

  41. Beljadid, A., Hafid, A. S., & Gendreau, M. (2007). Optimal design of broadband wireless mesh networks. In Proceedings of Global Telecommunications Conference, pp. 4840–4845.

  42. Intersil HFA3861B Chip Specification, http://www.chipdocs.com/pndecoder/datasheets/INTRS/HFA3861B.html.

  43. Borges, V. C., Pereira, D., Curado, M., & Monteiro, E. (2009). Routing metric for interference and channel diversity in multi-radio wireless mesh networks. In Proceedings of Ad-Hoc, Mobile and Wireless Networks, pp. 55–68.

Download references

Acknowledgments

We would like to thank Mr Vinicius da Cunha Martins Borges [43] for providing us with code for simulation. The research reported in this manuscript has been supported in part by Natural Sciences and Engineering Research Council of Canada (NSERC) and Bell Canada.

Author information

Authors and Affiliations

  1. Network Research Laboratory, University of Montreal, Montreal, Canada

    Mustapha Boushaba, Abdelhakim Hafid & Abdeltouab Belbekkouche

  2. CIRRELT and MAGI, École Polytechnique de Montreal, Montreal, Canada

    Michel Gendreau

Authors
  1. Mustapha Boushaba
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Abdelhakim Hafid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdeltouab Belbekkouche
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Michel Gendreau
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mustapha Boushaba.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Boushaba, M., Hafid, A., Belbekkouche, A. et al. Reinforcement learning based routing in wireless mesh networks. Wireless Netw 19, 2079–2091 (2013). https://doi.org/10.1007/s11276-013-0592-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-013-0592-y

Keywords

Search

Navigation