Skip to main content

Advertisement

SpringerLink
Log in

Schedule-Based Cooperative Multi-agent Reinforcement Learning for Multi-channel Communication in Wireless Sensor Networks

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Wireless sensor networks (WSNs) have become an important component in the Internet of things (IoT) field. In WSNs, multi-channel protocols have been developed to overcome some limitations related to the throughput and delivery rate which have become necessary for many IoT applications that require sufficient bandwidth to transmit a large amount of data. However, the requirement of frequent negotiation for channel assignment in distributed multi-channel protocols incurs an extra-large communication overhead which results in a reduction of the network lifetime. To deal with this requirement in an energy-efficient way is a challenging task. Hence, the Reinforcement Learning (RL) approach for channel assignment is used to overcome this problem. Nevertheless, the use of the RL approach requires a number of iterations to obtain the best solution which in turn creates a communication overhead and time-wasting. In this paper, a Self-schedule based Cooperative multi-agent Reinforcement Learning for Channel Assignment (SCRL CA) approach is proposed to improve the network lifetime and performance. The proposal addresses both regular traffic scheduling and assignment of the available orthogonal channels in an energy-efficient way. We solve the cooperation between the RL agents problem by using the self-schedule method to accelerate the RL iterations, reduce the communication overhead and balance the energy consumption in the route selection process. Therefore, two algorithms are proposed, the first one is for the Static channel assignment (SSCRL CA) while the second one is for the Dynamic channel assignment (DSCRL CA). The results of extensive simulation experiments show the effectiveness of our approach in improving the network lifetime and performance through the two algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

This study did not use data or datasets. This study is based on using randomly generated data by the simulator as input.

Code availability

Custom code is available upon request due to privacy or other restrictions.

References

  1. Phung, K. H., Lemmens, B., Goossens, M., Nowe, K., Tran, L., & Steenhaut, K. (2015). Schedule-based multi-channel communication in wireless sensor networks: A complete design and performance Evaluation. Ad-hoc Networks, 26, 88–102.

    Article  Google Scholar 

  2. Phung, K. H., Lemmens, B., Mihaylov, M., Tran, L., & Steenhaut, K. (2013). Adaptive learning based scheduling in multichannel protocol for energy-efficient data-gathering wireless sensor networks. International Journal of Distributed Sensor Networks, 9, 1–11.

    Article  Google Scholar 

  3. Hao, X. C., Gong, Q. Q., Hou, S., & Liu, B. (2014). Joint channel allocation and power control optimal algorithm based on non-cooperative game inwireless sensor networks. Wireless Personal Communications, 78(2), 1–15.

    Google Scholar 

  4. Hao, X. C., Zhang, Y. X., & Liu, B. (2017). Fair dynamic spectrum allocation using modied game theory for resource-constrained cognitive wireless sensor networks. Symmetry, 9(5), 73–87.

    Article  Google Scholar 

  5. Sengupta, S., Chatterjee, M., & Kwiat, K. A. (2010). A game theoretic framework for power control in wireless sensor networks. IEEE Transactions on Computers, 59(2), 231–42.

    Article  MathSciNet  Google Scholar 

  6. Anamika, S., & Siddhartha, C. (2020). A distributed reinforcement learning based sensor node scheduling algorithm for coverage and connectivity maintenance in wireless sensor network. Wireless Networks, 26, 4411–4429.

    Article  Google Scholar 

  7. Mu, Q., Haitao, Z., Shengchun, H., Li, Z., & Shan, W. (2017). Optimal channel selection based on online decision and offline learning in multichannel wireless sensor networks. Wireless Communications and Mobile Computing. https://doi.org/10.1155/2017/7902579.

    Article  Google Scholar 

  8. Rehan, W., Fischer, S., & Rehan, M. (2016). Machine-learning based channel quality and stability estimation for stream-based multichannel wireless sensor networks. Journal of Sensors (Basel). https://doi.org/10.3390/s16091476.

    Article  Google Scholar 

  9. Kim, Y., Shin, H., & Cha, H.(2008). Y-MAC: An energy-efficient multi-channel MAC protocol for dense wireless sensor networks. In: Proceedings of the 7th International Conference on Information Processing in Sensor Networks (IPSN08), St. Louis, MO, USA.

  10. Abdul Hamid, Md., Abdullah-Al-Wadud, M., & Ilyoung, C. (2010). A schedule-based multi-channel MAC protocol for wireless sensor networks. Sensors (Basel), 10(10), 9466–9480.

    Article  Google Scholar 

  11. Ozlem, D. I., Lodewijk, V. H., Pierre, J., & Paul, H. (2011). MC-LMAC: A multi-channel MAC protocol for wireless sensor networks. Ad Hoc Networks, 9(1), 73–94.

    Article  Google Scholar 

  12. Chen, J., Yu, Q., Chai, B., Sun, Y., Fan, Y., & Shen, X. (2015). Dynamic channel assignment for wireless sensor networks: A regret matching based approach. IEEE Transactions on Parallel and Distributed Systems, 26(1), 95–1068.

    Article  Google Scholar 

  13. Panos, NA., Efthymios, AV., & Aggelos, B. (2017). Inference-based distributed channel allocation in wireless sensor Networks. Journal of Information Theory (cs.IT), CoRR abs/1703.06652.

  14. Hao, X. C., Gong, Q. Q., Hou, S., & Liu, B. (2015). Energy efficient based channel assignment game algorithm for wireless sensor networks. Wireless Personal Communications, 85(4), 2749–2771.

    Article  Google Scholar 

  15. Mihaylov, M., Borgne, Y. A., Tuyls, K., & Nowé, A. (2012). Decentralised reinforcement learning for energy-efficient scheduling in wireless sensor networks. International Journal of Communication Networks and Distributed Systems, 9, 207–224.

    Article  Google Scholar 

  16. Khan, M. I., Kia, K., Ali, A., & Aslam, N. (2017). Energy-aware task scheduling by a true online reinforcement learning in wireless sensor networks. International Journal of Sensor Networks, 25(4), 244–258.

    Article  Google Scholar 

  17. Selvia, F. L. D., & Costa, A. H. R. (2019). A survey on transfer learning for multiagent reinforcement learning systems. Journal of Artifitial Intelligence Research, 64, 645–703.

    Article  MathSciNet  Google Scholar 

  18. Khan, M., & Rinner, B. (2014). Energy-aware task scheduling in wireless sensor networks based on cooperative reinforcement learning. Proceedings of the IEEE International Conference on Communications Workshops, Sydney, Australia, pp. 871–877.

  19. Yujia, G., Yurong, N., & Xiahai, G. (2021). Maximizing network throughput by cooperative reinforcement learning in clustered solar-powered wireless sensor networks. International Journal of Distributed. https://doi.org/10.1177/15501477211007411.

    Article  Google Scholar 

  20. Pal, A., & Nasipuri, . (2017). A distributed routing and channel selection for multi-channel wireless sensor networks. Journal of Sensor and Actuator Networks, 6(3), 1–13.

    Article  Google Scholar 

  21. Ibrahim, M., Borhanuddin, M. A., Sali, A., Rasid, M. F. A., & Mohamad, H. (2017). An energy efficient reinforcement learning based cooperative channel sensing for cognitive radio sensor networks. Pervasive and Mobile Computing, 35, 165–184.

    Article  Google Scholar 

  22. Xuan, T., Li, Z., Haijun, W., Yuli, S., Haitao, Z., Boon-Chong, S., Jibo, W., & Vector, C.M L. (2021). Cooperative Multi-Agent Reinforcement Learning Based Distributed Dynamic Spectrum Access in Cognitive Radio Networks. arXiv:2106.09274.

  23. Yao, F., & Jia, L. (2019). A collaborative multi-agent reinforcement learning anti-jamming algorithm in wireless networks. IEEE Wireless Communications Letters, 8(4), 1024–1027.

    Article  MathSciNet  Google Scholar 

  24. Kulshrestha, J., & Mishra, M. K. (2017). An adaptive energy balanced and energy eficient approach for data gathering in wireless sensor networks. Ad Hoc Networks, 54, 130–146.

    Article  Google Scholar 

  25. IEEE 802.15 WPAN Task Group, IEEE Std 802.15.3TM (2006) Wireless Medium Acces Control (MAC) and Physical Layer (PHY) Specifications for High Rate Wireless Personal Area Networks (WPAN). https://profsite.um.ac.ir/hyaghmae/ACN/WSNMAC1.pdf

  26. Morozs, N., Clarke, T., & Grace, D. (2013). A novel adaptive call admission control scheme for distributed reinforcement learning based dynamic spectrum access in cellular networks. Proc. Int. Symp. Wireless Commun. Syst. (ISWCS), pp. 1–5.

  27. Bowlingand, M., & Veloso, M. (2002). Multi agent learning using a variable learning rate. Artificial Intelligence, 136(2), 215–250.

    Article  MathSciNet  Google Scholar 

  28. Xuan, P., Hung, ML., David, FS., & Ara, N. (2018). Cooperative and distributed reinforcement learning of drones for field coverage. arXiv:1803.07250.

  29. HPC Lab (2016). JSensor: high performance computing, wireless sensor simulator. https://joubertlimadotcomdotbr.wordpress.com/jsensor-a-high-performance-java-simulator-for-sensor-networks/

  30. Gharbi, C., Aliouat, Z., & Benmohammed, M. (2018). A novel load balancing scheduling algorithm for wireless sensor networks. Journal of Network and Systems Management, 27(2), 430–462.

    Article  Google Scholar 

Download references

Funding

This research did not receive any specific funding and is being conducted as part of employment and higher degree of the authors.

Author information

Authors and Affiliations

  1. Mohamed Khider University, LIAM Laboratory, M’sila, Algeria

    Mohamed Sahraoui

  2. LaSTIC Laboratory of Batna2 University, Batna, Algeria

    Azeddine Bilami

  3. IEMN DOAE Laboratory, Hauts de France Polytechnic University, Valenciennes, France

    Abdelmalik Taleb-Ahmed

Authors
  1. Mohamed Sahraoui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Azeddine Bilami
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Abdelmalik Taleb-Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed Sahraoui.

Ethics declarations

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sahraoui, M., Bilami, A. & Taleb-Ahmed, A. Schedule-Based Cooperative Multi-agent Reinforcement Learning for Multi-channel Communication in Wireless Sensor Networks. Wireless Pers Commun 122, 3445–3465 (2022). https://doi.org/10.1007/s11277-021-09094-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-021-09094-8

Keywords

Search

Navigation