Skip to main content
SpringerLink
Log in

An efficient routing access method based on multi-agent reinforcement learning in UWSNs

  • Original Paper
  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

A large proportion of underwater data is collected in deep sea. Compared with the direct bottom-to-surface acoustic links, underwater sensor networks (UWSNs) with hierarchical network model topology are more efficient at transmitting huge amounts of data to sea surface. Base on reinforcement learning, an adaptive modulation and coding in depth based router (MC-DBR) algorithm was proposed. The MC-DBR is designed to reduce the energy consumption, time delay etc., while improve the communication performance. In MC-DBR, each node firstly uses HELLO packets to sense the neighbouring channel states. Then, each node updates its Q-value by multi-agent reinforcement learning based modulation and coding method (MARL-MC) algorithm. The energy consumption, the time delay, the modulation and coding methods and the packets collisions etc. are considered in MARL-MC to improve the overall performance of the whole network. The convergence and computation complexity of the MC-DBR were analyzed in detail. The performance of the MC-DBR was compared with the benchmark algorithms. The results showed that the MC-DBR can obtain lower end-to-end delay, higher packet delivery rate and lower average remaining energy of the network.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

References

  1. Lin, Jie, Wei, Yu., Zhang, Nan, Yang, Xinyu, Zhang, Hanlin, & Zhao, Wei. (2017). A survey on internet of things: Architecture, enabling technologies, security and privacy, and applications. IEEE Internet of Things Journal, 4(5), 1125–1142.

    Article  Google Scholar 

  2. Javaid, N., Shakeel, U., Ahmad, A., et al. (2019). DRADS: Depth and reliability aware delay sensitive cooperative routing for underwater wireless sensor networks. Wireless Network, 25(2), 777–789.

    Article  Google Scholar 

  3. Khasawneh, A., Latiff, M. S. B. A., Kaiwartya, O., et al. (2018). A reliable energy-efficient pressure-based routing protocol for underwater wireless sensor network. Wireless Network, 24, 2061–2075.

    Article  Google Scholar 

  4. Albukhary, R. A., & Bouabdallah, F. (2019). Time-variant balanced routing strategy for underwater wireless sensor networks. Wireless Network, 25, 3481–3495.

    Article  Google Scholar 

  5. Ahmad, Khasawneh, Muhammad, Shafie Bin, Latiff, Abd, Omprakash, Kaiwartya, & Hassan, Chizari. (2017). Next forwarding node selection in underwater wireless sensor networks (uwsns): Techniques and challenges. Information, 8(1), 1–30.

    Google Scholar 

  6. Kumari, S., Mishra, P. K., & Anand, V. (2020). Fault resilient routing based on moth flame optimization scheme for underwater wireless sensor networks. Wireless Network, 26, 1417–1431.

    Article  Google Scholar 

  7. Hai, Yan, Zhijie Jerry Shi, & Jun-Hong, Cui. (2008). Dbr: Depth-based routing for underwater sensor networks. In Amitabha Das, Hung Keng Pung, Francis Bu Sung Lee, and Lawrence Wai Choong Wong, editors, NETWORKING 2008 Ad Hoc and Sensor Networks, Wireless Networks, Next Generation Internet, pages 72–86. Springer, Berlin.

  8. Safia, Gul, Sana Hoor Jokhio, & Imran Ali, Jokhio. (2018). Light-weight depth-based routing for underwater wireless sensor network. In 2018 International Conference on Advancements in Computational Sciences (ICACS), pages 1–7.

  9. Tanveer, Khan, Israr, Ahmad, Waqas, Aman, Irfan, Azam, Zahoor Ali, Khan, Umar, Qasim, Sanam, Avais, & Nadeem, Javaid. (2016). Clustering depth based routing for underwater wireless sensor networks. In 2016 IEEE 30th International Conference on Advanced Information Networking and Applications (AINA), pages 506–515, Crans-Montana, Switzerland, May.

  10. Abdul, Wahid, Sungwon, Lee, Hong-Jong, Jeong, & Dongkyun, Kim. (2011). Eedbr: Energy-efficient depth-based routing protocol for underwater wireless sensor networks. In Advanced Computer Science and Information Technology, pages 223–234, Springer, Berlin

  11. Tanveer, Ahmed, Maham, Chaudhary, Kaleem, M., & Sajid, Nazir. (2016). Optimized depth-based routing protocol for underwater wireless sensor networks. In 2016 International Conference on Open Source Systems Technologies (ICOSST), pages 147–150

  12. Vishu, Modi, & Gupta, C. P. (2019). Magnetic induction based modified depth based routing for underwater wireless sensor networks. In 2019 International Conference on Intelligent Sustainable Systems (ICISS), pages 7–14, Palladam, India

  13. Diao, B. et al., (2016). A Reliable Depth-Based Routing Protocol with Network Coding for Underwater Sensor Networks,” In 2016 IEEE 22nd International Conference on Parallel and Distributed Systems (ICPADS), pp. 270-27.

  14. Li, Chao, Yongjun, Xu., Diao, Boyu, Wang, Qi., & An, Zhulin. (2016). Dbr-mac: A depth-based routing aware mac protocol for data collection in underwater acoustic sensor networks. IEEE Sensors Journal, 16(10), 3904–3913.

    Article  Google Scholar 

  15. Hamza, Fahim, Nadeem, Javaid, Umar, Qasim, Zahoor Ali, Khan, Shumaila, Javed, Amir, Hayat, Zafar, Iqbal, & Ghazanfar, Rehman. (2015). Interference and bandwidth aware depth based routing protocols in underwater wsns. In 2015 9th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, pages 78–85, Santa Catarina, Brazil

  16. Liang, Xiao, Qiangda, Li, Tianhua, Chen, En Cheng, & Huaiyu, Dai. (2015). Jamming games in underwater sensor networks with reinforcement learning. In 2015 IEEE Global Communications Conference (GLOBECOM), pages 1–6

  17. Liang, Xiao, Donghua, Jiang, Xiaoyue, Wan, Wei, Su, & Yuliang, Tang. (2018). Anti-jamming underwater transmission with mobility and learning. IEEE Communications Letters, 22(3):542–545

  18. Yan, Li, Liang, Xiao, Qiangda, Li, & Wei, Su. (2015). Spoofing detection games in underwater sensor networks. In OCEANS 2015 - MTS/IEEE Washington, pages 1–5, Washington, DC, Oct.

  19. Yan, J., Gong, Y., Chen, C., Luo, X., Guan, X. (2020). AUV-Aided Localization for Internet of Underwater Things: A Reinforcement-Learning-Based Method. In IEEE Internet of Things Journal, vol. 7, no. 10, pp. 9728-9746

  20. Di Valerio, V., Lo Presti, F., Petrioli, C., Picari, L., Spaccini, D., & Basagni. S. (2019). CARMA: Channel-Aware Reinforcement Learning-Based Multi-Path Adaptive Routing for Underwater Wireless Sensor Networks. In IEEE Journal on Selected Areas in Communications, vol. 37, no. 11, pp. 2634-2647, Nov.

  21. Han, Guangjie, Long, Xiaohan, Zhu, Chuan, Guizani, Mohsen, Bi, Yuanguo, & Zhang, Wenbo. (2019). An auv location prediction-based data collection scheme for underwater wireless sensor networks. IEEE Transactions on Vehicular Technology, 68(6), 6037–6049.

    Article  Google Scholar 

  22. Jin, C., Allen-Zhu, Z., Bubeck, S., & Jordan, M. I. (2018). Is Q-learning provably efficient? In Proc. Conf. Adv. Neural Inf. Process. Syst. (NIPS), Montreal, QC, Canada, pp. 4868–4878, Dec.

  23. Urick, R. J. (1983). Principles of Underwater Sound McGrawHill.

    Google Scholar 

  24. Melo, F. S., & Ribeiro, M. I. (2007). Convergence of Q-learning with linear function approximation. European Control Conference (ECC), 2007, 2671–2678.

    Google Scholar 

  25. Peng, J., & Williams, R. J. (1993). Convergence of stochastic iterative dynamic programming algorithms. Adaptive Behavior, 1, 437–454.

    Article  Google Scholar 

  26. Wei, Su., Tao, Jincheng, Pei, Yuehua, You, Xudong, Xiao, Liang, & Cheng, En. (2020). Reinforcement learning based efficient underwater image communication. IEEE Communications Letters, 25(3), 883–886.

    Google Scholar 

Download references

Acknowledgements

This work was supported by the National Natural Science Foundation of China (62071400, 62071402).

Author information

Authors and Affiliations

  1. Information and Communication Engineering, Xiamen University, Xiamen, China

    Wei Su, Keyu Chen, Jiamin Lin & Yating Lin

Authors
  1. Wei Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Keyu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jiamin Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yating Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Keyu Chen.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A Proof of Theorem 1

Appendix A Proof of Theorem 1

According to [22], the computational complexity of each agent in the MARL-MC Algorithm is given by \({\mathcal {O}}\left( {kIJ} \right) \). Thus, the computation complexity of the MARL-MC Algorithm is given by \({{\mathcal {O}}}\left( {k{I^2}J} \right) \). The computational complexity of the MC-DBR Algorithm mainly contains that of Q-value update and the retransmission of DATA packets. According to [22], the computational complexity of Q-value update in MC-DBR Algorithm is given by \({\mathcal {O}}\left( IJ \right) \). According to the design of algorithm, we have \(IJ \gg r_m\). Hence, the computation complexity of the MC-DBR Algorithm is given by

$$\begin{aligned} {\mathcal {O}}\left( IJ+r_m \right) = {\mathcal {O}}\left( IJ \right) . \end{aligned}$$
(A1)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, W., Chen, K., Lin, J. et al. An efficient routing access method based on multi-agent reinforcement learning in UWSNs. Wireless Netw 28, 225–239 (2022). https://doi.org/10.1007/s11276-021-02838-1

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-021-02838-1

Keywords

Search

Navigation