Abstract
The radio frequency spectrum is a scarce resource and cognitive radio has been under heavy research to improve the utilization of spectrum in the past thirty years. It is crucial to optimize the performance of cognitive radio for high values for practical applications while it has turned out to be very technically challenging. The conventional cognitive radio methods have strong pertinence and coupling because they are generally designed for a specific application environment. To address the problem of spectrum sharing with collision avoidance mechanisms in cognitive radio, in this paper we propose a new momentum-based Q-learning algorithm to accelerate reinforcement learning based spectrum sharing algorithms for cognitive radio transmitters. We conduct a performance evaluation study based on a simulation toolkit for the reinforcement learning research and the ns-3 network simulator “ns3-gym”. As a demonstrating case study, the proposed algorithm is able to capture the learnable patterns from a periodic channel occupation in a wireless environment and avoid channel collision effectively, finally improving channel efficiency and reducing the end-to-end time delay. The simulation results demonstrated that our proposed momentum Q-learning algorithm achieves a lower collision rate, faster convergence as well as stronger generalization capacity compared with two conventional algorithms including a greedy algorithm and a deep Q-learning network algorithm.
L. Zhu, Z. Zhou and Z. Peng—These authors contributed equally to this work.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Gao, Y., Dai, L.: Random access: packet-based or connection-based? IEEE Trans. Wirel. Commun. 18(5), 2664–2678 (2019)
Gao, Y., Roy, S.: Achieving proportional fairness for LTE-LAA and Wi-Fi coexistence in unlicensed spectrum. IEEE Trans. Wirel. Commun 19(5), 3390–3404 (2020)
Gao, Y., Dai, L., Hei, X.: Throughput optimization of multi-BSS IEEE 802.11 networks with universal frequency reuse. IEEE Trans. Commun. 65(8), 3399–3414 (2017)
Gawłowicz, P., Zubow, A.: Ns-3 meets OpenAI Gym: the playground for machine learning in networking research. In: Proceedings of the 22nd International ACM Conference on Modeling, Analysis and Simulation of Wireless and Mobile Systems, pp. 113–120 (2019)
Kassab, R., Destounis, A., Tsilimantos, D., Debbah, M.: Multi-agent deep stochastic policy gradient for event based dynamic spectrum access (2020). arXiv
Leibovitz, J.S.: The great spectrum debate: a commentary on the FCC spectrum policy task force’s report on spectrum rights and responsibilities. Yale J. Law Technol. 6, 390–414 (2003)
Lo, B., Akyildiz, I.: Reinforcement learning for cooperative sensing gain in cognitive radio ad hoc networks. Wirel. Netw. 19, 1237–1250 (2012)
Manzoor, S., Chen, Z., Gao, Y., Hei, X., Cheng, W.: Towards QoS-aware load balancing for high density software defined Wi-Fi networks. IEEE Access 8, 117623–117638 (2020)
Manzoor, S., Yin, Y., Gao, Y., Hei, X., Cheng, W.: A systematic study of IEEE 802.11 DCF network optimization from theory to testbed. IEEE Access 8, 154114–154132 (2020)
Mitola, J., Maguire, G.Q.: Cognitive radio: making software radios more personal. IEEE Pers. Commun. 6(4), 13–18 (1999)
Rajpoot, D.: Sensing-throughput analysis in NOMA-based CR network (2020). arXiv
Valcarce, A., Hoydis, J.: Towards joint learning of optimal signaling and wireless channel access (2020) arXiv
Wei, J.B., Wang, S., Zhao, H.T.: Cognitive wireless networks: key techniques and sate of the art. Tongxin Xuebao/J. Commun 32, 147–158 (2011)
Wu, C., Chowdhury, K., Di Felice, M., Meleis, W.: Spectrum management of cognitive radio using multi-agent reinforcement learning. In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: Industry Track, pp. 1705–1712. International Foundation for Autonomous Agents and Multiagent Systems, Richland (2010)
Wu, Y., Qian, L.P., Ni, K., Zhang, C., Shen, X.: Delay-minimization nonorthogonal multiple access enabled multi-user mobile edge computation offloading. IEEE J. Sel. Topics Signal Process. 13(3), 392–407 (2019)
Yin, H., et al.: NS3-AI: fostering artificial intelligence algorithms for networking research. In: Proceedings of the 2020 Workshop on Ns-3, pp. 57–64 (2020)
Yu, Y., Liew, S.C., Wang, T.: Non-uniform time-step deep Q-network for carrier-sense multiple access in heterogeneous wireless networks. IEEE Trans. Mob. Comput. (2020)
Acknowledgment
The authors would like to express their gratitude to the anonymous reviewers for their constructive comments which help us to improve the quality of this paper very much. This work was supported in part by the National Natural Science Foundation of China (no. 61972172) and the teaching research fund by the Huazhong University of Science and Technology (no. 2018077).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering
About this paper
Cite this paper
Zhu, L., Zhou, Z., Peng, Z., Hei, X. (2021). Accelerating Spectrum Sharing Algorithms for Cognitive Radio Transmitters in a Momentum Q-Learning Approach. In: Song, H., Jiang, D. (eds) Simulation Tools and Techniques. SIMUtools 2020. Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering, vol 369. Springer, Cham. https://doi.org/10.1007/978-3-030-72792-5_42
Download citation
DOI: https://doi.org/10.1007/978-3-030-72792-5_42
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-72791-8
Online ISBN: 978-3-030-72792-5
eBook Packages: Computer ScienceComputer Science (R0)