Abstract
In cognitive radio networks, an important issue is to share the detected available spectrum among different secondary users to improve the network performance. Although some work has been done for dynamic spectrum access, the learning capability of cognitive radio networks is largely ignored in the previous work. In this paper, we propose a reinforcement-learning-based double auction algorithm aiming to improve the performance of dynamic spectrum access in cognitive radio networks. The dynamic spectrum access process is modeled as a double auction game. Based on the spectrum access history information, both primary users and secondary users can estimate the impact on their future rewards and then adapt their spectrum access or release strategies effectively to compete for channel opportunities. Simulation results show that the proposed reinforcement-learning-based double auction algorithm can significantly improve secondary users’ performance in terms of packet loss, bidding efficiency and transmission rate or opportunity access.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Haykin S. (2005) Cognitive radio: Brain-empowered wireless communications. IEEE Journal on Selected Areas in Communications 23(2): 201–220
Yucek T., Arslan H. (2009) A survey of spectrum sensing algorithms for cognitive radio applications. IEEE Communications Surveys & Tutorials 11(1): 116–130
Haykin S., Thomson D. J., Reed J. H. (2009) Spectrum sensing for cognitive radio. IEEE Proceedings 97(5): 849–877
Yu F. R., Huang M., Tang H. (2010) Biologically inspired consensus-based spectrum sensing in mobile ad hoc networks with cognitive radios. IEEE Networks 24(3): 26–30
Yiping X., Chandramouli R., Stefan M., Sai Shankar N. (2006) Dynamic spectrum access in open spectrum wireless networks. IEEE Journal on Selected Areas in Communications, 24(3): 626–637
Zhu J., Liu K. J. R. (2007) Cognitive radios for dynamic spectrum access—dynamic spectrum sharing: A game theoretical overview. IEEE Communications Magazine 45(5): 1–5
Wang B., Wu Y., Ji Z., Liu K. J. R., Clancy T. C. (2008) Game theoretical mechanism design methods. IEEE Signal Processing Magazine 25(6): 74–84
Krishnamurthy V. (2009) Decentralized spectrum access amongst cognitive agents—an interacting multivariate global games approach. IEEE Transactions on Signal Processing 57(10): 3999–4013
Niyato D., Hossain E. (2008) Competitive spectrum sharing in cognitive radio networks: A dynamic game approach. IEEE Transactions on Wireless Communications 7(7): 2651–2660
Zhu J., Liu K. J. R. (2008) Multi-stage pricing game for collusion-resistant dynamic spectrum allocation. IEEE Journal on Selected Areas in Communications 26(1): 182–191
Zhu, J., & Liu, K. J. R. (2006). Belief-assisted pricing for dynamic spectrum allocation in wireless networks with selfish users. In Proceedings of IEEE SECON’06, pp. 119–127.
Niyato D., Hossain E. (2008) Market-equilibrium, competitive, and cooperative pricing for spectrum sharing in cognitive radio networks: Analysis and comparison. IEEE Transactions on Wireless Communications 7(11): 4273–4283
Niyato D., Hossain E. (2008) Spectrum trading in cognitive radio networks: A market-equilibrium-based approach. IEEE Wireless Communications 15(6): 71–80
Bobrow D. G. (1994) Artificial intelligence in perspective. MIT Press, Cambridge, MA
Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236), 71–80, 1–5.
Sutton R. S., Barto A. G. (1998) Reinforcement learning. MIT Press, Cambridge, MA
Yu F. R., Wong V. W. S., Leung V. C. M. (2008) A new QoS provisioning method for adaptive multimedia in wireless networks. IEEE Transactions on Vehicular Technology 57(3): 1899–1909
Bernardo F., Agusti R., Perez-Romero J., Sallent O. (2011) Intercell interference management in OFDMA networks: A decentralized approach based on reinforcement learning. IEEE Transactions on Systems, Man and Cybernetics, Part C, 41(6): 1–9
Gibbons R. D. (1992) Game theory for applied economists. Princeton University Presss, Princeton
Shankar, S., Chou, C. T., Challapali, K., & Mangold, S. (2005). Spectrum agile radio: Capacity and QoS implications of dynamic spectrum assignment. Proceedings of IEEE Globecom’05, pp. 2510–2516.
Fangwen F., van der Schaar M. (2009) Learning to compete for resources in wireless stochastic games. IEEE Transactions on Vehicular Technology 58(4): 1904–1919
Niyato D., Hossain E., Han Z. (2009) Dynamics of multiple-seller and multiple-buyer spectrum trading in cognitive radio networks: A game-theoretic modeling approach. IEEE Transactions on Mobile Computing 8(8): 1009–1022
Roy, N., Roy, A., & Das, S. K. (2005). A cooperative learning framework for Mobility-aware resource management in multi-inhabitant smart momes. In Proceedings of IEEE MobiQuitous’05, pp. 393–403.
Jakkola T., Singh S.P. (1994) On the convergence of stochastic iterative dynamic programming algorithms. Neural Computation 6(6): 1185–1201
Venkatesh T., Kiran Y. V., Murthy C. S. R. (2009) Joint path and wavelength selection using Q-learning in Optical burst switching networks. In: Proceedings of IEEE Globecom’09 (Vol. 15, no 6, pp. 1–5).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Teng, Y., Yu, F.R., Han, K. et al. Reinforcement-Learning-Based Double Auction Design for Dynamic Spectrum Access in Cognitive Radio Networks. Wireless Pers Commun 69, 771–791 (2013). https://doi.org/10.1007/s11277-012-0611-9
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11277-012-0611-9