Skip to main content
Springer Nature Link
Log in

A Reinforcement-Learning Based Cognitive Scheme for Opportunistic Spectrum Access

  • Published:
Wireless Personal Communications Aims and scope Submit manuscript

Abstract

Cognitive Radio enables secondary users (SUs) to access communication channels allocated to primary users (PUs). As prior knowledge of the channel characteristics is not available in practice, the SUs attempting to get communication access in a geographical area must act autonomously and fast in order to detect vacant communication channels. Addressing the SUs need for autonomous operation, this article proposes a reinforcement learning scheme that determines the sensing order of the available channels employing two alternative update rules. Under both alternative options, the SUs operate as independent agents processing information acquired solely from their own sensing mechanisms in order to assess the channels with respect to (i) the occupancy probability and (ii) the mean duration of vacant periods. The scheme capability of accurately estimating the various channel characteristics without any prior knowledge of the traffic pattern followed by the PUs is thoroughly investigated with regard to critical performance metrics in both static and dynamic transmission environments. The proposed scheme is compared with two existing channel selection schemes. The simulations show that the proposed scheme manages to prioritize channel selection according to the channel characteristics and that it outperforms both schemes under comparison in terms of channel utilization and energy efficiency.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

References

  1. Mitola, J., & Maguire, G. Q. (1999). Cognitive radio: Making software radios more personal. IEEE Personal Communications, 6(4), 13–18.

    Article  Google Scholar 

  2. Akyildiz, I. F., Lee, W. Y., Vuran, M. C., & Mohanty, S. (2006). Next generation/dynamic spectrum access/cognitive radio wireless networks: A survey. Computer Networks Journal, 50(13), 2127–2159.

    Article  MATH  Google Scholar 

  3. Ycek, T., & Arslan, H. (2009). A survey of spectrum sensing algorithms for cognitive radio applications. IEEE Communications Surveys and Tutorials, 11(1), 116–130.

    Article  Google Scholar 

  4. Ghasemi, A., & Sousa, E. S. (2008). Spectrum sensing in cognitive radio networks: Requirements, challenges and design trade-offs. IEEE Communications Magazine, 46(4), 32–39.

    Article  Google Scholar 

  5. Tragos, E. Z., Zeadally, S., Fragkiadakis, A. G., & Siris, V. A. (2013). Spectrum assignment on cognitive radio networks: A comprehensive survey. IEEE Communications Surveys & Tutorials, 15(3).

  6. Haykin, S. (2005). Cognitive radio: Brain-empowered wireless communications. IEEE Journal on Selected Areas in Communications, 25, 201–220.

    Article  Google Scholar 

  7. Geirhofer, S., Tong, L., & Sandler, M. (2007). Dynamic spectrum access in the time domain: Modeling and exploiting white space. IEEE Communications Magazine, 45(5), 66–72.

    Article  Google Scholar 

  8. Xiukui, Li, & Reza Zekavat, Seyed A. (2009). Cognitive radio based spectrum sharing: Evaluating channel availability via traffic pattern prediction. Journal of Communications and Networks, 11(2), 104–114.

    Article  Google Scholar 

  9. Canberk, B., Akyildiz, I. F., & Oktug, S. (2000). Primary user activity modeling using first-difference filter clustering and correlation in cognitive radio networks. IEEE/ACM Transactions on Networking, 19(1).

  10. Yun, G., Grammenos, R. C., Yang, Y., & Wang, W. (2010). Performance analysis of selective opportunistic spectrum access with traffic prediction. IEEE Transactions on Vehicular Technology, 59(4).

  11. Huang, J., Zhou, H., Chen, Y., Chen, B., Zhu, X., & Kong, R. (2013). Optimal channel sensing order for various applications in cognitive radio networks. Wireless Personal Communications, 71(3), 1721–1740.

    Article  Google Scholar 

  12. Jiang, H., Lai, L., Fan, R., & Poor, H. V. (2009). Optimal selection of channel sensing order in cognitive radio. IEEE Transactions on Wireless Communications, 8(1), 297–307.

    Article  Google Scholar 

  13. Chang, N. B., & Liu, M. (2009). Optimal channel probing and transmission scheduling for opportunistic spectrum access. IEEE/ACM Transactions on Networking, 17(6), 1805–1818.

    Article  Google Scholar 

  14. Cheng, H. T., & Zhuang, W. (2011). Simple channel sensing order in cognitive radio networks. IEEE Journal on Selected Areas in Communications, 29(4), 676–688.

    Article  Google Scholar 

  15. Liu, C.-H., Tran, J. A., Pawelczak, P., & Cabric, D. (2013). Traffic-aware channel sensing order in dynamic spectrum access networks. IEEE Journal on Selected Areas in Communications, 31(11), 2312–2323.

    Article  Google Scholar 

  16. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning. Cambridge, MA: MIT Press.

    Google Scholar 

  17. Watkins, C. J. C. H., & Dayan, P. (1992). Technical note: Q-learning. Machine Learning, 8(3/4), 279–292.

    Article  MATH  Google Scholar 

  18. Kaelbling, L. P., Littman, M. L., & Moore, A. W. (1996). Reinforcement learning: A survey. Journal of Artificial Intelligence Research, 4, 237–285.

    Google Scholar 

  19. Singh, S., Jaakkola, T., Littman, M. L., & Szepesvri, C. (2000). Convergence results for single-step on-policy reinforcement-learning algorithms. Machine Learning, 38, 287–308.

    Article  MATH  Google Scholar 

  20. Liang, Y. C., Zeng, Y., Peh, E. C., & Hoang, A. T. (2008). Sensing throughput tradeoff in cognitive radio networks. IEEE Transactions on Wireless Communications, 7(4), 1326–1337.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical and Computer Engineering, National Technical University of Athens, 15780, Athens, Greece

    Angeliki V. Kordali & Panayotis G. Cottis

Authors
  1. Angeliki V. Kordali
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Panayotis G. Cottis
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Angeliki V. Kordali.

Additional information

This research has been co-financed by the European Union (European Social Fund ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF)—Research Funding Program: Heracleitus II. Investing in knowledge society through the European Social Fund.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kordali, A.V., Cottis, P.G. A Reinforcement-Learning Based Cognitive Scheme for Opportunistic Spectrum Access. Wireless Pers Commun 86, 751–769 (2016). https://doi.org/10.1007/s11277-015-2955-4

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11277-015-2955-4

Keywords