Skip to main content
SpringerLink
Log in

A Survey and Performance Evaluation of Reinforcement Learning Based Spectrum Aware Routing in Cognitive Radio Ad Hoc Networks

  • Published:
International Journal of Wireless Information Networks Aims and scope Submit manuscript

Abstract

Cognitive radio technology is an assuring solution for under-utilization of licensed spectrum bands and overcrowding of unlicensed spectrum bands, in which secondary user is permitted to access the primary users’ spectrum in an opportunistic manner. Opportunistic access of the spectrum requires complex changes across all the layers of a network protocol stack. Cognitive radio has to be an autonomous agent in order to configure itself to dynamic spectrum environment. And, the characteristics of reinforcement learning, a subfield of artificial intelligence in which the agent learns the surrounding operating environment through continuous interaction and takes an optimum decision on the fly, is in compliance with features of self-organized cognitive radio ad hoc network. Therefore, reinforcement learning is an appropriate option for incorporating intelligence and self-adaptivity into cognitive radio. This paper provides a comprehensive survey on the application of reinforcement learning for efficient spectrum aware routing in cognitive radio ad hoc network. The preliminaries of cognitive radio ad hoc networks and reinforcement learning are first introduced, and a review is investigated in the proposed research area along with a discussion on open research challenges with an aim to promote research. From the survey, reinforcement learning incorporated cognitive radio can learn the unknown primary user network model and the learned model can be then used for finding a suitable route to meet the Quality of Service requirements. With this in mind, the paper also proposes a multi-objective reinforcement learning based spectrum aware routing protocol with an aim to maximize the probability of successful transmission using a minimum hop path. The simulated results prove the performance of the algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. J. Mitola. Cognitive radio for flexible mobile multimedia communications. In Proceedings of International Workshop on Mobile Multimedia Communications, San Diego, CA, 1999.

  2. Spectrum policy task force report. Federal Communication Commission, 2002.

  3. J. Marinho and E. Monteiro, Cognitive radio: survey on communication protocols, spectrum decision issues, and future research directions, Wireless Networks, Vol. 18, No. 2, pp. 147–164, 2012.

    Google Scholar 

  4. I. F. Akyildiz, B. F. Lo and R. Balakrishnan, Cooperative spectrum sensing in cognitive radio networks: a survey, Physical Communication, Vol. 4, No. 1, pp. 40–62, 2011.

    Google Scholar 

  5. I. F. Akyildiz, W. Lee and K. R. Chowdhury, CRAHNs: cognitive radio ad hoc networks, Ad Hoc Networks, Vol. 7, No. 5, pp. 810–836, 2009.

    Google Scholar 

  6. K. Patil, R. Prasad, and K. Skouby. A survey of worldwide spectrum occupancy measurement campaign for cognitive radio. In Proceedings of International Conference on Devices and Communications (ICDeCom), India, pp. 1–5, Feb. 2011.

  7. J. Xue, Z. Feng, and P. Zhang (2013) Spectrum occupancy measurements and analysis in beijing. IERI Procedia, 4:295–302. In 2013 International Conference on Electronic Engineering and Computer Science (EECS 2013).

    Google Scholar 

  8. S. D. Barnes, P. A. Jansen van Vuuren and B. T. Maharaj, Spectrum occupancy investigation: measurements in south africa, Measurement, Vol. 46, No. 9, pp. 3098–3112, 2013.

    Google Scholar 

  9. S. D. Barnes, P. R. Botha and B. T. Maharaj, Spectral occupation of TV broadcast bands: measurement and analysis, Measurement, Vol. 93, No. Supplement C, pp. 272–277, 2016.

    Google Scholar 

  10. D. Das and S. Das, A survey on spectrum occupancy measurement for cognitive radio, Wireless Personal Communications, Vol. 85, No. 4, pp. 2581–2598, 2015.

    Google Scholar 

  11. A. Ali and W. Hamouda, Advances on spectrum sesning for cognitive radio networks: theory and applications, IEEE Communication Surveys and Tutorials, Vol. 19, No. 2, pp. 1277–1304, 2017.

    Google Scholar 

  12. G. I. Tsiropoulos, O. A. Dobre, M. H. Ahmed and K. F. Baddour, Radio resource allocation techniques for efficient spectrum access in cognitive radio network, IEEE Communication Surveys and Tutorials, Vol. 18, No. 1, pp. 824–847, 2016.

    Google Scholar 

  13. K. Kumar, A. Prakash and R. Trpathi, Spectrum handoff in cognitive radio networks: a classification and comprehensive survey, Journal of Network and Computer Applications, Vol. 61, pp. 161–188, 2016.

    Google Scholar 

  14. M. Youssef, M. Ibrahim, M. Abdelatif, L. Chen and A. V. Vasilakos, Routing metrics of cognitive radio networks: a survey, IEEE Communications Surveys and Tutorials, Vol. 16, No. 1, pp. 92–109, 2014.

    Google Scholar 

  15. K. Singh and S. Moh, Routing protocols in cognitive radio ad hoc networks: a comprehensive review, Journal of Network and Computer Applications, Vol. 72, pp. 28–37, 2016.

    Google Scholar 

  16. Y. Saleem and M. Rehmani, Primary radio user activity models for cognitive radio networks: a survey, Journal of Network and Computer Applications, Vol. 43, pp. 1–16, 2014.

    Google Scholar 

  17. R. K. Sharma and D. B. Rawat, Advances on security threats and countermeasures for cognitive radio networks: a survey, IEEE Communication Surveys and Tutorials, Vol. 17, No. 2, pp. 1023–1043, 2015.

    Google Scholar 

  18. Y. Liang, K. Chen, G. Y. Li and P. Mahonen, Cognitive radio networking and communications: an overview, IEEE Transactions on Vehicular Technology, Vol. 60, No. 7, pp. 3386–3407, 2011.

    Google Scholar 

  19. J. Wang, M. Ghosh and K. Challapali, Emerging cognitive radio applications: a survey, IEEE Communications Magazine, Vol. 49, No. 3, pp. 74–81, 2011.

    Google Scholar 

  20. P. C. Jain. Rural wireless broad band internet access in wireless regional area network using cognitive radio. In Proceedings of International conference on signal processing and communication, Noida, India, 2013.

  21. K. Katzis and H. Ahmadi, Challenges Implementing Internet of Things (IoT) Using Cognitive Radio Capabilities in 5G Mobile Networks, Springer International Publishing, Cham, 2016. pp. 55–76.

    Google Scholar 

  22. A. Athar, M. H. Rehmani and A. Rachedi, Cognitive-radio-based Internet of Things: applications, architectures, spectrum related functionalities, and future research directions, IEEE Wireless Communications, Vol. 24, No. 3, pp. 17–25, 2017.

    Google Scholar 

  23. S. Sengupta and K. P. Subbalakshmi, Open research issues in multi-hop cognitive radio networks, IEEE Communications Magazine, Vol. 51, No. 4, pp. 168–176, 2013.

    Google Scholar 

  24. T. Yucek and H. Arslan, A survey of spectrum sensing algorithms for cognitive radio applications, IEEE Communications Surveys and Tutorials, Vol. 11, No. 1, pp. 116–130, 2009.

    Google Scholar 

  25. S. Russell and P. Norvig, Artificial Intelligence: A Modern Approach3rd ed., Prentice Hall Press, Upper Saddle River, 2009.

    MATH  Google Scholar 

  26. M. Bkassiny, Y. Li and S. K. Jayaweera, A survey on machine learning techniques in cognitive radios, IEEE Communications Surveys and Tutorials, Vol. 15, No. 3, pp. 1136–1159, 2013.

    Google Scholar 

  27. J. Qadir, Artificial intelligence based cognitive routing for cognitive radio networks, Artificial Intelligence Review, Vol. 45, No. 1, pp. 25–96, 2016.

    Google Scholar 

  28. I. Macaluso, D. Finn, B. Ozgul and L. A. DaSilva, Complexity of spectrum activity, IEEE Journal on Selected Areas in Communications, Vol. 31, No. 11, pp. 2237–2248, 2013.

    Google Scholar 

  29. Richard S. Sutton and Andrew G. Barto, Introduction to Reinforcement Learning, vol. 1st, MIT Press, Cambridge, 1998.

    MATH  Google Scholar 

  30. S. J. Gershman and N. D. Daw, Reinforcement learning and episodic memory in humans and animals: an integrative framework, Annual Review on Psychology, Vol. 68, pp. 101–128, 2017.

    Google Scholar 

  31. W. Wang, A. Kwasinski, D. Niyato and Z. Han, A survey on model-free strategy learning in cognitive wireless network, IEEE Communications Surveys and Tutorials, Vol. 18, No. 3, pp. 1717–1757, 2016.

    Google Scholar 

  32. H. A. A. Al-Rawi, M. A. Ng and K. A. Yau, Application of reinforcement learning to routing in distributed wireless networks: a review, Artificial Intelligence Review, Vol. 48, No. 3, pp. 381–416, 2015.

    Google Scholar 

  33. O. Sigaud and O. Buffer, Markov Decision Processes in Artificial Intelligence, Wiley, Hoboken, 2012.

    Google Scholar 

  34. D. M. Roijers, P. Vamplew, S. Whilson and R. Dazeley, A survey of multi-objective sequential decision making, Journal of Artificial Intelligence Research, Vol. 48, pp. 67–113, 2013.

    MathSciNet  MATH  Google Scholar 

  35. W. Wang. Multi-objective sequential decision making. Ph. D Thesis, 2014.

  36. H. Li, Adaptive Learning in Cognitive Radio, Springer, Singapore, 2019. pp. 1083–1121.

    Google Scholar 

  37. B. Xia, M. H. Wahab, Y. Yang, Z. Fan, and M. Sooriyabandara. Reinforcement learning based spectrum aware routing in multi hop cognitive radio networks. In Proceedings of Fourth International Conference on CROWNCOM, Hanover, Germany, 2009.

  38. M. D. Felice, K. R. Chowdhury, A. Kassler, W. Kim and L. Bononi, End-to-end protocols for cognitive radio ad hoc networks: an evaluation study, Performance Evaluation, Vol. 68, No. 9, pp. 859–875, 2011.

    Google Scholar 

  39. C. Wu, K. Chowdhury, and M. Di Felice. Spectrum management of cognitive radio using multi-agent reinforcement learning. In Proceedings of 9th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2010), Toronto, Canada, pp. 1705–1712, May 2010.

  40. S. Shamaee, M. E. Shiri, M. Ebrahim and M. Sabaei, A reinforcement learning based routing in cognitive radio networks for primary users with multi-stage periodicity, Wireless Personal Communications, Vol. 101, No. 1, pp. 465–490, 2018.

    Google Scholar 

  41. Z. Yin, Y. Wang and C. Wu, Reinforcement learning spectrum management paradigm in cognitive radio using novel state and action sets, Procedia Computer Science, Vol. 129, pp. 433–437, 2018.

    Google Scholar 

  42. C . R. Stevenson, G. Chouinard, Z. Lei, W. Hu, S. J. Shellhammer and W. Caldwell, IEEE 802.22: the first cognitive radio wireless regional area network standard, Communications Magazine, Vol. 47, No. 1, pp. 130–138, 2009.

    Google Scholar 

  43. H. A. Al-Rawi and L. L. A. Yau. Route selection for minimizing interference to primary users in cognitive radio networks: a reinforcement learning approach. In Proceedings of Symposium on Computational Intelligence for Communication Systems and Networks, 2013.

  44. H. A. A. Al-Rawi, K. A. Yau, H. Mphammad, N. Ramli and W. Hashim, Reinforcement learning for routing in cognitive radio ad hoc networks, The Scientific World Journal, Vol. 2014, p. 22, 2014.

    Google Scholar 

  45. B. Pourpeighambar, M. Deghan and M. Sabaei, Multi-agent learning based routing for delay minimization in cognitive radio networks, Journal of Network and Computer Appilcations, Vol. 84, pp. 82–92, 2017.

    Google Scholar 

  46. B. Pourpeighambar, M. Dehghan and M. Sabaei, Non-cooperative reinforcement learning based routing in cognitive radio networks, Computer Communications, Vol. 106, pp. 11–23, 2017.

    Google Scholar 

  47. S. S. Barve and P. Kulkarni, Multi agent reinforcement learning based opportunistic routing in cognitive radio sensor network, Mobile Network Application, Vol. 19, No. 6, pp. 720–730, 2014.

    Google Scholar 

  48. A. C. Talay and D. T. Altilar, Self adaptive routing for dynamic spectrum access in cognitive radio networks, Journal of Network and Computer Applications, Vol. 36, No. 4, pp. 1140–1151, 2013.

    Google Scholar 

  49. Y. Saleem, K. A. Yau, H. Mohamad, N. Ramli, M. H. Rahmani and Q. Ni, Clustering and reinforcement-learning-based routing for cognitive radio networks, IEEE Wireless Communications, Vol. 24, No. 4, pp. 146–151, 2017.

    Google Scholar 

  50. M. Musavi, K. Yau, A. R. Syed, H. Mohamad and N. B. Ramli, Route selection over clustered cognitive radio networks: an experimental evaluation, Computer Communications, Vol. 129, pp. 138–151, 2018.

    Google Scholar 

  51. K. Zheng, H. Li, R. C. Qiu, and S. Gong. Multi objective reinforcement learning based routing in cognitive radio networks: Walking in a random maze. In Proceedings of International Conference on Computing, Networking and Communications, Maui, USA, 2012.

  52. C. Messikh and N. Zarour. Towards a multi-objective reinforcement learning based routing protocol for cognitive radio networks. In 2018 International Conference on Smart Communications in Network Technologies (SaCoNeT), pp. 84–89, Oct 2018.

  53. B. F. Lo, A survey of common control channel design in cognitive radio networks, Physical Communication, Vol. 4, pp. 26–39, 2011.

    Google Scholar 

  54. H. A. A. Al-Rawi, K. A. Yau, H. Mohamad, N. Ramli, and W. Hashim. A reinforcement learning based routing scheme for cognitive radio ad hoc networks. In Proceedings of Seventh International Conference on Wireless and Mobile Networking Conference, Vilamoura, 2014.

  55. C. Liu, X. Xu and D. Hu, Multiobjective reinforcement learning: a comprehensive overview, IEEE Transactions on Systems Man and Cybernetics Systems, Vol. 45, No. 3, pp. 385–398, 2015.

    Google Scholar 

  56. Y. Wang, G. Zheng, H. Ma, Y. Li and J. Li, A joint channel selection and routing protocol for cognitive radio network, Wireless Communications and Mobile Computing, Vol. 2018, p. 7, 2018.

    Google Scholar 

  57. A. R. Syed, K. A. Yau, J. Qadir, H. Mohamad, N. B. Ramli and S. L. Keoh, Route selection for multi hop cognitive radio network using reinforcement learning: an experimental study, IEEE Access, Vol. 4, pp. 6304–6324, 2016.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Information and Communication Technology, Manipal Institute of Technology, Manipal Academy of Higher Education (MAHE), Manipal, 576104, India

    Rashmi Naveen Raj

  2. Department of Computer and Science Engineering, Manipal Institute of Technology, Manipal Academy of Higher Education (MAHE), Manipal, 576104, India

    Ashalatha Nayak

  3. Department of Electronics and Communication, Manipal Institute of Technology, Manipal Academy of Higher Education (MAHE), Manipal, 576104, India

    M. Sathish Kumar

Authors
  1. Rashmi Naveen Raj
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ashalatha Nayak
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. M. Sathish Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rashmi Naveen Raj.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Naveen Raj, R., Nayak, A. & Kumar, M.S. A Survey and Performance Evaluation of Reinforcement Learning Based Spectrum Aware Routing in Cognitive Radio Ad Hoc Networks. Int J Wireless Inf Networks 27, 144–163 (2020). https://doi.org/10.1007/s10776-019-00463-6

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10776-019-00463-6

Keywords

Search

Navigation