Skip to main content
SpringerLink
Log in

Multi-Agent Reinforcement Learning Based Opportunistic Routing and Channel Assignment for Mobile Cognitive Radio Ad Hoc Network

  • Published:
Mobile Networks and Applications Aims and scope Submit manuscript

Abstract

Opportunistic spectrum access using cognitive radio technology enables exploring vacant licensed spectrum bands and thereby improving the spectrum utilization. However, it will have a significant impact on upper layer performance like routing as the reliable knowledge of topology and channel statistics are not available, especially in Mobile Cognitive Radio Ad hoc Network (MCRAN). To address specific requirements of MCRAN, this paper is proposing online opportunistic routing algorithm using multi-agent reinforcement learning. The proposed routing scheme jointly addresses, link and relay selection based on transmission success probabilities. This sophisticated learning mechanism successfully explores opportunities in partially observable and non-stationary environment of MCRAN. Simulation results show the effectiveness of this algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Akyildiz IF et al (2006) Next generation/dynamic spectrum access/cognitive radio wireless networks: a survey. Elsevier Int J Comput Netw 50:2127–2159

    Article  MATH  Google Scholar 

  2. Sengupta S, Subbalakshmi KP (2013) Open research issues in multi-hop cognitive radio networks. IEEE Commun Mag 51(4):168–176

    Article  Google Scholar 

  3. Pan M et al (2012) Spectrum harvesting and sharing in multi-hop CRNs under uncertain spectrum supply. IEEE J Sel Areas Commun 30(2):369–378

    Article  Google Scholar 

  4. Thomas Hou Y et al (2008) Spectrum sharing for multi-hop networking with cognitive radios. IEEE J Sel Areas Commun 26(1):146–155

    Article  Google Scholar 

  5. Huisheng M et al. (2008) Spectrum aware routing for multi-hop cognitive radio networks with a single transceiver. 3rd International Conference on Cognitive Radio Oriented Wireless Networks and Communications CrownCom 2008. 1-6, doi:10.1109/CROWNCOM.2008.4562509

  6. Badoi CI et al. (2010) IPSAG: an IP spectrum aware geographic routing algorithm proposal for multi-hop cognitive radio networks. 8th International Conference on Communications (COMM), 491–496. doi: 10.1109/ICCOMM.2010.5509020

  7. Zeeshan M et al. (2010) Backup channel and cooperative channel switching on-demand routing protocol for multi-hop cognitive radio ad hoc networks (BCCCS). 6th International Conference on Emerging Technologies (ICET 2010), 394–399

  8. Cesana M et al (2011) Routing in cognitive radio networks: challenges and solutions. Elsevier Int J Ad Hoc Netw 9(3):228–248

    Article  Google Scholar 

  9. Lee J-J, Lim J (2014) Cognitive routing for multi-hop mobile cognitive radio ad hoc networks. J Commun Netw 16(2):155–161

    Article  Google Scholar 

  10. Royer EM, Toh CK (1999) A review of current routing protocols for ad hoc mobile wireless networks. IEEE pers Commun 6(2):46–55

    Article  Google Scholar 

  11. How KC et al (2011) Routing and QoS provisioning in cognitive radio networks. Elsevier Int J Comput Netw 55(1):330–342

    Article  Google Scholar 

  12. Di Felice M et al (2010) End-to-end protocol for cognitive radio ad hoc networks: an evaluation study. Int J Perform Eval 68(9):859–875

    Article  Google Scholar 

  13. Huang X-L et al (2011) Stability-capacity-adaptive routing for high-mobility multihop cognitive radio networks. IEEE Trans Veh Technol 60(6):2714–2729

    Article  Google Scholar 

  14. Cacciapuoti AS et al (2012) Reactive routing for mobile cognitive radio ad hoc networks. Elsevier J Ad Hoc Netw 10:803–815

    Article  Google Scholar 

  15. Liu Y et al (2012) Spectrum-aware opportunistic routing in multi-hop cognitive radio networks. IEEE J Sel Areas Commun 30(10):1958–1968

    Article  Google Scholar 

  16. Ding L et al (2010) Cross-layer routing and dynamic spectrum allocation in cognitive radio ad hoc network. IEEE Trans Veh Technol 59(4):1969–1979

    Article  Google Scholar 

  17. Chunsheng X et al. (2005) A novel layered graph model for topology formation and routing in dynamic spectrum access networks. First IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks DySPAN,308–317 doi:10.1109/ DYSPAN. 2005.1542647

  18. Guan Q et al (2010) Prediction-based topology control and routing in cognitive radio mobile ad hoc networks. IEEE Trans Veh Technol 59(9):4443–4452

    Article  Google Scholar 

  19. Chowdhury KR, Akyildiz IF (2011) CRP: a routing protocol for cognitive radio ad hoc networks. IEEE J Sel Areas Commun 29(4):794–804

    Article  Google Scholar 

  20. Bhorkar AA et al (2012) Adaptive opportunistic routing for wireless ad hoc networks. IEEE/ACM Trans Netw 20(1):243–256

    Article  Google Scholar 

  21. Xia B et al. (2009) Reinforcement learning based spectrum-aware routing in multi-hop cognitive radio networks. 4th IEEE International conference on Cognitive Radio Oriented Wireless Networks and Communications CROWNCOM 1–5

  22. Wu C et al (2010) Spectrum management of cognitive radio using multi-agent reinforcement learning. 9th Int. Conf. on Autonomous Agents and Multiagent Systems (AAMAS 2010), Toronto

    Google Scholar 

  23. Abedi O, Berangi R (2013) Mobility assisted spectrum aware routing protocol for cognitive radio ad hoc networks. Journal of Zhejiang University-SCIENCE C (Computers & Electronics) 14(11): 873–886 ISSN 1869–1951; ISSN 1869-196X (Online)www.zju.edu.cn/jzus; www.springerlink.com

  24. Oo MZ, Othman M (2012) Analytical studies of interaction between mobility models and single-multi paths routing protocols in mobile ad hoc networks. Springer Wirel Personal Commun 64(2):379–402

    Article  Google Scholar 

  25. Ariyakhajorn J et al. (2006) A comparative study of random waypoint and gauss-markov mobility models in the performance evaluation of MANET. Int. Symposium on Communications and Information Technologies ISCIT pp. 894,899 Oct. 18–20

  26. Liu B-H et al (2011) Message-efficient locatiodiction for mobile objects in wireless sensor networks using a maximum likelihood technique. IEEE Trans Comput 60(6):865–878

    Article  MathSciNet  Google Scholar 

  27. Sharma M et al. (2007) Channel selection under interference temperature model in multi-hop cognitive mesh networks. 2nd IEEE International Symposium on New Frontiers in Dynamic Spectrum Access Networks DySPAN doi: 10.1109/DYSPAN.2007.25

  28. Wei-bing LIU, Xian-jia WANG (2009) Dynamic decision model in evolutionary games based on reinforcement learning. Int J Syst Eng-Theory Pract 2(3):28–33

    Google Scholar 

  29. Duana Y et al (2007) Application of reinforcement learning in robot soccer. Eng Appl Artif Intell 20:936–950

    Article  Google Scholar 

  30. Huang Z et al (2011) Reinforcement learning based resource allocation in business process management. Int J Data Knowl Eng 70:127–145

    Article  Google Scholar 

  31. Usaha W, Barria JA (2007) Reinforcement learning for resource allocation in LEO satellite networks. IEEE Trans Syst, Man Cybern-Part B: Cybern 37(3):515–527

    Article  Google Scholar 

  32. EI-Alfy EI-S et al (2006) A learning approach for prioritized handoff channel allocation in mobile multimedia network. IEEE Trans Wirel Commun 5(7):1651–1660

    Article  Google Scholar 

  33. Nie J, Haykin S (1999) A dynamic channel assignment policy through Q-learning. IEEE Trans Neural Netw 10(6):1443–1455

    Article  Google Scholar 

  34. Kaebling LP et al (1996) Reinforcement learning: a survey. J Artif Intell Res 4:237–285

    Google Scholar 

  35. Sharma R, Gopal M (2010) Synergizing reinforcement learning and game theory-a new direction for control. Elsevier Int J Appl Soft Comput 10:675–688

    Article  Google Scholar 

  36. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge

    Google Scholar 

Download references

Acknowledgments

This work is partially supported by the Research Promotion Scheme of All India Council for Technical Education, India, Grant No. 20/AICTE/RIFD/RPS(POLICY-III)78/2013-14. The author would like to thank the reviewers for their helpful and constructive comments.

Author information

Authors and Affiliations

  1. Bharati Vidyapeeth Deemed University College of Engineering, Pune, Maharashtra, India

    Sunita S. Barve

  2. Anomaly Solutions Pvt. Ltd., Pune, India

    Parag Kulkarni

Authors
  1. Sunita S. Barve
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Parag Kulkarni
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sunita S. Barve.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Barve, S.S., Kulkarni, P. Multi-Agent Reinforcement Learning Based Opportunistic Routing and Channel Assignment for Mobile Cognitive Radio Ad Hoc Network. Mobile Netw Appl 19, 720–730 (2014). https://doi.org/10.1007/s11036-014-0551-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11036-014-0551-6

Keywords

Search

Navigation