Skip to main content
Springer Nature Link
Log in

Adaptive maximum-lifetime routing in mobile ad-hoc networks using temporal difference reinforcement learning

  • Original Paper
  • Published:
Evolving Systems Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Mobile ad-hoc NETworks (MANETs) are very dynamic environments. A routing protocol for MANETs should be adaptive in order to operate correctly in presence of variable network conditions. Reinforcement learning (RL) is a recently used technique to achieve adaptive routing in MANETs. In comparison to other machine learning and computational intelligence techniques, RL achieves optimal results at low processing and medium memory costs. To deal with adaptive energy-aware routing issue in MANETs, a RL-based maximum-lifetime routing strategy is proposed. Each mobile node learns how to adjust its route-request packets forwarding-rate according to its energy profile. In terms of RL-resolution methods, Q-Learning, SARSA, Q(λ) and SARSA(λ) which are Temporal difference RL-algorithms are used. The proposed RL model is implemented on the top of AODV routing protocol. Simulation results show that the RL-based AODV achieved good performances in comparison to Time-Delay and Probability based AODV. Particularly, the Q-Learning based AODV has marked the best global performances in terms of energy efficiency and end to end delay.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  • Bekmezci I, Sahingoz OK, Temel S (2013) Flying ad-hoc networks (FANETs): a survey. Ad-Hoc Networks, http://dx.doi.org/10.1016/j.adhoc.2012.12.004

  • Binbin Z, Quan L, Shouling Z (2010) Using statistical network link model for routing in ad-hoc networks with multi-agent reinforcement learning. In: International Conference on Advanced Computer Control

  • Boukerche A (2009) Algorithms and protocols for wireless and mobile ad-hoc networks. John Wiley & Sons Inc, Ottawa, Canada

  • Boyan JA, Littman ML (1994) Packet routing in dynamically changing networks: a reinforcement learning approach. Adv Neural Inf Process Syst 6:671–678

    Google Scholar 

  • Can J-C, Manzoni P (2000) A performance comparison of energy consumption for mobile ad-hoc network routing protocols. In: Proceedings IEEE/ACM MASCOTS’ 2000

  • Caro G-D, Ducatelle F, Gambardella LM (2005) AntHocNet: an adaptive nature-inspired algorithm for routing in mobile ad-hoc networks. European Transactions on Telecommunications: special Issue on Self-organisation in Mobile Networking 16:443–455

    Google Scholar 

  • Chang Y-H, Ho T (2004) Mobilized ad-hoc networks: A reinforcement learning approach. In: ICAC Proceedings of the First International Conference on Autonomic Computing IEEE Computer Society

  • Chen S, Nahrstedt K (1999) Distributed quality-of-service routing in ad-hoc networks. IEEE J Sel Areas Commun 17(8):1488–1505

    Article  Google Scholar 

  • Chettibi S, Chikhi S (2011) A survey of Reinforcement Learning Based routing protocols for Mobile ad-hoc Networks. Recent Trends in Wireless and Mobile Networks, Communications in Computer and Information Science, LNCS, Springer, 162:1–13

  • Cho W, Kim SL (2002) A fully distributed routing algorithm for maximizing lifetime of a wireless ad-hoc network. In: Proceedings IEEE International Workshop-Mobile and Wireless Communication Network

  • Cho W, Kim D, Kim T, Kim SH (2011)Time Delay On-Demand Multipath routing protocol in mobile ad-hoc networks. In: Third International Conference on Ubiquitous and Future Networks

  • Dowling J, Curran E, Cunningham R, Cahill V (2005) Using feedback in collaborative reinforcement learning to adaptively optimize MANET routing. IEEE Transactions Systems Man, Cybern 35:360–72

    Google Scholar 

  • Ducatelle F, Di Caro G, Gambardella LM (2009) Principles and applications of swarm intelligence for adaptive routing in telecommunications networks. Swarm Intell 5:169–184

    Google Scholar 

  • Feeney LM (2001) An energy consumption model for performance analysis of routing protocols for mobile ad-hoc networks. Mobile Netw Appl 6(3):239–249

    Article  MATH  Google Scholar 

  • Forster A (2007) Machine learning techniques applied to wireless ad-hoc networks: guide and survey. In: Proceedings of 3rd International Conference on Intelligent Sensors, Sensor Networks and Information Processing

  • Friedman M (1937) The use of ranks to avoid the assumption of normality implicit in the analysis of variance. J Am Stat Assoc 32:674–701

    Article  Google Scholar 

  • Hedrick C (1988) Routing Information Protocol. Request for Comments 1058 http://www.ietf.org/rfc/rfc1058.txt. Accessed 04 February 2013

  • Jayashree S, Siva Ram C (2007) A taxonomy of energy management protocols for ad-hoc wireless networks. IEEE Communications Magazine

  • Kalyanakrishnan S, Stone P (2011) Characterizing reinforcement learning methods through parameterized learning problems. J Mach Learn 84:205–247

    Article  MathSciNet  Google Scholar 

  • Kim D, Garcia-Luna-Aceves JJ, Obraczka K, Cano JC, Manzoni P (2002) Power-aware routing based on the energy drain rate for mobile ad-hoc networks. In: 11th International Conference on Computer Communications and Networks

  • Kulkarni R, Forster A, Venayagamoorthy G (2011) Computational intelligence in wireless sensor networks: a survey. IEEE Commun Surv Tutor 13(1):68–96

    Article  Google Scholar 

  • Li F, Wang Y (2007) Routing in vehicular ad-hoc networks: a survey. Veh Technol Mag 2:12–22

    Article  Google Scholar 

  • Li J, Cordes D, Zhang J (2005) Power-aware routing protocols in ad-hoc wireless networks. IEEE Transactions on Wireless Communications pp 69–81

  • Liu C, Kaiser J (2003) A survey of mobile ad-hoc network routing protocols. Technical Report, University of Ulm, Germany

  • Macone D, Oddi G, Pietrabissa A (2012) MQ-Routing: mobility-, GPS- and energy-aware routing protocol in MANETs for disaster relief scenarios. J Ad Hoc Netw 10:861–878

    Google Scholar 

  • Maneenil K, Usaha W (2005) Preventing malicious nodes in ad-hoc networks using reinforcement learning. In: The 2nd International Symposium on Wireless Communication Systems

  • MATLAB (2013) http://www.mathworks.com. Accessed 05 July 2013

  • McQuillan JM, Richer I, Rosen EC (1980) The new routing algorithm for the ARPANet. IEEE Trans Commun 28(5):711–719

    Article  Google Scholar 

  • Meghanathan N (2008) Exploring the stability-energy consumption-delay-network lifetime tradeoff of mobile ad-hoc network routing protocols. J Netw 3(2):17–28

    Google Scholar 

  • Naruephiphat W, Usaha W (2008) Balancing tradeoffs for energy-efficient routing in MANETs based on reinforcement learning. In: The IEEE 67th Vehicular Technology Conference

  • Natsheh E (2008) A survey on fuzzy reasoning applications for routing protocols in wireless ad-hoc networks. IJBDCN 4(2):22–37

    Google Scholar 

  • Nurmi P (2007) Reinforcement Learning for Routing in Ad-Hoc Networks. In: Proceedings 5th International Symposium on Modeling and Optimization in Mobile, Ad-Hoc, and Wireless Networks. IEEE Computer Society

  • Perkins CE, Royer EM (1999) Ad-hoc on-demand distance vector routing. In: Proceeding of WMCSA’ 99:90–100

  • Perkins CE, Watson TJ (1994) Highly dynamic destination sequenced distance vector routing (DSDV) for mobile computers. ACM SIGCOMM Conference on Communications Architectures

  • Petrowski A, Aissanou F, Benyahia I, Houcke S (2010) Multi-criteria reinforcement learning based on a Russian Doll method for network routing. In: 5th IEEE International Conference on Intelligent Systems

  • Senouci MS, Pujolle G (2004) Energy efficient routing in wireless ad-hoc networks. In: IEEE International Conference on Communications

  • Sutton R, Barto A (1998) Reinforcement Learning. MIT Press, Cambridge

  • Tao T, Tagashira S, Fujita S (2005) LQ-routing protocol for mobile ad-hoc networks. In: Proceedings of the Fourth Annual ACIS International Conference on Computer and Information Science

  • Tavli B, Heinzelman W (2006) Mobile ad-hoc networks: energy-efficient real-time data communications. Netherlands, Springer, ISBN-13 978-1-4020-4633-9

  • The network simulator, NS version 2 (2013) http://www.isi.edu/nsnam/ns/. Accessed 04 February 2013

  • Usaha W, Barria JA (2004) A reinforcement learning ticket-based probing path discovery scheme for MANETs. Ad Hoc Netw J 2:319–334

    Article  Google Scholar 

  • Vassileva N, B-Arroyo F (2008) A survey of routing protocols for maximizing the lifetime of ad-hoc wireless networks. Int J Softw Eng Appl 2(3):77–90

    Google Scholar 

  • Wang X, Li L, Ran C (2004) An energy-aware probability routing in MANETS. In: IEEE Workshop on IP Operations and Management

  • Woo M, Singh S, Raghavendra C (1998) Power-aware routing in mobile ad-hoc networks. In: Proceedings of International Conference on Mobile Computing and Networking

  • Woo K, Yu C, Youn HY, Lee B (2001) Non-blocking, localized routing algorithm for balanced energy consumption in mobile ad-hoc networks. International Symposium MASCOTS’ 2001

  • Wu C, Kumekawa K, Kato T et al (2010) Distributed reinforcement learning approach for vehicular ad-hoc networks. IEEE Trans Commun E93-B(6):1431–1442

    Article  Google Scholar 

  • Xiang J, Sesay S, Wang Y, He J (2007) Ad-hoc network state aware routing protocol. In: IEEE Wireless Communications and Networking Conf

  • Yau K-LA, Komisarczuk P, Teal PD (2012) Reinforcement learning for context awareness and intelligence in wireless networks: review, new features and open issues. J Netw Comput Appl 35:253–267

    Article  Google Scholar 

  • Yu C, Lee B, Youn H (2003) Energy efficient routing protocols for mobile ad-hoc networks. J Wirel Commun Mob Comput 3(8):959–973

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. SCAL TEAM, MISC Laboratory, University of Constantine 2, Constantine, Algeria

    Saloua Chettibi & Salim Chikhi

Authors
  1. Saloua Chettibi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Salim Chikhi
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Saloua Chettibi.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chettibi, S., Chikhi, S. Adaptive maximum-lifetime routing in mobile ad-hoc networks using temporal difference reinforcement learning. Evolving Systems 5, 89–108 (2014). https://doi.org/10.1007/s12530-013-9093-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12530-013-9093-6

Keywords