Skip to main content
SpringerLink
Log in

Distributed resource management in wireless sensor networks using reinforcement learning

  • Published:
Wireless Networks Aims and scope Submit manuscript

Abstract

In wireless sensor networks (WSNs), resource-constrained nodes are expected to operate in highly dynamic and often unattended environments. Hence, support for intelligent, autonomous, adaptive and distributed resource management is an essential ingredient of a middleware solution for developing scalable and dynamic WSN applications. In this article, we present a resource management framework based on a two-tier reinforcement learning scheme to enable autonomous self-learning and adaptive applications with inherent support for efficient resource management. Our design goal is to build a system with a bottom-up approach where each sensor node is responsible for its resource allocation and task selection. The first learning tier (micro-learning) allows individual sensor nodes to self-schedule their tasks by using only local information, thus enabling a timely adaptation. The second learning tier (macro-learning) governs the micro-learners by tuning their operating parameters so as to guide the system towards a global application-specific optimization goal (e.g., maximizing the network lifetime). The effectiveness of our framework is exemplified by means of a target tracking application built on top of it. Finally, the performance of our scheme is compared against other existing approaches by simulation. We show that our two-tier reinforcement learning scheme is significantly more efficient than traditional approaches to resource management while fulfilling the application requirements.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Notes

  1. In order to reduce the related overhead, we assume that the rewards are piggybacked into the messages.

References

  1. Alex, H., Kumar, M., & Shirazi, B. (2008). Midfusion: An adaptive middleware for information fusion in sensor network applications. Information Fusion, 9(3), 332–343. doi:10.1016/j.inffus.2005.05.007.

  2. Byers, J. W., & Nasser, G. (2000). Utility-based decision-making in wireless sensor networks. In: MobiHoc (pp. 143–144). ACM. doi:10.1145/514151.514178.

  3. Di Francesco, M., Das, S. K., & Anastasi, G. (2011). Data collection in wireless sensor networks with mobile elements: A survey. ACM Transactions on Sensor Networks 8(1). doi:10.1145/1993042.1993049.

  4. Frank, C., & Römer, K. (2005). Algorithms for generic role assignment in wireless sensor networks. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.59.8091; http://www.vs.inf.ethz.ch/publ/papers/sensys05.roleassignment.ps.

  5. Hadim, S., & Mohamed, N. (2006). Middleware challenges and approaches for wireless sensor networks. IEEE Distributed Systems Online 7(3), Article no. 0603–o3001.

    Google Scholar 

  6. Heinzelman, W., Murphy, A., Carvalho, H., & Perillo, M. (2004). Middleware to support sensor network applications. IEEE Network 18,(1), 6–14.

    Article  Google Scholar 

  7. Intanagonwiwat, C., Govindan, R., & Estrin, D. (2000). Directed diffusion: A scalable and robust communication paradigm for sensor networks. In MOBICOM (pp. 56–67). doi:10.1145/345910.345920.

  8. J-sim: Component-based, compositional simulation environment. http://sites.google.com/site/jsimofficial/.

  9. Kagan Tumer, D. H. W. (2004). Collectives and the design of complex systems. Berlin: Springer.

    Book  MATH  Google Scholar 

  10. Kogekar, S., Neema, S., Eames, B., Koutsoukos, X., Ledeczi, A., & Maroti, M. (2004). Constraint-guided dynamic reconfiguration in sensor networks. In: Proceedings of the 3rd international symposium on information processing in sensor networks (IPSN-04) (pp. 379–387). New York: ACM Press.

  11. Krishnamachari, B., Wicker, S. B., Béjar, R., & Fernández, C. (2003) On the complexity of distributed self-configuration in wireless networks. Telecommunication Systems 22(1–4), 33–59. doi:10.1023/A:1023426501170.

    Google Scholar 

  12. Leen-Kiat Soh Costa Tsatsoulis, H. S. (2003). Distributed sensor networks: A multiagent perspective, chap. In A satisfying, negotiated and learning coalition formation architecture (pp. 109–137). Kluwer.

  13. Liu, T., & Martonosi, M. (2003). Impala: A middleware system for managing autonomic, parallel sensor systems. In: PPoPP’03: Proceedings of the 9th ACM SIGPLAN symposium on principles and practice of parallel programming (pp. 107–118).

  14. Mahadevan, S., & Conell, J. (1992). Automatic programming of behavior-based robots using reinforcement learning. Artificial Intelligence 55(2), 311–365.

    Article  Google Scholar 

  15. Mainland, G., Parkes, D. C., & Welsh, M. (2005). Decentralized, adaptive resource allocation for sensor networks. In NSDI’05: Proceedings of the 2nd symposium on networked systems design & implementation (pp. 315–328).

  16. Marron, P. J., Lachenmann, A., Minder, D., Hahner, J., Sauter, R., & Rothermel, K. (2005). TinyCubus: A flexible and adaptive framework sensor networks. In: EWSN’05: Proceedings of the 2nd European workshop on wireless sensor networks (pp. 278–289).

  17. Modi, P. J., Shen, W., Tambe, M., & Yokoo, M. (2005). Adopt: Asynchronous distributed constraint optimization with quality guarantees. Artificial Intelligence, 161(1–2), 149–180.

    Article  MathSciNet  MATH  Google Scholar 

  18. Ortiz, C., Rauenbush, T., Hsu, E., & Vincent, R. (2003). Distributed sensor networks: A multiagent perspective. In Dynamic resource-bounded negotiation in non-additive domains (pp. 61–106). Kluwer.

  19. Sadagopan, N., Singh, M., & Krishnamachari, B. (2006). Decentralized utility-based sensor network design. MONET 11(3), 341–350. doi:10.1007/s11036-006-5187-8.

    Google Scholar 

  20. Shah, K., Di Francesco, M., Anastasi, G., & Kumar, M. (2011). A framework for resource-aware data accumulation in sparse wireless sensor networks. Computer Communications 34(17), 2094–2103. doi:10.1016/j.comcom.2011.06.010.

    Article  Google Scholar 

  21. Shah, K., & Kumar, M. (2007). Distributed independent reinforcement learning (DIRL) approach to resource management in wireless sensor networks. In: IEEE internatonal conference on mobile adhoc and sensor systems, 2007 (MASS 2007) (pp. 1–9). doi:10.1109/MOBHOC.2007.4428658.

  22. Shah, K., & Kumar, M. (2008). Resource management in wireless sensor networks using collective intelligence. In: International conference on intelligent sensors, sensor networks and information processing, 2008 (ISSNIP 2008) (pp. 423–428). doi:10.1109/ISSNIP.2008.4762025.

  23. Sutton, R. (1984). Temporal credit assignment in reinforcement learning. Ph.D. thesis, Department of Computer Science, University of Massachusetts, Amherst, MA. Published as COINS Technical Report 84-2.

  24. Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: The MIT Press. http://www.cs.ualberta.ca/sutton/book/ebook/the-book.html.

  25. Terfloth, K., Wittenburg, G., & Schiller, J. H. (2006). FACTS—a rule-based middleware architecture for wireless sensor networks. In COMSWARE IEEE.

  26. Victor Lesser Charles Ortiz, M. T. (2003). Distributed sensor networks: A multiagent perspective. Kluwer.

  27. Watkins, C. (1989). Learning from delayed rewards. Ph.D. thesis, University of Cambridge, England.

  28. Watkins, C. J., & Dayan, P. (1992). Q-learning. Machine Learning 8(3), 279–292.

    MATH  Google Scholar 

  29. Wolpert, D., & Tumer, K. (1999). An introduction to collective intelligence. CoRR cs.LG/9908014. http://arxiv.org/abs/cs.LG/9908014 (Informal publication).

  30. Wolpert, D., & Tumer, K. (2002). Collective intelligence, data routing and braess’ paradox. Journal of Artificial Intelligence Research (JAIR), 16, 359–387. doi:10.1613/jair.995.

  31. Yu, Y., Krishnamachari, B., & Prasanna, V. K. (2004). Issues in designing middleware for wireless sensor networks. IEEE Network, 18(1), 15–21.

    Article  Google Scholar 

Download references

Acknowledgments

The research work presented in this article was supported by the National Science Foundation (NSF) under Grants ECCS-0824120 and CNS-0721951.

Author information

Authors and Affiliations

  1. University of Texas at Arlington, Arlington, TX, USA

    Kunal Shah & Mohan Kumar

  2. Aalto University, Espoo, Finland

    Mario Di Francesco

Authors
  1. Kunal Shah
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mario Di Francesco
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohan Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Kunal Shah.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Shah, K., Di Francesco, M. & Kumar, M. Distributed resource management in wireless sensor networks using reinforcement learning. Wireless Netw 19, 705–724 (2013). https://doi.org/10.1007/s11276-012-0496-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11276-012-0496-2

Keywords

Search

Navigation