Skip to main content
Springer Nature Link
Log in

ADP-based optimal sensor scheduling for target tracking in energy harvesting wireless sensor networks

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper proposes a novel sensor scheduling scheme based on adaptive dynamic programming, which makes the sensor energy consumption and tracking error optimal over the system operational horizon for wireless sensor networks with solar energy harvesting. Neural network is used to model the solar energy harvesting. Kalman filter estimation technology is employed to predict the target location. A performance index function is established based on the energy consumption and tracking error. Critic network is developed to approximate the performance index function. The presented method is proven to be convergent. Numerical example shows the effectiveness of the proposed approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Rout RR, Ghosh SK (2013) Enhancement of lifetime using duty cycle and network coding in wireless sensor networks. IEEE Trans Wirel Commun 12(2):656–667

    Article  Google Scholar 

  2. Li Y, Chen CS, Song Y, Wang Z, Sun Y (2009) Enhancing real-time delivery in wireless sensor networks with two-hop information. IEEE Trans Ind Inform 5(2):113–122

    Article  Google Scholar 

  3. Shi L, Jia QS, Mo YL, Sinopoli B (2011) Sensor scheduling over a packet-delaying network. Automatica 47:1089–1092

    Article  MathSciNet  MATH  Google Scholar 

  4. Mo YL, Ambrosino R, Sinopoli B (2011) Sensor selection strategies for state estimation in energy constrained wireless sensor networks. Automatica 47:1330–1338

    Article  MathSciNet  MATH  Google Scholar 

  5. Koutsopoulos I, Stańczak S (2012) The impact of transmit rate control on energy-efficient estimation in wireless sensor networks. IEEE Trans Wirel Commun 11(9):3261–3271

    Article  Google Scholar 

  6. Wei D, Jin Y, Vural S, Moessner K, Tafazolli R (2011) An energy-efficient clustering solution for wireless sensor networks. IEEE Trans Wirel Commun 10(11):3973–3983

    Article  Google Scholar 

  7. Li B, Li H, Wang W, Yin Q, Liu H (2013) Performance analysis and optimization for energy-efficient cooperative transmission in random wireless sensor network. IEEE Trans Wirel Commun 12(9):4647–4657

    Article  Google Scholar 

  8. Wu Y, Liu W (2013) Routing protocol based on genetic algorithm for energy harvesting-wireless sensor networks. IET Wirel Sens Syst 3(2):112–118

    Article  Google Scholar 

  9. Alippi C, Galperti C (2008) An adaptive system for optimal solar energy harvesting in wireless sensor network nodes. IEEE Trans Circuits Syst I 55(6):1742–1750

    Article  MathSciNet  Google Scholar 

  10. Werbos PJ (1977) Advanced forecasting methods for global crisis warning and models of intelligence. Gen Syst Yearb 22:25–38

    Google Scholar 

  11. Werbos PJ (1991) A menu of designs for reinforcement learning over time. In: Miller WT, Sutton RS, Werbos PJ (eds) Neural networks for control. MIT Press, Cambridge, pp 67–95

    Google Scholar 

  12. Wei Q, Wang D, Zhang D (2013) Dual iterative adaptive dynamic programming for a class of discrete-time nonlinear systems with time-delays. Neural Comput Appl 23(7–8):1851–1863

    Article  Google Scholar 

  13. Wang Z, Liu D (2013) A data-based state feedback control method for a class of nonlinear systems. IEEE Trans Ind Inform 9(4):2284–2292

    Article  Google Scholar 

  14. Wei Q, Liu D (2014) A novel iterative \(\theta\)-adaptive dynamic programming for discrete-time nonlinear systems. IEEE Trans Autom Sci Eng 11(4):1176–1190

    Article  Google Scholar 

  15. Heydari A, Balakrishnan SN (2013) Finite-horizon control-constrained nonlinear optimal control using single network adaptive critics. IEEE Trans Neural Netw Learn Syst 24(1):145–157

    Article  Google Scholar 

  16. Liu D, Wei Q (2013) Finite-approximation-error-based optimal control approach for discrete-time nonlinear systems. IEEE Trans Cybern 43(2):779–789

    Article  Google Scholar 

  17. Xu X, Hou Z, Lian C, He H (2013) Online learning control using adaptive critic designs with sparse kernel machines. IEEE Trans Neural Netw Learn Syst 24(5):762–775

    Article  Google Scholar 

  18. Wei Q, Liu D (2012) An iterative \(\epsilon\)-optimal control scheme for a class of discrete-time nonlinear systems with unfixed initial state. Neural Netw 32(6):236–244

    Article  MathSciNet  MATH  Google Scholar 

  19. Liu D, Javaherian H, Kovalenko O, Huang T (2008) Adaptive critic learning techniques for engine torque and air–fuel ratio control. IEEE Trans Syst Man Cybern Part B Cybern 38(4):988–993

    Article  Google Scholar 

  20. Wei Q, Liu D (2014) Stable iterative adaptive dynamic programming algorithm with approximation errors for discrete-time nonlinear systems. Neural Comput Appl 24(6):1355–1367

    Article  Google Scholar 

  21. Prokhorov DV, Wunsch DC (1997) Adaptive critic designs. IEEE Trans Neural Netw 8(5):997–1007

    Article  Google Scholar 

  22. Liang J, Molina DD, Venayagamoorthy GK, Harley RG (2013) Two-level dynamic stochastic optimal power flow control for power systems with intermittent renewable generation. IEEE Trans Power Syst 28(3):2670–2678

    Article  Google Scholar 

  23. Ni Z, He H, Wen J, Xu X (2013) Goal representation heuristic dynamic programming on maze navigation. IEEE Trans Neural Netw Learn Syst 24(12):2038–2050

    Article  Google Scholar 

  24. Ni Z, He H, Wen J (2013) Adaptive learning in tracking control based on the dual critic network design. IEEE Trans Neural Netw Learn Syst 24(6):913–928

    Article  Google Scholar 

  25. Wei Q, Liu D (2014) Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification. IEEE Trans Autom Sci Eng 11(4):1020–1036

    Article  Google Scholar 

  26. Song R, Zhang H (2013) The finite horizon optimal control for a class of time-delay affine nonlinear system. Neural Comput Appl 22(2):229–235

    Article  Google Scholar 

  27. Jiang Y, Jiang ZP (2014) Robust adaptive dynamic programming and feedback stabilization of nonlinear systems. IEEE Trans Neural Netw Learn Syst 25(5):882–893

    Article  Google Scholar 

  28. Bian T, Jiang Y, Jiang ZP (2014) Adaptive dynamic programming and optimal control of nonlinear nonaffine systems. Automatica 50(10):2624–2632

    Article  MathSciNet  MATH  Google Scholar 

  29. Jiang ZP, Jiang Y (2013) Robust adaptive dynamic programming for linear and nonlinear systems: an overview. Eur J Control 19(5):417–425

    Article  MathSciNet  MATH  Google Scholar 

  30. Xu X, Lian C, Zuo L, He H (2014) Kernel-based approximate dynamic programming for real-time online learning control: an experimental study. IEEE Trans Control Syst Technol 22(1):146–156

    Article  Google Scholar 

  31. Molina D, Venayagamoorthy GK, Liang J, Harley RG (2013) Intelligent local area signals based damping of power system oscillations using virtual generators and approximate dynamic programming. IEEE Trans Smart Grid 4(1):498–508

    Article  Google Scholar 

  32. Squartini S, Lu J, Wei Q (2013) The neural paradigm for complex systems: new algorithms and applications. Neural Comput Appl 22(2):203–204

    Article  Google Scholar 

  33. Xu H, Jagannathan S (2013) Stochastic optimal controller design for uncertain nonlinear networked control system via neuro dynamic programming. IEEE Trans Neural Netw Learn Syst 24(3):471–484

    Article  Google Scholar 

  34. Modares H, Lewis FL (2014) Optimal tracking control of nonlinear partially-unknown constrained-input systems using integral reinforcement learning. Automatica 50(7):1780–1792

    Article  MathSciNet  MATH  Google Scholar 

  35. Kiumarsi B, Lewis FL, Modares H, Karimpur A, Naghibi-Sistani MB (2014) Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics. Automatica 50(4):1167–1175

    Article  MathSciNet  MATH  Google Scholar 

  36. Modares H, Lewis FL, Naghibi-Sistani MB (2014) Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems. Automatica 50:193–202

    Article  MathSciNet  MATH  Google Scholar 

  37. Zhang H, Wei Q, Liu D (2011) An iterative adaptive dynamic programming method for solving a class of nonlinear zero-sum differential games. Automatica 47(1):207–214

    Article  MathSciNet  MATH  Google Scholar 

  38. Vamvoudakis KG, Lewis FL (2011) Multi-player non-zero-sum games: online adaptive learning solution of coupled Hamilton–Jacobi equations. Automatica 47(8):1556–1569

    Article  MathSciNet  MATH  Google Scholar 

  39. Zhang H, Wei Q, Luo Y (2008) A novel infinite-time optimal tracking control scheme for a class of discrete-time nonlinear systems via the greedy HDP iteration algorithm. IEEE Trans Syst Man Cybern Part B Cybern 38(4):937–942

    Article  Google Scholar 

  40. Huang Y, Liu D (2014) Neural-network-based optimal tracking control scheme for a class of unknown discrete-time nonlinear systems using iterative ADP algorithm. Neurocomputing 125:46–56

    Article  Google Scholar 

  41. Wei Q, Liu D (2015) Neural-network-based adaptive optimal tracking control scheme for discrete-time nonlinear systems with approximation errors. Neurocomput Part A 149(3):106–115

    Article  MathSciNet  Google Scholar 

  42. Modares H, Lewis FL (2014) Linear quadratic tracking control of partially-unknown continuous-time systems using reinforcement learning. IEEE Trans Autom Control 59:3051–3056

    Article  MathSciNet  Google Scholar 

  43. Kiumarsi B, Lewis FL, Naghibi-Sistani MB, Karimpour A (2015) Approximate dynamic programming for optimal tracking control of unknown linear systems using measured data. IEEE Trans Cybern. doi:10.1109/TCYB.2014.2384016

  44. Hengster-Movric K, You K, Lewis FL, Xie L (2013) Synchronization of discrete-time multi-agent systems on graphs using Riccati design. Automatica 49(2):414–423

    Article  MathSciNet  MATH  Google Scholar 

  45. Song R, Xiao W, Zhang H, Sun C (2014) Adaptive dynamic programming for a class of complex-valued nonlinear systems. IEEE Trans Neural Netw Learn Syst 25(9):1733–1739

    Article  Google Scholar 

  46. Bian T, Jiang Y, Jiang ZP (2014) Decentralized adaptive optimal control of large-scale systems with application to power systems. IEEE Trans Ind Electron. doi:10.1109/TIE.2014.2345343

  47. Jiang Y, Jiang ZP (2013) Robust adaptive dynamic programming with an application to power systems. IEEE Trans Neural Netw Learn Syst 24:1150–1156

    Article  MATH  Google Scholar 

  48. Wei Q, Liu D, Shi G (2015) A novel dual iterative Q-learning method for optimal battery management in smart residential environments. IEEE Trans Ind Electron 62(4):2509–2518

    Article  Google Scholar 

  49. Wei Q, Wang F-Y, Liu D, Yang X (2014) Finite-approximation-error-based discrete-time iterative adaptive dynamic programming. IEEE Trans Cybern 44(12):2820–2833

    Article  Google Scholar 

  50. Wei Q, Liu D (2014) A novel iterative \(\theta\)-adaptive dynamic programming for discrete-time nonlinear systems. IEEE Trans Autom Sci Eng 11(4):1176–1190

    Article  Google Scholar 

  51. Wei Q, Liu D (2014) Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification. IEEE Trans Autom Sci Eng 11(4):1020–1036

    Article  Google Scholar 

  52. Kiumarsi B, Lewis FL (2015) Actor-critic based optimal tracking for partially unknown nonlinear discrete-time systems. IEEE Trans Neural Netw Learn Syst 26(1):140–151

    Article  MathSciNet  Google Scholar 

  53. Modares H, Lewis FL, Naghibi-Sistani MB (2013) Adaptive optimal control of unknown constrained-input systems using policy iteration and neural networks. IEEE Trans Neural Netw Learn Syst 24(10):1513–1525

    Article  Google Scholar 

  54. Xiao W, Song, R (2012) Self-learning sensor scheduling for target tracking in wireless sensor networks based on adaptive dynamic programming. In: 10th world congress on intelligent control and automation, pp 1056–1061

  55. Xiao W, Song R (2012) Adaptive dynamic programming for sensor scheduling in energy-constrained wireless sensor networks. In: 15th international conference on information fusion, pp 991–996

  56. Fadare DA (2009) Modelling of solar energy potential in Nigeria using an artificial neural network model. Appl Energy 86:1410–1422

    Article  Google Scholar 

  57. Maheswararajah S, Halgamuge SK, Premaratne M (2009) Sensor scheduling for target tracking by suboptimal algorithms. IEEE Trans Veh Technol 58(3):1467–1479

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 61304079, 61374105, in part by the Beijing Natural Science Foundation under Grants 4132078, 4143065, in part by the China Postdoctoral Science Foundation under Grant 2013M530527, and in part by Fundamental Research Funds for the Central Universities under Grant FRF-TP-14-119A2, and in part by the Open Research Project from SKLMCCS under Grants 20150104, 20120106.

Author information

Authors and Affiliations

  1. School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, 100083, China

    Ruizhuo Song & Wendong Xiao

  2. The State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Qinglai Wei

Authors
  1. Ruizhuo Song
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Qinglai Wei
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Wendong Xiao
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Qinglai Wei.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Song, R., Wei, Q. & Xiao, W. ADP-based optimal sensor scheduling for target tracking in energy harvesting wireless sensor networks. Neural Comput & Applic 27, 1543–1551 (2016). https://doi.org/10.1007/s00521-015-1954-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-015-1954-4

Keywords