Skip to main content
Springer Nature Link
Log in

A fuzzy reinforcement learning approach to thermal unit commitment problem

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

    We’re sorry, something doesn't seem to be working properly.

    Please try refreshing the page. If that doesn't work, please contact support so we can address the problem.

Abstract

Unit commitment problem (UCP) aims at optimizing generation cost for meeting a given load demand under several operational constraints. We propose to use fuzzy reinforcement learning (RL) approach for efficient and reliable solution to the unit commitment problem. In particular, we cast UCP as a multiagent fuzzy reinforcement learning task wherein individual generators act as players for optimizing the cost to meet a given load over a twenty-four-hour period. Unit commitment task has been fuzzified, and the most optimal unit commitment solution is generated by employing RL on this fuzzy multigenerator setup. Our proposed multiagent RL framework does not assume any a priori task or system knowledge, and the generators gradually learn to produce most optimal output solely based on their collective generation. We look at the UCP as a sequential decision-making task with reward/penalty to reduce the collective generation cost of generators. To the best of our knowledge, ours is a first attempt at solving UCP by employing fuzzy reinforcement learning. We test our approach on a ten-generating-unit system with several equality and inequality constraints. Simulation results and comparisons against several recent UCP solution methods prove superiority and viability of our proposed multiagent fuzzy reinforcement learning technique.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Abbreviations

\(U_{g,h}\) :

Output power of unit \(g\) at time h

\(FC_{g} (U_{g,h} )\) :

Fuel cost of unit \(g\) when its output power is \(U_{g,h}\)

\(S_{\text{up}} (g,h)\) :

Start-up cost of unit \(g\) at time \(h\)

\(S_{\text{down}} (g,h)\) :

Shutdown cost of unit \(g\) at time \(h\)

\(HS_{\text{up}} (g,h)\) :

Hot start-up cost of unit \(g\) at time \(h\)

\(CS_{\text{up}} (g,h)\) :

Cold start-up cost of unit \(g\) at time \(h\)

\(X_{g,h}\) :

ON/OFF state of unit \(g\) at time \(h\)

\(U_{g,\hbox{max} }\) :

Maximum power generation of unit \(g\)

\(U_{g,\hbox{min} }\) :

Minimum power generation of unit \(g\)

\(H_{g,h}^{\text{OFF}}\) :

Continuously OFF time duration of unit \(g\) at time \(h\)

\(H_{g,h}^{\text{ON}}\) :

Continuously ON time duration of unit \(g\) at time \(h\)

\(H_{g}^{\text{Up}}\) :

Minimum uptime of unit \(g\)

\(H_{g}^{\text{Down}}\) :

Minimum downtime of unit \(g\)

\(H_{g}^{\text{Cold}}\) :

Cold start hours of unit \(g\)

\(a_{g} ,b_{g} ,c_{g}\) :

Fuel cost coefficients of unit \(g\)

\(p_{{{\text{demand}},h}}\) :

System demand at time \(h\)

\(SR_{h}\) :

Spinning reserve at time \(h\)

\(\gamma\) :

Discount factor

\(\eta\) :

Learning rate parameter

References

  1. Wood AJ, Wollenberg BF (2012) Power generation, operation, and control. 2nd edn. John Wiley

  2. Padhy NP (2004) Unit commitment—a bibliographical survey. IEEE Trans Power Syst. doi:10.1109/TPWRS.2003.821611

    Google Scholar 

  3. Mukherjee S, Adrian EC (1989) Implementation of a lagrangian relaxation based unit commitment problem. IEEE Trans Power Syst. doi 10(1109/59):41687

    Google Scholar 

  4. Ongsakul W, Petcharaks N (2004) Unit commitment by enhanced adaptive Lagrangian relaxation. IEEE Trans Power Syst. doi:10.1109/TPWRS.2003.820707

    Google Scholar 

  5. Chandram K, Subrahmanyam N, Sydulu M (2011) Unit commitment by improved pre-prepared power demand table and Muller method. Int J Electr Power Energy Syst. doi:10.1016/j.ijepes.2010.06.022

    Google Scholar 

  6. Hosseini SH, Khodaei A, Aminifar F (2007) A novel straightforward unit commitment method for large-scale power systems. IEEE Trans Power Syst. doi:10.1109/TPWRS.2007.907443

    Google Scholar 

  7. Cheng C-P, Liu C-W, Liu C-C (2000) Unit commitment by Lagrangian relaxation and genetic algorithms. IEEE Trans, POWER Syst, p 15

    Google Scholar 

  8. Venkatesh B, Yu P, Gooi HB, Choling D (2008) Fuzzy MILP unit commitment incorporating wind generators. IEEE Trans Power Syst. doi:10.1109/TPWRS.2008.2004724

    Google Scholar 

  9. Liang R-H, Kang F-C (2000) Thermal generating unit commitment using an extended mean field annealing neural network. IEE proc-Gener Transm Distrib 147(3):164–170. doi:10.1049/ipgtd:20000303

    Google Scholar 

  10. Zhuang Galiana Senior Member FF (1990) Unit commitment by simulated annealing. IEEE Trans, Power Syst, p 5

    Google Scholar 

  11. Mantawy AH, Abdel-Magid YL, Selim SZ (1998) Unit commitment by tabu search. IEE Proc - Gener Transm Distrib 145:56. doi:10.1049/ip-gtd:19981681

    Article  Google Scholar 

  12. Logenthiran T, Srinivasan D (2010) Particle swarm optimization for unit commitment problem. PMAS. 642–647

  13. Juste KA, Kitu H, Tunaka E, Hasegawa J (1999) An evolutionary programming solution to the unit commitment problem. IEEE Trans Power Syst 14:1452–1459

    Article  Google Scholar 

  14. Sisworahardjo NS, El-Keib AA (2002) Unit Commitment Using the Ant Colony Search Algorithm.Large Eng. syst.Conf. Power Eng. 2-6

  15. Patra S, Goswami SK, Goswami B (2008) Differential evolution algorithm for solving unit commitment with ramp constraints. Electr Power Components Syst 36:771–787. doi:10.1080/15325000801911377

    Article  Google Scholar 

  16. Eslamian M, Hosseinian SH, Vahidi B (2009) Bacterial foraging-based solution to the unit-commitment problem. IEEE Trans Power Syst. doi:10.1109/TPWRS.2009.2021216

    Google Scholar 

  17. Ebrahimi J, Hosseinian SH, Gharehpetian GB (2011) Unit commitment problem solution using shuffled frog leaping algorithm. IEEE Trans Power Syst 26:573–581. doi:10.1109/TPWRS.2010.2052639

    Article  Google Scholar 

  18. Roy PK (2013) Solution of unit commitment problem using gravitational search algorithm. Int J Electr Power Energy Syst 53:85–94. doi:10.1016/j.ijepes.2013.04.001

    Article  Google Scholar 

  19. Roy PK, Sarkar R (2014) Solution of unit commitment problem using quasi-oppositional teaching learning based algorithm. Int J Electr Power Energy Syst 60:96–106. doi:10.1016/j.ijepes.2014.02.008

    Article  Google Scholar 

  20. Rameshkumar J, Ganesan S, Abirami M, Subramanian S (2016) Cost, emission and reserve pondered pre-dispatch of thermal power generating units coordinated with real coded grey wolf optimisation. IET Gener Trans Distribut 10(4):972–985. doi:10.1049/iet-gtd.2015.0726

    Google Scholar 

  21. Srinivasan D, Chazelas J (2004) A priority list-based evolutionary algorithm to solve large scale unit commitment problem. IEEE international conference on power system technology (PowerCon 2004) pp 21–24

  22. Saberl AY, Senjyul T, Miyagil T (2006) Fuzzy unit commitment using absolutely stochastic simulated annealing. IEEE Trans Power Syst 21(2):955–964

    Article  Google Scholar 

  23. Zhao B, Guo CX, Bai BR, Cao YJ (2006) An improved particle swarm optimization algorithm for unit commitment. Int J Electr Power Energy Syst 28:482–490. doi:10.1016/j.ijepes.2006.02.011

    Article  Google Scholar 

  24. Lau TW, Chung CY, Wong KP et al (2009) Quantum-inspired evolutionary algorithm approach for unit commitment. IEEE Trans Power Syst. doi:10.1109/TPWRS.2009.2021220

    Google Scholar 

  25. Jeong YW, Park JB, Jang SH, Lee KY (2010) A new quantum-inspired binary PSO: application to unit commitment problems for power systems. IEEE Trans Power Syst. doi:10.1109/TPWRS.2010.2042472

    Google Scholar 

  26. Damousis IG, Bakirtzis AG, Dokopoulos PS (2004) A solution to the unit-commitment problem using integer-coded genetic algorithm. IEEE Trans Power Syst. doi:10.1109/TPWRS.2003.821625

    Google Scholar 

  27. Datta D, Dutta S (2012) A binary-real-coded differential evolution for unit commitment problem. Int J Electr Power Energy Syst 42:517–524. doi:10.1016/j.ijepes.2012.04.048

    Article  Google Scholar 

  28. Yuan X, Su A, Nie H et al (2009) Application of enhanced discrete differential evolution approach to unit commitment problem. Energy Convers Manag 50:2449–2456. doi:10.1016/j.enconman.2009.05.033

    Article  Google Scholar 

  29. Chandrasekaran K, Simon SP, Padhy NP (2013) Binary real coded firefly algorithm for solving unit commitment problem. Inf Sci (Ny) 249:67–84. doi:10.1016/j.ins.2013.06.022

    Article  Google Scholar 

  30. Farsangi MM, Barati M (2014) Solving unit commitment problem by a binary shuffled frog leaping algorithm. IET Gener Transm Distrib 8:1050–1060. doi:10.1049/iet-gtd.2013.0436

    Article  Google Scholar 

  31. Wu Z, Chow TWS (2012) Binary neighbourhood field optimisation for unit commitment problems. doi:10.1049/iet-gtd.2012.0096

    Google Scholar 

  32. Han D, Jian J, Yang L (2014) Outer approximation and outer-inner approximation approaches for unit commitment problem. IEEE Trans Power Syst. doi:10.1109/TPWRS.2013.2253136

    Google Scholar 

  33. Niknam T, Bavafa F, Azizipanah-Abarghooee R (2013) New self-adaptive bat-inspired algorithm for unit commitment problem. doi:10.1049/iet-smt.2013.0252

    Google Scholar 

  34. Quan R, Jian J, Yang L (2015) An improved priority list and neighborhood search method for unit commitment. Int J Electr Power Energy Syst 67:278–285. doi:10.1016/j.ijepes.2014.11.025

    Article  Google Scholar 

  35. Yuan X, Ji B, Zhang S et al (2014) A new approach for unit commitment problem via binary gravitational search algorithm. Appl Soft Comput 22:249–260. doi:10.1016/j.asoc.2014.05.029

    Article  Google Scholar 

  36. Chen PH (2012) Two-level hierarchical approach to unit commitment using expert system and elite PSO. IEEE Trans Power Syst. doi:10.1109/TPWRS.2011.2171197

    Google Scholar 

  37. Quan H, Srinivasan D, Khosravi A (2015) Incorporating wind power forecast uncertainties into stochastic unit commitment using neural network-based prediction intervals. IEEE Trans Neural Networks Learn Syst. doi:10.1109/TNNLS.2014.2376696

    MathSciNet  Google Scholar 

  38. Xie Y-G, Chiang H-D (2010) A novel solution methodology for solving large-scale thermal unit commitment problems. Electr Power Components Syst 38:1615–1634. doi:10.1080/15325008.2010.492453

    Article  Google Scholar 

  39. Ahmed MH, Bhattacharya K, Salama MMA (2012) Stochastic unit commitment with wind generation penetration. Electr Power Components Syst 40:1405–1422. doi:10.1080/15325008.2012.694969

    Article  Google Scholar 

  40. Govardhan M, Roy R (2015) Economic analysis of unit commitment with distributed energy resources. Int J Electr Power Energy Syst 71:1–14. doi:10.1016/j.ijepes.2015.01.028

    Article  Google Scholar 

  41. Kamboj VK, Bath SK, Dhillon JS (2016) Implementation of hybrid harmony/random search algorithm considering ensemble and pitch violation for unit commitment problem. Int J Electr Power Energy Syst 77:228–249. doi:10.1016/j.ijepes.2015.11.045

    Article  Google Scholar 

  42. Mahdavi MS, Vahidi B, Babamalek G et al (2016) A novel optimized fuzzy approach based on monte carlo method for system load, wind turbine and photovoltaic unit uncertainty modeling in unit commitment. Electr Power Components Syst 44:833–842. doi:10.1080/15325008.2016.1138343

    Article  Google Scholar 

  43. Tavakoli A, Sanjari MJ, Karami H et al (2015) Imperialistic competitive algorithm based unit commitment considering risk of cascading blackout. Electr Power Components Syst 43:374–383. doi:10.1080/15325008.2014.963261

    Article  Google Scholar 

  44. Abedinia O, Naslian MD, Bekravi M (2014) A new stochastic search algorithm bundled honeybee mating for solving optimization problems. Neural Comput Appl 25:1921–1939. doi:10.1007/s00521-014-1682-1

    Article  Google Scholar 

  45. Kamboj VK (2016) A novel hybrid PSO???GWO approach for unit commitment problem. Neural Comput Appl 27:1643–1655. doi:10.1007/s00521-015-1962-4

    Article  Google Scholar 

  46. Al-Betar MA, Awadallah MA, Khader AT et al (2016) Economic load dispatch problems with valve-point loading using natural updated harmony search. Neural Comput Appl. doi:10.1007/s00521-016-2611-2

    Google Scholar 

  47. Li F-D, Wu M, He Y, Chen X (2012) Optimal control in microgrid using multi-agent reinforcement learning. ISA Trans 51:743–751. doi:10.1016/j.isatra.2012.06.010

    Article  Google Scholar 

  48. Boubertakh H, Tadjine M, Glorennec P-Y, Labiod S (2010) Tuning fuzzy PD and PI controllers using reinforcement learning. ISA Trans 49:543–551. doi:10.1016/j.isatra.2010.05.005

    Article  Google Scholar 

  49. Treesatayapun C (2008) Fuzzy-rule emulated networks, based on reinforcement learning for nonlinear discrete-time controllers. ISA Trans 47:362–373. doi:10.1016/j.isatra.2008.07.001

    Article  Google Scholar 

  50. Yen GG, Hickey TW (2004) Reinforcement learning algorithms for robotic navigation in dynamic environments. ISA Trans 43:217–230. doi:10.1016/S0019-0578(07)60032-9

    Article  Google Scholar 

  51. Wiering M, van Otterlo M (2012) Reinforcement learning: State-of-the-Art. Adaptation, Learning, and Optimization, vol 12. Springer, Berlin. doi:10.1007/978-3-642-27645-3

    Book  Google Scholar 

  52. Buoniu L, Babuška R, De Schutter B, Ernst D (2010) Reinforcement learning and dynamic programming using function approximators. vol 39. CRC press

  53. Jouffe L (1998) Fuzzy inference system learning by reinforcement methods. IEEE Trans Syst Man Cybern Part C (Applications Rev 28:338–355. doi: 10.1109/5326.704563

  54. Rahimiyan M, Mashhadi HR (2010) An adaptive -learning algorithm developed for agent-based computational modeling of electricity market. IEEE Trans Syst Man Cybern Part C Appl Rev 40:547–556. doi:10.1109/TSMCC.2010.2044174

    Article  Google Scholar 

  55. Rajabi Mashhadi H, Rahimiyan M (2011) Measurement of power supplier’s market power using a proposed fuzzy estimator. IEEE Trans Power Syst 26:1836–1844. doi:10.1109/TPWRS.2011.2144626

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Division of Instrumentation and Control Engineering, Netaji Subhas Institute of Technology, New Delhi, 110078, New Delhi, India

    Nandan Kumar Navin & Rajneesh Sharma

Authors
  1. Nandan Kumar Navin
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Rajneesh Sharma
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Nandan Kumar Navin.

Ethics declarations

Conflict of interest

This research work does not have any financial or non-financial interest from any funding agencies. This work has carried out at Advanced Power and Control Research Lab of NSIT Delhi.

Appendix

Appendix

See Tables 5 and 6.

Table 5 Unit data for 10-unit system
Full size table
Table 6 Load data for 24 H
Full size table

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Navin, N.K., Sharma, R. A fuzzy reinforcement learning approach to thermal unit commitment problem. Neural Comput & Applic 31, 737–750 (2019). https://doi.org/10.1007/s00521-017-3106-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-017-3106-5

Keywords