Intraday Multireservoir Hydropower Optimization with Alternative Deep Reinforcement Learning Configurations

Abstract

Managing multireservoir hydropower systems in an intraday context poses unique challenges due to the need for frequent decisions in response to fluctuating energy prices. While Reinforcement Learning (RL) methods have been applied to long-term management, this paper addresses the gap in short-term planning within a single day. We use an alternative RL algorithm and investigate various modeling approaches for the intraday multireservoir optimization problem. Through extensive experiments using real hydropower system data, we analyze the performance of different RL agents and benchmark them against random and greedy policies. Results demonstrate that optimal modeling choices, including reward adjustment, sufficient forecast information, and grouping of actions, significantly impact performance. Moreover, our findings suggest that Soft Actor-Critic, a RL algorithm that has not been applied before in this domain, is a viable alternative to methods such as Q-learning. Overall, this study contributes to the understanding of RL techniques in hydropower optimization and provides valuable insights for practical implementation in real-world scenarios.

This work was supported both by the project Project IA4TES (Advanced Intelligent Technologies for Sustainable Energy Transition) with file number TSI-100408-2021 from the 2021 AI R&D Missions Program, within the framework of the Spain Digital Agenda 2025 and the National Artificial Intelligence Strategy, funded by the Recovery, Transformation, and Resilience Plan and co-financed with European funds from the Recovery and Resilience Facility (RRF), Next Generation EU and the Spanish Agencia Estatal de Investigación for the support provided by the Ministerio de Ciencia e Innovación of Spain (Grant Ref. PID2022-137748OB-C31 funded by MCIN/AEI/10.13039/501100011033) and “ERDF A way of making Europe”.