Bio-inspired heuristic dynamic programming for high-precision real-time flow control in a multi-tributary river system

Abstract

Rivers are a type of time-varying, highly nonlinear systems, especially with multiple tributaries. To address low precision and poor real-time flow control of multi-tributary river systems, a heuristic dynamic programming algorithm inspired by regulation mechanisms in the human body (Bio-HDP) is proposed. Bio-HDP consists of several control units including the model network (MNCU), the action network (ANCU), the critic network (CNCU), and the central coordination (CCCU), all of which are developed based on the interactions and mechanisms in human’s NEI system (nervous, endocrine, and immune systems). The MNCU approximates the actual system through learning actual data on the control object. The ANCU calculates the control variable according to the error between the current state of the system acquired from the MNCU and the set value. The CNCU is used to evaluate whether the current control variable obtained by the ANCU can meet the control requirements. Finally, the CCCU conducts coordinated regulation of the above three control units in real-time according to the current state of the system, and makes the control mechanism faster and more accurate. By incorporating a biological feedback mechanism into the MNCU, the ANCU, and the CNCU, we adopt a neural network with multi-level feedback for the control mechanism. We select the Chongyang River system in China as the control object to verify the effects of the Bio-HDP control algorithm. The results show that the Bio-HDP algorithm improves speed, adaptability, and accuracy of the algorithm in comparison to other algorithms, and exhibits timely and accurate control effects.

Introduction

Floods are a natural disaster caused by the rapid increase in water volume/level of rivers or lakes during extreme natural events such as rainstorms, rapid melting of ice and snow, and storm surges [1], [2]. In China, rapid growth in population, expansion of farmlands, lake reclamation, deforestation, and unstainable land and water management have changed the state of the land surface and confluence conditions thereby exacerbating the frequency and severity of floods [3]. Therefore, it is an urgent task to strengthen river regulation and conduct precise and timely flow control to mitigate flood hazards [4]. For a river with multiple tributaries, there is a closely hydraulic connection between the mainstream and tributaries, and the amount of runoff received by the mainstream is also closely related to the amount of water from each tributary. Most of the mainstream basins flow through developed socio-economic areas [5], [6]. The amount of water supplied directly affects the socio-economic development of the mainstream basin and has a significant impact on the water dependent ecosystems [7]. Therefore, the spatiotemporal dynamics on the confluence of multi-tributary rivers can provide an informative basis for water resources management, protection, and planning, and it is of great theoretical and practical significance for flood control in the basin where rivers with multiple tributaries converge. An effective control tool is required to regulate the river flow at the confluence of multiple tributaries, in order to reduce the harm of flood to social economy and human safety.

In the early 20th century, dams mostly relied on manual control to manipulate the speed of flows and change the level of water reserved within a catchment area [8]. As manual control depended mostly on the experience of operators, control errors often tended to be large [9]. In 1910, the Proportion–integral–derivative (PID) controller was first proposed [10], [11], where only three parameters need to be adjusted. The controller was widely applied in industrial process control [12]. However, this control approach has deficiencies, specifically in control accuracy and its anti-interference ability, and it is difficult to achieve timely and accurate control of water resource systems with time-varying properties and high nonlinearity in water resource systems dynamics​ [13], [14], [15]. After the emergence of PID, intelligent controllers began to be proposed. [16] designed a back-propagation (BP) neural network, which obtains control rules required by the system through continuously learning the system structure, parameters, uncertainty, and nonlinearity. Moreover, it rectifies the challenge associated with PID parameters, which cannot satisfy nonlinearity and uncertainty of the system in the process of parameter presetting based on expert experience. For example, [17] combined a BP neural network with the PID approach to control water quality that changed based on environmental responses during thermal power generation. They found that the combined method improved the oscillation amplitude and adjustment time of the system when the pre-set value varied. Furthermore, the study by [18] optimized the selection of flood control scheduling schemes and the control scheme of a water resource system through the BP-Set Pair Analysis (BP-SPA) algorithm. However, they only evaluated the existing control schemes without involving the design of control schemes. Additionally, the BP neural network is easily trapped in a local minimum, and its adaptability to time-varying systems is weak due to the complexity of resource scheduling. In order to address the defect of the BP neural network, some scholars introduced optimization algorithms such as genetic algorithm (GA) and adaptive dynamic programming algorithm into the BP neural network. For example, [19] applied an adaptive fuzzy control to adjust the non-negative water supply pressure process, and the experiment result demonstrated that the control algorithm can regulate the water supply more accurately than the PID law, but the accuracy of model recognition was still not satisfactory. Furthermore, [20] applied GA, particle swarm optimization (PSO) and shark machine learning algorithm (SMLA) to the decision-making process of reservoir operation rules, respectively; however, these control algorithms still existed a certain vulnerability value, even the SMLA which exhibited the best control effect but still had 21.9% vulnerability value. In summary, the control effects of highly nonlinear and time-varying river systems need to be further improved.

The aforementioned problems can be improved by heuristic dynamic programming (HDP) proposed by [21]. Compared to traditional adaptive dynamic programming algorithms [22], which has been improved for robust control under uncertain conditions [23], [24], the HDP algorithm can change the control strategy in real-time, simplify the structure of the control system, and increase the flexibility in the use of the algorithm. Therefore, HDP is expected to improve the real-time nature of water resource regulation, and make water resource regulation more flexible and precise. To the best of the authors’ knowledge, there is no solid application of HDP in the field of water resources management.

Although HDP has the advantages mentioned above, this algorithm contains less feedback information, leading to deficient control effect, such as big errors and large overshoot. Therefore, the control adaptability and stability of a system solely using the HDP is insufficient. Due to the difficulties in simultaneously controlling multiple variables, as well as high nonlinearity and high time variability of the multi-tributary river system, there are few studies on the control of such river systems. Based on the literature, most studies in this field focused on the prediction of multi-tributary river systems [25], [26], [27]. Therefore, water resource regulation and control urgently need advanced control theory and algorithms with strong adaptability, reliable stability, and high control accuracy.

On the other hand, human body has offered many innovative inspirations and solutions to engineering problems (e.g., [28], [29], [30]). The biological regulation and control system in human body has outstanding capabilities of synergy, learning, and memory [31], which includes nervous system, endocrine system, and immune system, i.e. NEI system. These three systems work collaboratively to maintain the stability of the human biological system through relevant regulatory mechanisms such as the ultra-short feedback mechanism and central nervous cooperative regulation mechanism. Since the structure of HDP with three networks is similar to that of the three internal subsystems contained in the NEI system, the combination of the HDP algorithm and biological regulation and control mechanisms in human body may efficiently implement water resource control, particularly in multi-tributary river system control, with increased stability, accuracy, and adaptability.

This paper proposes a biologically inspired HDP algorithm (Bio-HDP) which is inspired by the cooperative regulation mechanism of the human’s NEI system. The algorithm includes four key components: the model network control unit (MNCU), the action network control unit (ANCU), the critic network control unit (CNCU) [32], and the central coordination control unit (CCCU). Compared with the model network, action network and critic network in the HDP architecture, Bio-HDP adds CCCU as a new regulatory unit, which can coordinate each network according to the system control, making the control timelier and more stable. Bio-HDP exhibits satisfactory self-learning, self-adaptive, robust, and fault-tolerant capabilities. We apply the algorithm into the dam control of Chongyang River in China to regulate and control water flows of multi-tributary river system under complex and changing conditions. [33] built a flood flow warning system for this river based on actual on-site water volume data. In reality, this river still relies on artificial experience control, and the control effect is not ideal, since the predictive control effect on natural disasters such as floods is weak. Based on the dataset provided by [33], we built a river system simulation model with the Bio-HDP control algorithm applied. The system simulation model inputs the state quantities of the controlled object and other information to the controller. The control algorithm obtains control variables through the prediction of the MNCU, the calculation of the ANCU, the evaluation of the CNCU and the co-regulation of CCCU. Finally, the controller outputs control variables to the actuator, i.e. gate, to control the river flow. The simulation results show that Bio-HDP algorithm shows better control effects compared to existing control algorithms.

The main contributions of this paper are summarized as follows: (1) A new dynamic programming control algorithm, called biological-inspired heuristic dynamic programming (Bio-HDP), is proposed in this work by integrating the biological regulatory mechanism with the adaptive dynamic programming algorithm. (2) In the proposed algorithm, the ultra-short feedback mechanism of the endocrine system is innovatively incorporated into the neural network, and its convergence speed is increased by improving the structure of the neural network. (3) The proposed control algorithm is applied into addressing a real-world challenge—regulating a multi-tributary system. Our simulation experiments demonstrate that the control algorithm can successfully implement high-precision real-time flow control in Chongyang River.

The rest of this paper is organized as follows: Section 2 presents an overview of the heuristic dynamic programming algorithm and relevant biological regulation mechanisms used in the Bio-HDP algorithm. In Section 3, based on the heuristic dynamic programming algorithm, the Bio-HDP algorithm is proposed by integrating control mechanisms in the biological network. The Bio-HDP structure, control components/units, and workflow are also presented. Section 4 provides an application of the proposed Bio-HDP algorithm into high-precision real-time flow control of a multi-tributary river system. Section 5 shows the simulation results of the application, verifying and analyzing the control effects of the proposed Bio-HDP algorithm. The final section provides a summary of the study.