An echo state network based adaptive dynamic programming approach for time-varying parameters optimization with application in algal bloom prediction

https://doi.org/10.1016/j.asoc.2022.108796Get rights and content

Highlights

  • A time-varying algae growth model (TAGM) is proposed for algal bloom prediction.

  • CMA-ES is used to estimate the fixed parameters.

  • Adaptive dynamic programming with ESN is used to solve the incremental parameters.

  • Time series feature is employed to enhance the prediction accuracy.

  • TAGM is adopted to the actual lakes with better interpretability and precision.

Abstract

The prediction of algal bloom is one of the important links in eutrophication prevention. Chlorophyll a concentration is the indicating variable of algal bloom, and its time series is non-stationary and non-linear, which brings challenges to its effective prediction. Although the current algae growth model (AGM) can directly describe the algal bloom dynamics, the fixed parameters limit the adaptability of the model. If the fixed parameters are dynamically adjusted, the trend of chlorophyll a concentration can be better captured. Therefore, the adaptive dynamic programming (ADP) approach is used to optimize the parameters of the AGM. The ADP contains an action network and a critic network by echo state network, where the action network is used to output the increment value of the fixed parameters, and the critic network is used to approximate the performance index function. In this paper, the input of the action network uses the time series features extracted by the relevant variables, so that the time-varying parameters of the AGM have better dynamic characteristics. We verify the effectiveness of the proposed model through the dataset of the North Canal and Taihu Lake, and the convergence analysis proves the theoretical reliability. In this way, the improved mechanism model with time-varying parameters not only maintains the better interpretability of the original AGM, but also further enhances the prediction accuracy and adaptability by extracting inherent interactive features from the relevant variables.

Introduction

Due to the excessive discharge of urban sewage, the eutrophication of water in lakes and reservoirs has become more serious. In the case of suitable environmental conditions, the algae will multiply in eutrophic water, and bring algal bloom on the water surface [1]. The algal bloom not only destroys the biodiversity in water bodies, but also threatens water safety. At present, the formation mechanism of algal bloom is still unclear, researchers are also working to study the causes of multiple outbreaks of the algal bloom [2]. As a complex ecological phenomenon, the evolution of algal bloom is a time-delay biochemical reaction process, which has the characteristics of nonlinearity, time-variation, and strong coupling. Establishing an accurate and reliable algal bloom prediction model is one of the important missions to control the water eutrophication, which has urgent research value [3].

At present, the prediction of algal bloom mainly focuses on the evolution law of time series of indicating variables, such as chlorophyll a concentration. The complexity of algal bloom makes the time series of chlorophyll a concentration show non-stationary, catastrophic, and chaotic characteristics [4]. These make it difficult to predict accurately. In recent years, the data-driven model has increasingly become an effective method for the time series prediction [5]. The data-driven model does not require prior knowledge and can effectively extract the intrinsic information from big data. From machine learning to deep learning, the data-driven model for the nonlinear description of time series is obviously improved [6], [7], [8]. However, in order to accurately predict the multi-variate complex dynamics of algal bloom, the data-driven model needs a multi-scale dataset. In the past, most of the monitoring data of the water environment were collected by manual with a large period. Although the data collected for many years is conducive to the study of the overall change law (such as data from GLEON network), but large sampling span (such as one year) is not conducive to modeling with high quality. This brings a great challenge for algal bloom prevention by data-driven model, which relies on data source. Up to present, the sensors can automatically monitor and store data with small sampling interval, but the sufficient of accumulated years and dimensions are not enough. The application in the algal bloom prediction by the data-driven model is limited. As a type of black-box model, the prediction mechanism of the data-driven model is still being explored. Regarding the mechanism of algal bloom, researchers have conducted in-depth studies on ecological mechanisms and chemical reactions [9], [10], [11]. These mechanism models were originally used to deal with the problem of water quality analysis with good application effects and clear physical interpretability through partial differential equations. For algal bloom has many influencing variables and certain sensitivity, it is very hard to describe completely. At the same time, the mechanism model also has high requirements on the specific variables for simulated operation and parameters calibration. Therefore, how to simplify the mechanism model and using the appropriate parameter calibration method to improve the prediction accuracy are urgent problems to be solved.

In view of this, it is one of the important directions to predict algal bloom by fusing data-driven model with the mechanism model. First, with the classical mechanism model, the general evolution law of key variables can be determined and the modeling of nonlinear processes can be reduced. Then the data-driven modeling or parameter calibration of the unclear part is used to capture the hidden information with the accumulated monitoring data. The algae growth model (AGM) through the nutrient cycle of the current research is ripe for modeling [12]. However, the parameters of the current mechanism model are generally fixed by the intelligent optimization algorithm, the description of the dynamic characteristics of algal bloom is lacking. Therefore, this paper considers the dynamic parameters optimization of the mechanism model. Thus, the fusion of the mechanism model and the data-driven model can be fully realized. Specifically, based on the fixed parameters, the dynamic local adjustment rule is adopted to capture the time-varying and catastrophic changes of algal bloom. On the one hand, the original mechanism model with fixed parameters has a certain dynamics, reducing the complexity of modeling. On the other hand, the dynamic parameter increment value is learned through data-driven methods to describe complex features more accurately. Thus, the mechanism model with time-varying parameters can more effectively and flexibly describe the dynamic characteristics of algal bloom in a wider range of scales.

Adaptive dynamic programming (ADP) is an optimal control approach that combines reinforcement learning and neural network technology, which has been widely concerned since the introduction from Werbos [13]. The core idea is to approximate the control law and performance index function through the action network and critic network, and effectively solve the “dimensionality disaster” problem in the dynamic programming method [14]. At present, the ADP is theoretically used to study the optimal control of various time-delay systems, chaotic systems, and multi-agent systems [15], [16], [17]. In practice, the ADP has been widely used in optimal transportation control, smart grid management and other fields [18], [19]. The ADP also achieves online optimization of process parameters in the industry [20]. Here, the echo state network (ESN) for function approximation, as a special recurrent neural network, can process dynamic time series information and plays an important role in dealing with nonlinear optimal problems [21], [22]. It can be seen from this that the ADP, including neural network, can better solve the problem of algal bloom prediction with nonlinear characteristics, and has a great advantage in the dynamic optimization of mechanism model parameters.

In this paper, an algal bloom mechanism model with time-varying parameters via ESN-based ADP is proposed. First, the fixed parameters of the mechanism model are calibrated through covariance matrix adaptation evolutionary strategies (CMA-ES). Then, the incremental value of parameters is optimized by ADP. In ADP, the action network is used to approximate the nonlinear relationship between the time series feature and the parameter increment. The critic network is used to approximate the performance index function. In particular, implementing the above mapping relationship with ESN can enhance the dynamic memory ability of the time-varying mechanism model. The main contributions of the paper are listed as follows.

  • (1)

    To solve the problem of dynamic prediction of algal bloom, we apply the ADP approach based on the reinforcement learning idea to the parameter optimization of the mechanism model, which has practical application significance.

  • (2)

    We decompose the calibration of time-varying parameters into two parts: the fixed value and the incremental value. The former is solved iteratively by CMA-ES. The latter transforms the problem into an optimal control problem, which is handled by the ADP approach. This decomposition scheme can not only ensure the stability of the proposed model, but also improve the dynamic property of the original AGM. The most important thing is that its structure is simple, the principle is clear, and it is easy to apply to the actual lake and reservoir.

  • (3)

    To further improve the dynamic information processing ability of the action network from water quality data, we performed unsupervised feature extraction on the multivariate time series data with sliding window and used it as the input of the action network. In this way, the proposed method combines the advantages of the mechanism model and data-driven model, which further improves the dynamic information contained in time-varying parameters.

  • (4)

    The proposed method was validated on the North Canal and the Taihu Lake dataset, and the convergence was theoretically analyzed. It made the proposed method not only effective in application but also theoretically reliability.

The arrangements of subsequent sections of this paper are as follows. Section 2 describes the original AGM and proposes an ADP framework for improving the AGM with time-varying parameters. At the same time, it introduces the content of fixed parameters calibration using CMA-ES. Section 3 gives the specific steps of the ADP method on dynamic parameters optimization. Not only the iterative learning process based on the mechanism model is derived, but also the convergence analysis is given. In Section 4, we verify the effectiveness of the proposed method through the actual datasets of two lakes. Section 5 shows the conclusion.

Section snippets

Mechanism model

An algae growth model (AGM) denoted by nutrient cycle dynamics is as follows [12]. dcadt=GT(t)GI(t)GN(t)T1ca(t)dNdt=T2N(t)N0gNGN(t)ca(t)+dNca(t)where ca(t) (unit: μg/L) is the concentration of chlorophyll a, which is the indicating variable of the algal bloom [12]; N(t) (unit: mg/L) is the total nitrogen concentration, which characterizes the algae level of nutrients in the water body. As shown in Eq. (2), we further decompose GT(t), GI(t), and GN(t) in the mechanism model (1), fully

Background of TAGM

The dynamics of AGM with time-varying parameters can be written as ca(k+1)=ca(k)+Δtgmax+Δθ3(k)I(k)N(k)×1.066T(t)20I(t)+kI+Δθ4(k)N(k)+kN+Δθ5(k)T1+Δθ1(k)ca(k)N(k+1)=N(k)+ΔtT2+Δθ2(k)N(k)N0+Δθ6(k)gN+Δθ8(k)N(t)ca(k)N(t)+kN+Δθ5(k)+dN+Δθ7(k)ca(k)where Δθ(k)R8×1 is the incremental parameters of AGM. The new parameter vector θ(k)=θ0+Δθ(k) is still subject to its constraint range θminθ(k)θmax.

To ensure that the time-varying parameters of the mechanism model can change steadily, we set the

Experimental results

To verify the effectiveness of the TAGM for algal bloom prediction, we selected sensor data from the North Canal dataset in Beijing, China, and the Taihu Lake dataset in Jiangsu Province, China. There are 393 groups in the North Canal dataset and 1094 groups in the Taihu Lake dataset. In addition to the four variables included in the mechanism model (chlorophyll a, total nitrogen, illumination, and water temperature), the North Canal dataset contains five variables (phycocyanin, dissolved

Conclusions

In view of the necessity of algal bloom prediction in eutrophication management, an ESN-based ADP approach is proposed to optimize the parameters of the mechanism model. The TAGM through the ADP approach realize the effective fusion of the mechanism model and the data-driven model. This hybrid method not only achieves the precise modeling and accurate prediction of algal bloom, but also improves the function of dynamic information processing while retaining the interpretability of the mechanism

CRediT authorship contribution statement

Huiyan Zhang: Conceptualization, Methodology. Bo Hu: Writing – original draft, Writing – review & editing. Xiaoyi Wang: Funding acquisition, Supervision. Li Wang: Supervision. Jiping Xu: Software. Qian Sun: Funding acquisition. Zhiyao Zhao: Investigation.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This research was financially supported by National Social Science Fund of China (19BGL184), Beijing excellent talent training support project for young top-notch team (2018000026833TD01), National Natural Science Foundation of China (61802010, 61903008, 61703008), Beijing Municipal Natural Science Foundation (4194074), and the 2020 Graduate Research Capacity Improvement Program of Beijing Technology and Business University . Those supports are gratefully acknowledged.

Huiyan Zhang received the Ph.D. degree in Power system and its automation from Graduate School of Chinese Academy of Sciences, Beijing, China, in 2006. Her main research interests include subjective and objective information fusion, Bayesian estimation and big data tendency analysis.

References (40)

  • WangX. et al.

    Echo state networks regulated by local intrinsic plasticity rules for regression

    Neurocomputing

    (2019)
  • SongQ. et al.

    Effects of connectivity structure of complex echo state network on its prediction performance for nonlinear time series

    Neurocomputing

    (2010)
  • JaegerH. et al.

    Special issue on echo state networks and liquid state machines

    Neural Netw.

    (2007)
  • LiuF. et al.

    A boundedness result for the direct heuristic dynamic programming

    Neural Netw

    (2012)
  • SrivastavaA. et al.

    Status, alert system, and prediction of cyanobacterial bloom in South Korea

    Biomed Res. Int.

    (2015)
  • R. Upadhyay, N. Kumari, V. Rai, Wave of chaos and pattern formation in spatial predator–prey systems with holling type...
  • ValentinaA. et al.

    A novel application of an adaptable modeling approach to the management of toxic microalgal bloom events in coastal areas

    Harmful Algae

    (2017)
  • WangL. et al.

    An approach of improved dynamic deep belief nets modeling for algae bloom prediction

    Cluster Comput.

    (2019)
  • WangX. et al.

    A novel water quality mechanism modeling and eutrophication risk assessment method of lakes and reservoirs

    Nonlinear Dynam.

    (2019)
  • WangF. et al.

    Adaptive dynamic programming: an introduction

    IEEE Comput. Intell. Mag.

    (2009)
  • Cited by (3)

    • Criminal Law Risk Management and Prediction Method based on Echo State Network

      2023, International Journal of Advanced Computer Science and Applications

    Huiyan Zhang received the Ph.D. degree in Power system and its automation from Graduate School of Chinese Academy of Sciences, Beijing, China, in 2006. Her main research interests include subjective and objective information fusion, Bayesian estimation and big data tendency analysis.

    Bo Hu received the B.E. degree in automation from Beijing Technology and Business University, Beijing, China, in 2018. He is currently pursuing the master’s degree in control theory and control engineering with Beijing Technology and Business University. His main research interests include artificial neural network, reinforcement learning and time series analysis.

    Xiaoyi Wang received his Ph.D. degree in control theory and control engineering from School of Automation, Beijing Institute of Technology, Beijing, China in 2006. His current research interest covers 1) water environment modeling, optimization and decision-making, and optimal control; 2) risk assessment and early-warning of food supply chain.

    Li Wang received the Ph.D. degree in control science and engineering from Beihang University in 2011. She has been with School of Computer and Information Engineering, Beijing Technology and Business University as a Lecturer from 2011 to 2014 and as an Associate Professor since 2014. Her present major interest is modeling, analysis and prediction of environmental systems and big data tendency analysis.

    Jiping Xu was born in 1979, teacher in Computer and Information Engineering Academy, Associate Professor, Doctor, Youth Talents of State Governance in Beijing, member of Beijing Association of Automation. His current research area include intelligence fusion and Intelligent Service Platform.

    Qian Sun received the Ph.D. degree in instrument science and technology from Beihang University, in 2014. Since 2014 she has been with College of Computer and Information Engineering, Beijing Technology and Business University as a lecturer. Her research interests include camera sensor networks, video/image processing, and target tracking.

    Zhiyao Zhao received his Ph.D. degree in guidance, navigation and control from School of Automation Science and Electrical Engineering, Beihang University, Beijing, China in 2017. He has been a lecturer with Beijing Technology and Business University since 2017, and an associate professor since 2019. His current research interest covers prognostics and health management (PHM), stochastic hybrid systems (SHS) and multirotor unmanned aerial vehicles (Multirotor-UAV).

    View full text