Reservoir Computing in Reduced Order Modeling for Chaotic Dynamical Systems

Abstract

The mathematical concept of chaos was introduced by Edward Lorenz in the early 1960s while attempting to represent atmospheric convection through a two-dimensional fluid flow with an imposed temperature difference in the vertical direction. Since then, chaotic dynamical systems are accepted as the foundation of the meteorological sciences and represent an indispensable testbed for weather and climate forecasting tools. Operational weather forecasting platforms rely on costly partial differential equations (PDE)-based models that run continuously on high performance computing architectures. Machine learning (ML)-based low-dimensional surrogate models can be viewed as a cost-effective solution for such high-fidelity simulation platforms. In this work, we propose an ML method based on Reservoir Computing - Echo State Neural Network (RC-ESN) to accurately predict evolutionary states of chaotic systems. We start with the baseline Lorenz-63 and 96 systems and show that RC-ESN is extremely effective in consistently predicting time series using Pearson’s cross correlation similarity measure. RC-ESN can accurately forecast Lorenz systems for many Lyapunov time units into the future. In a practical numerical example, we applied RC-ESN combined with space-only proper orthogonal decomposition (POD) to build a reduced order model (ROM) that produces sequential short-term forecasts of pollution dispersion over the continental USA region. We use GEOS-CF simulated data to assess our RC-ESN ROM. Numerical experiments show reasonable results for such a highly complex atmospheric pollution system.

Appendices

A Non Linear Transformation in RC-ESN Hidden Space

Following the same approach proposed in [3], we defined three options for the nonlinear transformations $\mathcal {T}$:

$$\begin{aligned} \mathcal {T}_1: \ \ \tilde{r}_{ij} =&\ r_{i, j}, \ \ \mathrm {if} \mod (j, 2) = 0\\ \tilde{r}_{ij} =&\ r_{i, j}^2, \ \ \mathrm {if} \mod (j, 2) \ne 0\\ \mathcal {T}_2: \ \ \tilde{r}_{ij} =&\ r_{i, j}, \ \ \mathrm {if} \mod (j, 2) = 0\\ \tilde{r}_{ij} =&\ r_{i, j-1}\,r_{i, j-2}, \ \ \mathrm {if} \mod (j, 2) \ne 0 \ \mathrm {and} \,\, j> 1\\ \mathcal {T}_3: \ \ \tilde{r}_{ij} =&\ r_{i, j}, \ \ \mathrm {if} \mod (j, 2) = 0\\ \tilde{r}_{ij} =&\ r_{i, j-1}\,r_{i, j+1}, \ \ \mathrm {if} \mod (j, 2) \ne 0 \ \mathrm {and} \,\, j > 1 \end{aligned}$$

B Single Shot RC-ESN Loss Function Minimization

The least-squares problem stated in Eq. 5 aims at finding the reservoir-to-output matrix $\mathbf {W}_{out}\in \mathbb {R}^{n_{i} \times D}$. We can equivalently solve that problem by writing the following linear system [2]:

$$\begin{aligned} \mathbf {W}_{out}= \mathbf {X} \, \tilde{\mathbf {R}}^{T} (\tilde{\mathbf {R}} \tilde{\mathbf {R}}^{T} + \alpha \mathbf {I}), \end{aligned}$$

(12)

where $\mathbf {X} \in \mathbb {R}^{n_{i} \times N}$ is the matrix that stacks every input data $\mathbf {x}(t)$ shifted by one time step for all N time steps, and $\tilde{\mathbf {R}} \in \mathbb {R}^{D \times N}$ is the matrix that stacks every corresponding $\mathcal {T}$-transformed hidden states $\mathbf {\tilde{r}}(t)$. The parameter $\alpha$ is the ridge regularization, and $\mathbf {I}$ is the identity matrix.