Accuracy and Architecture Studies of Residual Neural Network Method for Ordinary Differential Equations

Abstract

In this paper, we investigate residual neural network (ResNet) method to solve ordinary differential equations. We verify the accuracy order of ResNet ODE solver matches the accuracy order of the data. Forward Euler, Runge–Kutta2 and Runge–Kutta4 finite difference schemes are adapted generating three learning data sets, which are applied to train three ResNet ODE solvers independently. The well trained ResNet solvers obtain 2nd, 3rd and 5th orders of one step errors and behave just as its counterpart finite difference method for linear and nonlinear ODEs with regular solutions. In particular, we carry out (1) architecture study in terms of number of hidden layers and neurons per layer to obtain optimal network structure; (2) target study to verify the ResNet solver is as accurate as its finite difference method counterpart; (3) solution trajectory simulations. A sequence of numerical examples are presented to demonstrate the accuracy and capability of ResNet solver.

Appendix A

In the appendix, we revisit the one-step error between target and the exact solution through interpolation polynomial approximation. For one step error, orders of $O(\Delta ^2)$, $O(\Delta ^3)$ and $O(\Delta ^5)$ are obtained for the first order forward Euler method (3.2), second order Runge–Kutta2 method (3.3) and fourth order Runge–Kutta4 method (3.4) with $\Delta$ as the step size.

Case I: ${\textbf {y}}_j^2$ obtained from Forward Euler method (3.2)

Given ${\textbf {y}}_j^1={\textbf {x}}_j(t_0)$, subtract the exact solution ${\textbf {x}}_j(t_0+\Delta )$ of (3.1) from ${\textbf {y}}_j^2$ of the forward Euler method (3.2), we have

$$\begin{aligned} \Vert {\textbf {y}}_j^2-{\textbf {x}}_j(t_0+\Delta )\Vert _2&=\left\| \int _{t_0}^{t_0+\Delta } \left( {\textbf {F}}({\textbf {x}}_j(t_0), t_0)- {\textbf {F}}({\textbf {x}}_j(t),t)\right) ~dt\right\| _2 \nonumber \\&= \left\| \int _{t_0}^{t_0+\Delta } \frac{d}{dt}{\textbf {F}}({\textbf {x}}_j(\xi (t)),\xi (t)) (t-t_0) ~dt\right\| _2 \nonumber \\&=\frac{\Delta ^2}{2}\left\| \frac{d}{dt}{\textbf {F}}({\textbf {x}}_j(\eta ),\eta ) \right\| _2\le C\Delta ^2. \end{aligned}$$

(A.1)

Here $\frac{d}{dt}{\textbf {F}}({\textbf {x}}(t),t)=\frac{\partial {\textbf {F}}}{\partial {\textbf {x}}}{\textbf {F}}+ \frac{\partial {\textbf {F}}}{\partial t}$ refers to the complete derivative to the t variable, with $\frac{\partial {\textbf {F}}}{\partial {\textbf {x}}}$ denoting the Jacobian matrix of the vector function ${\textbf {F}}$ on variable ${\textbf {x}}(t)$ and $\frac{d {\textbf {x}}}{dt}={\textbf {F}}$. Forward Euler method can be considered as a constant quadrature rule approximation to the integral of the ODE system (3.1). Weighted mean value theorem is applied to estimate the error term.

Case II: ${\textbf {y}}_j^2$ obtained from 2nd order Runge–Kutta method (3.3)

Again we have ${\textbf {y}}_j^1={\textbf {x}}_j(t_0)$. Subtract ${\textbf {x}}_j(t_0+\Delta )$ of (3.1) from ${\textbf {y}}_j^2$ of the second order Runge–Kutta method (3.3), we have

$$\begin{aligned} \Vert {\textbf {y}}_j^2-{\textbf {x}}_j(t_0+\Delta )\Vert _2&=\left\| \Delta \times \left( \frac{k_1+k_2}{2}\right) -\int _{t_0}^{t_0+\Delta } {\textbf {F}}({\textbf {x}}_j(t),t) ~dt\right\| _2 \\&\le \left\| \Delta \times \left( \frac{k_1+\widetilde{k_2}}{2}\right) -\int _{t_0}^{t_0+\Delta } {\textbf {F}}({\textbf {x}}_j(t),t) ~dt\right\| _2 +\frac{\Delta }{2}\Vert k_2-\widetilde{k_2}\Vert _2, \end{aligned}$$

where $k_2={\textbf {F}}({\textbf {y}}_j^1+\Delta k_1, t_0+\Delta )$, $k_1={\textbf {F}}({\textbf {y}}_j^1,t_0)$ and $\widetilde{k_2}={\textbf {F}}({\textbf {x}}_j(t_0+\Delta ), t_0+\Delta )$. With the $O(\Delta )$ local truncation error of the forward Euler method approximating ${\textbf {x}}_j(t_0+\Delta )$ and applying the Lipschitz continuity of ${\textbf {F}}$ of the dynamic system, we have

$$\begin{aligned} \Vert k_2-\widetilde{k_2}\Vert _2&= \left\| {\textbf {F}}({\textbf {y}}_j^1+\Delta k_1, t_0+\Delta )-{\textbf {F}}({\textbf {x}}_j(t_0+\Delta ), t_0+\Delta )\right\| _2 \\&\le L\Vert {\textbf {y}}_j^1+\Delta k_1-{\textbf {x}}_j(t_0+\Delta )\Vert _2 \le C\Delta ^2. \end{aligned}$$

Here C represents a generic constant. The error from the two-points quadrature rule can be estimated as

$$\begin{aligned}&\left\| \Delta \times \left( \frac{k_1+\widetilde{k_2}}{2}\right) -\int _{t_0}^{t_0+\Delta } {\textbf {F}}({\textbf {x}}_j(t),t) ~dt\right\| _2 =\left\| \int _{t_0}^{t_0+\Delta } ({\textbf {G}}_1(t)- {\textbf {F}}({\textbf {x}}_j(t),t)) ~dt\right\| _2 \\&\quad =\frac{1}{2}\left\| \frac{d^2}{dt^2}{\textbf {F}}({\textbf {x}}_j(\eta ),\eta )\right\| _2\left| \int _{t_0}^{t_0+\Delta } (t-t_0)\left( t-(t_0+\Delta )\right) ~dt\right| \\&\quad =\frac{\Delta ^3}{12}\left\| \frac{d^2}{dt^2}{\textbf {F}}({\textbf {x}}_j(\eta ),\eta )\right\| _2 \le C\Delta ^3. \end{aligned}$$

Combine the above arguments, we have

$$\begin{aligned} \Vert {\textbf {y}}_j^2-{\textbf {x}}_j(t_0+\Delta )\Vert _2\le C\Delta ^3. \end{aligned}$$

(A.2)

Here ${\textbf {G}}_1(t)$ denotes the linear interpolation polynomial that interpolates ${\textbf {F}}({\textbf {x}}(t),t)$ at $t_0$ and $t_0+\Delta$. And $\frac{d^2}{dt^2}{\textbf {F}}({\textbf {x}}(\cdot ),\cdot )$ denotes the complete second derivative of ${\textbf {F}}({\textbf {x}}(t),t)$ to t. This 2-stage Runge–Kutta method can be considered as a trapezoidal quadrature rule approximating the integration.

Case III: ${\textbf {y}}_j^2$ obtained from 4th order Runge–Kutta method (3.4)

With ${\textbf {y}}_j^1={\textbf {x}}_j(t_0)$ and subtract ${\textbf {x}}_j(t_0+\Delta )$ of (3.1) from ${\textbf {y}}_j^2$ of the fourth order Runge-Kutta method (3.4), we have

$$\begin{aligned} \Vert {\textbf {y}}_j^2-{\textbf {x}}_j(t_0+\Delta )\Vert _2&=\left\| \frac{\Delta (k_1+3k_2+3k_3+k_4)}{8}-\int _{t_0}^{t_0+\Delta } {\textbf {F}}({\textbf {x}}_j(t),t) ~dt\right\| _2 \\&\le \left\| \frac{\Delta (k_1+3\widetilde{k_2}+3\widetilde{k_3}+\widetilde{k_4)}}{8}-\int _{t_0}^{t_0+\Delta } {\textbf {F}}({\textbf {x}}_j(t),t) ~dt\right\| _2\\&\quad +\left\| \frac{\Delta (k_1+3k_2+3k_3+k_4)}{8}-\frac{\Delta (k_1+3\widetilde{k_2}+3\widetilde{k_3}+\widetilde{k_4)}}{8}\right\| _2. \end{aligned}$$

Terms of $k_2, k_3$ and $k_4$ are from the Runge–Kutta4 method (3.4), with $k_1={\textbf {F}}({\textbf {y}}_j^1,t_0)={\textbf {F}}({\textbf {x}}_j(t_0),t_0)$. We have $\widetilde{k_2}={\textbf {F}}({\textbf {x}}_j(t_0+\frac{\Delta }{3}),t_0+\frac{\Delta }{3})$, $\widetilde{k_3}={\textbf {F}}({\textbf {x}}_j(t_0+\frac{2\Delta }{3}),t_0+\frac{2\Delta }{3})$ and $\widetilde{k_4}={\textbf {F}}({\textbf {x}}_j(t_0+\Delta ),t_0+\Delta )$ introduced that $k_2, k_3$ and $k_4$ approximate. Rewrite the Runge–Kutta4 method of (3.4) as a one-step method, ${\textbf {y}}_j^2={\textbf {y}}_j^1+\Delta \Phi \left( t_0,{\textbf {y}}_j^1,{\textbf {F}}({\textbf {y}}_j^1),\Delta \right)$, we have

$$\begin{aligned}&\left\| \frac{\Delta (k_1+3k_2+3k_3+k_4)}{8}-\frac{\Delta (k_1+3\widetilde{k_2}+3\widetilde{k_3}+\widetilde{k_4)}}{8}\right\| _2 \\&\quad = \Delta \left\| \Phi \left( t_0,{\textbf {y}}_j^1,{\textbf {F}}({\textbf {y}}_j^1),\Delta \right) -\Phi \left( t_0,{\textbf {x}}_j(t_0),{\textbf {F}}({\textbf {x}}_j(t_0)),\Delta \right) \right\| _2 \le C\Delta ^5. \end{aligned}$$

Here C represents a generic constant. The error from the four-points quadrature rule can be estimated as

$$\begin{aligned}&\left\| \Delta \times \left( \frac{k_1+3\widetilde{k_2}+3\widetilde{k_3}+\widetilde{k_4}}{8}\right) -\int _{t_0}^{t_0+\Delta } {\textbf {F}}({\textbf {x}}_j(t),t) ~dt\right\| _2 \\&\quad = \left\| \int _{t_0}^{t_0+\Delta } ({\textbf {G}}_3(t)- {\textbf {F}}({\textbf {x}}_j(t),t)) ~dt\right\| _2 \\&\quad \le C\left\| \frac{d^4 {\textbf {F}}}{dt^4}\right\| _2 \left| \int _{t_0}^{t_0+\Delta } (t-t_0)(t-(t_0+\frac{\Delta }{3}))(t-(t_0+\frac{2\Delta }{3}))\left( t-(t_0+\Delta )\right) ~dt\right| \le C \Delta ^5. \end{aligned}$$

Again C represents a generic constant. Summarize the above arguments, we have

$$\begin{aligned} \Vert {\textbf {y}}_j^2-{\textbf {x}}_j(t_0+\Delta )\Vert _2\le C\Delta ^5. \end{aligned}$$

(A.3)

Here ${\textbf {G}}_3(t)$ denotes the cubic interpolation polynomial that interpolates ${\textbf {F}}({\textbf {x}}(t),t)$ at $t_0$, $t_0+\Delta /3$, $t_0+2\Delta /3$ and $t_0+\Delta$. And $\frac{d^4{\textbf {F}}}{dt^4}$ denotes the complete fourth derivative of ${\textbf {F}}({\textbf {x}}(t),t)$ to t variable at somewhere. This version of 4-stage Runge–Kutta method can be considered as the three-eighth Simpson quadrature rule approximating the integration.