An Information-Theoretic Framework for Improving Imperfect Dynamical Predictions Via Multi-Model Ensemble Forecasts

Abstract

This work focuses on elucidating issues related to an increasingly common technique of multi-model ensemble (MME) forecasting. The MME approach is aimed at improving the statistical accuracy of imperfect time-dependent predictions by combining information from a collection of reduced-order dynamical models. Despite some operational evidence in support of the MME strategy for mitigating the prediction error, the mathematical framework justifying this approach has been lacking. Here, this problem is considered within a probabilistic/stochastic framework which exploits tools from information theory to derive a set of criteria for improving probabilistic MME predictions relative to single-model predictions. The emphasis is on a systematic understanding of the benefits and limitations associated with the MME approach, on uncertainty quantification, and on the development of practical design principles for constructing an MME with improved predictive performance. The conditions for prediction improvement via the MME approach stem from the convexity of the relative entropy which is used here as a measure of the lack of information in the imperfect models relative to the resolved characteristics of the truth dynamics. It is also shown how practical guidelines for MME prediction improvement can be implemented in the context of forced response predictions from equilibrium with the help of the linear response theory utilizing the fluctuation–dissipation formulas at the unperturbed equilibrium. The general theoretical results are illustrated using exactly solvable stochastic non-Gaussian test models.

Appendices

Appendix 1: Some Simple Proofs of General Results from Sect. 3

Here, we complement the discussion of Sect. 3 by providing simple derivations and proofs of the facts established in that section.

Information Criterion II in (12) Derivation of this criterion relies on the convexity properties (10) of the relative entropy (e.g., Cover and Thomas 2006), which leads to the following upper bound on the lack of information in the MME mixture density $\pi ^{\textsc {mme}}_t$ (1) relative to the least-biased estimate of the marginal truth density $\pi _t$

$$\begin{aligned} \mathcal {P}(\pi _t^{\textsc {l}},\pi ^{\textsc {mme}}_t)= & {} \mathcal {P}(\pi _t^{\textsc {l}},\textstyle \sum _i\alpha _i\pi ^{{\textsc {m}}_i}_t)\leqslant \sum _i\alpha _i\mathcal {P}(\pi _t^{\textsc {l}},\pi ^{{\textsc {m}}_i}_t)\nonumber \\= & {} \sum _{i\ne {\diamond }}\alpha _i\mathcal {P}(\pi _t^{\textsc {l}},\pi ^{{\textsc {m}}_i}_t)+\alpha _{\diamond }\mathcal {P}(\pi _t^{\textsc {l}},\pi ^{{\textsc {m}}_{\diamond }}_t). \end{aligned}$$

(54)

where $\alpha _i\geqslant 0$ and $\sum _i\alpha _i = 1$ so that

$$\begin{aligned} {\mathcal {P}}\left( \pi _t^{\textsc {l}},\pi ^{\textsc {mme}}_t\right) -{\mathcal {P}} \left( \pi _t^{\textsc {l}},\pi ^{{\textsc {m}}_{\diamond }}_t\right) \leqslant \sum _{i\ne {\diamond }}\alpha _i{\mathcal {P}}\left( \pi _t^{\textsc {l}},\pi ^{{\textsc {m}}_i}_t\right) +\left( \alpha _{\diamond }-1\right) {\mathcal {P}}\left( \pi _t^{\textsc {l}},\pi ^{{\textsc {m}}_{\diamond }}_t\right) ,\nonumber \\ \end{aligned}$$

(55)

where we used the fact that ${\mathcal {P}}\geqslant 0$. Clearly, the information criterion in (7) is always satisfied when the right-hand side in (55) satisfies

(56)

which, after rearranging terms, gives the sufficient condition in (12) with $\varDelta =0$, i.e.,

$$\begin{aligned}&\sum _{i\ne {\diamond }}\beta _i\,{\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}},\pi ^{{\textsc {m}}_i}) <{\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}},\pi ^{{\textsc {m}}_{\diamond }}),\nonumber \\&\beta _i = \alpha _i\,(1-\alpha _{\diamond })^{-1}, \quad \sum _{i\ne {\diamond }} \beta _i=1. \end{aligned}$$

(57)

This sufficient condition is too restrictive when ${\textsc {m}}_{\diamond }$ coincides with the best imperfect model ${\textsc {m}}^*_{{\mathcal {I}},{\textsc {l}}}$ in (8) since there is no non-trivial MME satisfying (56). Based on the basic convexity properties of the relative entropy illustrated in Fig. 2 and Fact 1, it is clear that unless ${\mathcal {P}}_{\mathcal {I}}(\pi ^{\textsc {l}},\pi ^{{\textsc {m}}^*_{{\mathcal {I}},{\textsc {l}}}})=0$, an MME with a smaller error does exist and the condition (57) needs to be relaxed in order to be applicable in such cases. The uncertainty parameter $\varDelta$ in

$$\begin{aligned}&\sum _{i\ne {\diamond }}\beta _i\,{\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}},\pi ^{{\textsc {m}}_i}) <{\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}},\pi ^{{\textsc {m}}_{\diamond }})+\varDelta ,\nonumber \\&\quad \beta _i = \alpha _i\,(1-\alpha _{\diamond })^{-1}, \quad \sum _{i\ne {\diamond }} \beta _i=1, \end{aligned}$$

(58)

allows for including models in the ensemble with error ${\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}},\pi ^{{\textsc {m}}^*_{{\mathcal {I}},{\textsc {l}}}})\leqslant {\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}},\pi ^{\textsc {m}})<{\mathcal {P}}_{\mathcal {I}}(\pi ^{{\textsc {l}}}, \pi ^{{\textsc {m}}^*_{{\mathcal {I}},{\textsc {l}}}})+\varDelta$ so that the MME prediction error is $0\leqslant {\mathcal {P}}_{\mathcal {I}}(\pi ^{\textsc {l}},\pi ^{{\textsc {mme}}}) \leqslant {\mathcal {P}}_{\mathcal {I}}(\pi ^{\textsc {l}},\pi ^{{\textsc {m}}_{\diamond }})+\varDelta$, as illustrated in Fig. 2c.

Proof of Fact 3

The proof is straightforward and follows by a direct calculation consisting of two steps:

• (1) We start by rewriting the condition (12) in terms of the least-biased densities defined in (14) which leads to

  $$\begin{aligned}&{\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t,\pi ^{{\textsc {m}}_{\diamond },{\textsc {l}}_2}_t)+\varDelta > \sum _{i\ne {\diamond }}\beta _i\,{\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t,\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t)\nonumber \\&\quad +\,\sum _{i\ne {\diamond }}\beta _i\,{\mathbb {E}}^{\pi ^{{\textsc {l}}_1}} \Big [\log \frac{\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}}{\pi ^{{\textsc {m}}_i}}-\log \frac{\pi ^{{\textsc {m}}_{\diamond },{\textsc {l}}_2}}{\pi ^{{\textsc {m}}_{\diamond }}} \Big ]; \end{aligned}$$

  (59)

  note that this last term vanishes identically when $\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}=\pi ^{{\textsc {m}}_i}$ and the MME contains only least-biased models.

• (2) Next, we notice that the relative entropy between two least-biased densities $\pi ^{{\textsc {l}}_1}_t$ and $\pi ^{{\textsc {m}},{\textsc {l}}_2}_t$ is given by

  $$\begin{aligned} {\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t,\pi ^{{\textsc {m}},{\textsc {l}}_2}_t)&=\log C^{\textsc {m}}_t+{\pmb {\theta }}^{{\textsc {m}}}_t\cdot \bar{\pmb {E}}_t -\big (\log C_t + {\pmb {\theta }}_t\cdot \bar{\pmb {E}}_t\,\big )\nonumber \\&=\log \frac{\;\;C^{\textsc {m}}_t}{C_t}+ ({\pmb {\theta }}^{{\textsc {m}}}_t\!{-}\,\,{\pmb {\theta }}_t)\cdot \bar{\pmb {E}}_t, \end{aligned}$$

  (60)

  where ${\overline{\pmb {E}}}_t$ is the vector of expectations of the functionals $E_i$ defined in (15) with respect to the truth marginal density $\pi _t$, and the Lagrange multipliers in (14), ${\pmb {\theta }}_t = {\pmb {\theta }}\big ({\overline{\pmb {E}}}_t\big ),\,{\pmb {\theta }}^{{\textsc {m}}}_t = {\pmb {\theta }}^{{\textsc {m}}}\big ({\overline{\pmb {E}}}^{\textsc {m}}_t\big )$, are defined as

  $$\begin{aligned} {\pmb {\theta }_t} = (\theta _1(t),\dots ,\theta _{{\textsc {l}}_1}(t))^\text {T}, \quad \,\, {\pmb {\theta }}^{{\textsc {m}}}_t = (\theta _1^{\textsc {m}}(t),\dots ,\theta ^{\textsc {m}}_{{\textsc {l}}_2}(t),0,\dots ,0_{{\textsc {l}}_1})^\text {T}, \quad \,\, {\textsc {l}}_1\geqslant {\textsc {l}}_2, \end{aligned}$$

  while the normalization constants in the least-biased densities are $C_t = C\big ({\overline{\pmb {E}}}_t\big ),\,C^{\textsc {m}}_t =C^{\textsc {m}}\big ({\overline{\pmb {E}}}^{\textsc {m}}_t\big )$.

The condition in (16) is obtained by combining (59) with (60).

Proof of Fact 4

The condition in (21) for improvement in the prediction skill via MME in the context of initial value problem can be obtained as follows: Consider the representation of the true expected values ${\overline{\pmb {E}}}_t$ of the functionals $E_i(\pmb {u})$ with respect to the truth marginal density $\pi _t(\pmb {u})$ in the form

$$\begin{aligned} {\overline{\pmb {E}}}_t = {\overline{\pmb {E}}}_{0}+\delta \tilde{{\overline{\pmb {E}}}}_t, \qquad {\pmb {\theta }}_t = {\pmb {\theta }}_{0}+\delta \tilde{\pmb {\theta }}_t\big ({\overline{\pmb {E}}}_t\big ), \qquad \tilde{{\overline{\pmb {E}}}}_{t=0} =\tilde{\pmb {\theta }}_{t=0}=0, \end{aligned}$$

(61)

these are smooth at $\delta =0$ when the decomposition $\pi _t=\pi _0+\delta {\tilde{\pi }}_t$ is smooth at $\delta =0$ which holds under minimal hypothesis described in Hairer and Majda (2010) so that

$$\begin{aligned} \qquad C_t= C_0\big (1- \delta \tilde{\pmb {\theta }}_t\cdot {\overline{\pmb {E}}}_{0}\big )+{\mathcal {O}}\big (\delta ^2\big ), \end{aligned}$$

(62)

The lack of information in (12) between the least-biased approximation of the truth $\pi ^{{\textsc {l}}_1}_t$ and the imperfect model density $\pi ^{{\textsc {m}}_i}_t$ can be written as

$$\begin{aligned} {\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t,\pi ^{{\textsc {m}}_i}_t)={\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t, \pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t)+\int \!\!{\mathrm{d}}\pmb {u}\,\pi ^{{\textsc {l}}_1}_t\log \frac{\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t}{\pi ^{{\textsc {m}}_i}_t}, \end{aligned}$$

(63)

similarly to the result leading to (59). The lack of information in the perturbed least-biased density, $\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t$, of the imperfect model relative to the least-biased perturbation of the truth, $\pi ^{{\textsc {l}}_1}_t$, can be expressed through (65)–(68) in the following form

$$\begin{aligned} {\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t,\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t) = {\mathcal {P}}(\pi _0^{{\textsc {l}}_1},\pi _t^{{\textsc {m}}_i,{\textsc {l}}_2}) +({\pmb {\theta }}^{{\textsc {m}}_i}_t-{\pmb {\theta }}_0)\cdot \delta \tilde{{\overline{\pmb {E}}}}_t +{\mathcal {O}}(\delta ^2). \end{aligned}$$

(64)

Substituting (64) into (16) leads to the desired condition (21). $\square$

Proof of Fact 5

The condition in (25) for improvement in the prediction skill via MME obtained by perturbing single-model predictions can be obtained as follows: Consider the condition (16) in the case when the ensemble members ${\textsc {m}}_i\in {\mathcal {M}}$ are obtained from the single model ${\textsc {m}}_{\diamond }\in {\mathcal {M}}$ through perturbing some parameters of the single model; we assume that the statistics of the model depend smoothly on these parameters and that the perturbations are non-singular (which required minimal assumptions Hairer and Majda 2010 of hypoelliptic noise in the truth dynamics) so that the evolution of the statistical moments ${\overline{\pmb {E}}}_t^{{\textsc {m}}_i}$ and their functions in the least-biased densities (60) of the ensemble members can be written, for $\epsilon \ll 1$, as

$$\begin{aligned}&(a) \;\;{\overline{\pmb {E}}}_t^{{\textsc {m}}_i,\epsilon } = {\overline{\pmb {E}}}_t^{{\textsc {m}}_{\diamond }}+\epsilon \,\tilde{{\overline{\pmb {E}}}}_t^{{\textsc {m}}_i},\end{aligned}$$

(65)

$$\begin{aligned}&(b)\;\;{\pmb {\theta }}_t^{{\textsc {m}}_i,\epsilon } = {\pmb {\theta }}_{t}^{{\textsc {m}}_{\diamond }}+\epsilon \,{\tilde{\pmb {\theta }}}^{{\textsc {m}}_i}_t \big (\,{\overline{\pmb {E}}}_t^{{\textsc {m}}_i}\big )+{\mathcal {O}}\big (\epsilon ^2\big ),\end{aligned}$$

(66)

$$\begin{aligned}&(c) \;\;C^{{\textsc {m}}_i,\epsilon }_t = C^{{\textsc {m}}_{\diamond }}_t\big (1- \epsilon \,\tilde{\pmb {\theta }}_t^{{\textsc {m}}_i}\cdot {\overline{\pmb {E}}}_{t}^{{\textsc {m}}_{\diamond }}\big )+{\mathcal {O}}\big (\epsilon ^2\big ), \end{aligned}$$

(67)

where

$$\begin{aligned} {\tilde{\pmb {\theta }}}_t^{{\textsc {m}}_i} = \Big ({\overline{\pmb {E}}}_t^{{\textsc {m}}_{\diamond }}\cdot \!\nabla \theta _1^{{\textsc {m}}_i} |_{\varepsilon =0},\;\,{\overline{\pmb {E}}}_t^{{\textsc {m}}_{\diamond }}\cdot \!\nabla \theta _2^{{\textsc {m}}_i}|_{\varepsilon =0},\ldots ,\,{\overline{\pmb {E}}}_t^{{\textsc {m}}_{\diamond }} \cdot \!\nabla \theta _{{\textsc {l}}_1}^{{\textsc {m}}_i}|_{\varepsilon =0}\Big )^\text {T}. \end{aligned}$$

(68)

The lack of information in the perturbed least-biased density, $\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t$, of the imperfect model relative to the least-biased perturbation of the truth, $\pi ^{{\textsc {l}}_1}_t$, can be expressed through (65)–(68) in the following form

$$\begin{aligned} {\mathcal {P}}(\pi ^{{\textsc {l}}_1}_t,\pi ^{{\textsc {m}}_i,{\textsc {l}}_2}_t)&=\log \big (C^{{\textsc {m}}_i}_t/C_t\big )+({\pmb {\theta }}^{{\textsc {m}}_i}_t{-} \,{\pmb {\theta }}_t)\,{\cdot }\,\bar{\pmb {E}}_t\nonumber \\&= {\mathcal {P}}(\pi _t^{{\textsc {l}}_1},\pi _t^{{\textsc {m}}_{\diamond }})+\epsilon \,\tilde{\pmb {\theta }}^{{\textsc {m}}_i}_t\cdot ({\overline{\pmb {E}}}_t-{\overline{\pmb {E}}}_t^{{\textsc {m}}_{\diamond }}) +{\mathcal {O}}\big (\epsilon ^2\big ), \end{aligned}$$

(69)

which is obtained by combining (65)–(67). Substituting (69) into the general condition (16) leads to the desired condition (25). $\square$

Proof of Fact 6

The proof of the condition (31) is simple but tedious and follows from the short-time asymptotic expansion of the relative entropy between the Gaussian truth and the Gaussian models. Consider the state vector $\pmb {u}\in {\mathbb {R}}^K$ for resolved dynamics and assume that short times the statistics of the Gaussian truth density $\pi _t^{{\textsc {g}}}={\mathcal {N}}(\pmb {\mu }_t,R_t)$ and of the Gaussian model density $\pi ^{{\textsc {m}}_i}_t={\mathcal {N}}(\pmb {\mu }_t^{{\textsc {m}}_i},R_t^{{\textsc {m}}_i})$ are

$$\begin{aligned} \pmb {\mu }_t = \pmb {\mu }_0+\delta \, {\tilde{\pmb {\mu }}}_t, \qquad R_t = R_0+\delta {\tilde{R}}_t, \qquad \delta {\tilde{\pmb {\mu }}}_0=\delta {\tilde{R}}_0=0, \end{aligned}$$

(70)

and

$$\begin{aligned} \pmb {\mu }_t^{{\textsc {m}}_i} = \pmb {\mu }_0^{{\textsc {m}}_i}+\delta \, {\tilde{\pmb {\mu }}}_t^{{\textsc {m}}_i}, \qquad R_t^{{\textsc {m}}_i} = R_0^{{\textsc {m}}_i}+\delta {\tilde{R}}_t^{{\textsc {m}}_i}, \qquad \delta {\tilde{\pmb {\mu }}}_0^{{\textsc {m}}_i}=\delta {\tilde{R}}_0^{{\textsc {m}}_i}=0. \end{aligned}$$

(71)

Then, the relative entropy between the Gaussian truth density $\pi _t^{{\textsc {g}}}$ and a Gaussian model density $\pi ^{{\textsc {m}}_i}_t$

$$\begin{aligned} {\mathcal {P}}(\pi _t^{\textsc {g}},\pi _t^{{\textsc {m}}_i}) =\textstyle \frac{1}{2}(\varDelta \pmb {\mu }_t^i)^T(R_t^{{\textsc {m}}_i})^{-1} \varDelta \pmb {\mu }_t^i+\textstyle \frac{1}{2}\Big [{\mathrm{tr}} \big [ R_t(R_t^{{\textsc {m}}_i})^{-1}\big ]-\ln \det \big [ R_t(R_t^{{\textsc {m}}_i})^{-1}\big ]-K\Big ], \end{aligned}$$

(72)

with $\varDelta \pmb {\mu }_t^i:=\pmb {\mu }_t-\pmb {\mu }_t^{{\textsc {m}}_i}$ can be expressed as

$$\begin{aligned} {\mathcal {P}}(\pi _t^{\textsc {g}},\pi _t^{{\textsc {m}}_i}) = {\mathcal {P}}(\pi _0^{\textsc {g}},\pi _0^{{\textsc {m}}_i})+\delta (X^\mu +X^R)+\delta ^2(Y^\mu +Y^{\mu ,R}+Y^{R,R})+{\mathcal {O}}(\delta ^3), \end{aligned}$$

(73)

which is valid at times short enough so that the changes in moments $\delta {\tilde{\pmb {\mu }}}, \delta {\tilde{R}},\,\delta {\tilde{\pmb {\mu }}}^{{\textsc {m}}_i}, \delta {\tilde{R}}^{{\textsc {m}}_i}$ are small; the respective coefficients in (73) are given by

$$\begin{aligned} X^{\mu }= & {} \textstyle \frac{1}{2}\Big [ (\varDelta \pmb {\mu }_0^i)^T(R_0^{{\textsc {m}}_i})^{-1} \varDelta {\tilde{\pmb {\mu }}}_t^i+(\varDelta {\tilde{\pmb {\mu }}}_t^i)^T(R_0^{{\textsc {m}}_i})^{-1} \varDelta \pmb {\mu }_0^i\Big ],\\ X^R= & {} -\textstyle \frac{1}{2}(\varDelta \pmb {\mu }_0^i)^T(R_0^{{\textsc {m}}_i})^{-1} {\tilde{R}}_t^{{\textsc {m}}_i}(R_0^{{\textsc {m}}_i})^{-1} \varDelta \pmb {\mu }_0^i \!+\!\textstyle \frac{1}{2}\,{\mathrm{tr}}\, \Big [({\mathcal {I}}-R_0(R_0^{{\textsc {m}}_i})^{-1}){\tilde{R}}_t^{{\textsc {m}}_i} (R_0^{{\textsc {m}}_i})^{-1}\Big ]\nonumber \\&+\,\textstyle \frac{1}{2}\,{\mathrm{tr}}\, \Big [ {\tilde{R}}_t(R_0^{{\textsc {m}}_i})^{-1}\Big ], \\ Y^{\mu ,\mu }= & {} \textstyle \frac{1}{2}(\varDelta {\tilde{\pmb {\mu }}}_t^i)^T (R_0^{{\textsc {m}}_i})^{-1} \varDelta {\tilde{\pmb {\mu }}}_t^i,\\ Y^{\mu ,R}= & {} \textstyle \frac{1}{2}\Big [ (\varDelta \pmb {\mu }_0^i)^T (R_0^{{\textsc {m}}_i})^{-1}{\tilde{R}}_t^{{\textsc {m}}_i}(R_0^{{\textsc {m}}_i})^{-1} \varDelta {\tilde{\pmb {\mu }}}_t^i+(\varDelta {\tilde{\pmb {\mu }}}_t^i)^T (R_0^{{\textsc {m}}_i})^{-1}{\tilde{R}}_t^{{\textsc {m}}_i}(R_0^{{\textsc {m}}_i})^{-1} \varDelta \pmb {\mu }_0^i \Big ],\\ Y^{R,R}= & {} \textstyle \frac{1}{2}\Big [ (\varDelta \pmb {\mu }_0^i)^T (R_0^{{\textsc {m}}_i})^{-1}{\tilde{R}}^{{\textsc {m}}_i}_t (R_0^{{\textsc {m}}_i})^{-1} {\tilde{R}}^{{\textsc {m}}_i}_t(R_0^{{\textsc {m}}_i})^{-1}\varDelta \pmb {\mu }_0^i\\&-\,\textstyle \frac{1}{2}\,{\mathrm{tr}}\, \Big [({\mathcal {I}}-R_0 (R^{{\textsc {m}}_i}_0)^{-1})({\tilde{R}}^{{\textsc {m}}_i}_t(R_0^{{\textsc {m}}_i})^{-1})^2\Big ] -\textstyle \frac{1}{2}\,{\mathrm{tr}}\, \Big [{\tilde{R}}_t(R_0^{{\textsc {m}}_i})^{-1} {\tilde{R}}^{{\textsc {m}}_i}_t(R_0^{{\textsc {m}}_i})^{-1}\Big ]\nonumber \\&+\,\textstyle \frac{1}{4} \Big ({\mathrm{tr}} \big [ {\tilde{R}}^{{\textsc {m}}_i}_t(R_0^{{\textsc {m}}_i})^{-1}\big ] \Big )^2. \end{aligned}$$

For correct initial conditions, $\pmb {\mu }^{{\textsc {m}}_i}_0=\pmb {\mu }_0,\,R_0^{{\textsc {m}}_i}=R_0$, the above formulas simplify to

$$\begin{aligned} X^R&= \textstyle \frac{1}{2}\,{\mathrm{tr}}\, \Big [ {\tilde{R}}_t(R_0)^{-1}\Big ], \quad Y^{\mu ,\mu } = \textstyle \frac{1}{2}(\varDelta {\tilde{\pmb {\mu }}}_t^i)^T(R_0)^{-1} \varDelta {\tilde{\pmb {\mu }}}_t^i,\end{aligned}$$

(74)

$$\begin{aligned} Y^{R,R}&=-\textstyle \frac{1}{2}{\mathrm{tr}} \Big [{\tilde{R}}_t(R_0)^{-1}{\tilde{R}}^{{\textsc {m}}_i}_t(R_0)^{-1}\Big ] +\textstyle \frac{1}{4}\Big ({\mathrm{tr}} \big [ {\tilde{R}}^{{\textsc {m}}_i}_t(R_0)^{-1}\big ]\Big )^2, \end{aligned}$$

(75)

with the remaining coefficients identically zero. Substituting the relative entropy between ${\mathcal {P}}(\pi _t^{\textsc {g}},\pi _t^{{\textsc {m}}_i})$ in the form (73) with the coefficients (74)–(75) into the general necessary condition (12) for improving the prediction via MME yields the condition (31). $\square$

Proof of Fact 7

We assume that the perturbations of the equilibrium truth and model densities are smooth in response to the forcing perturbations so that the perturbed densities $\pi ^\delta _t = \pi _\mathrm{eq}+\delta {\tilde{\pi }}_t$ are differentiable at $\delta =0$; this holds under relatively mild assumptions hypoelliptic noise as shown in (Hairer and Majda 2010). Thus, based on the linear response theory combined with the fluctuation–dissipation formulas (e.g., Majda et al. 2005), the density perturbations remain small for sufficiently small external perturbations which also implies that the moment perturbations remain small for all time. Derivation of the condition (78) relies on the smallness of the moment perturbations which allows for an asymptotic expansion of the relative entropy as in (73) but with $\pmb {\mu }_0=\pmb {\mu }_\mathrm{eq}=\pmb {\mu }^{{\textsc {m}}_i}_\mathrm{eq},\,R_0=R_\mathrm{eq}=R^{{\textsc {m}}_i}_\mathrm{eq}$ which leads to expansion coefficients in (73)

$$\begin{aligned} X^R&= \textstyle \frac{1}{2}{\mathrm{tr}} \Big [ {\tilde{R}}_t(R_\mathrm{eq})^{-1}\Big ], \quad Y^{\mu ,\mu } = \textstyle \frac{1}{2}({\tilde{\pmb {\mu }}}_t -{\tilde{\pmb {\mu }}}_t^{{\textsc {m}}_i})^T(R_\mathrm{eq})^{-1} ({\tilde{\pmb {\mu }}}_t-{\tilde{\pmb {\mu }}}_t^{{\textsc {m}}_i}),\end{aligned}$$

(76)

$$\begin{aligned} Y^{R,R}&=-\textstyle \frac{1}{2}{\mathrm{tr}} \Big [{\tilde{R}}_t (R_\mathrm{eq})^{-1}{\tilde{R}}^{{\textsc {m}}_i}_t(R_\mathrm{eq})^{-1}\Big ]+\textstyle \frac{1}{4}\Big ({\mathrm{tr}} \big [ {\tilde{R}}^{{\textsc {m}}_i}_t(R_\mathrm{eq})^{-1} \big ]\Big )^2, \end{aligned}$$

(77)

with the remaining coefficients identically zero. The general condition for improvement in forced response prediction via MME in the Gaussian framework is

$$\begin{aligned} D_{\pmb {\beta },{\mathcal {I}}}(\{{\tilde{\pmb {\mu }}}-{\tilde{\pmb {\mu }}}^{{\textsc {m}}_i}\}) +E_{\pmb {\beta },{\mathcal {I}}}(\{{\tilde{R}}^{{\textsc {m}}_i}\})+F_{\pmb {\beta },{\mathcal {I}}}({\tilde{R}}, \{{\tilde{R}}^{{\textsc {m}}_i}\})+\varDelta +{\mathcal {O}}(\delta )>0, \end{aligned}$$

(78)

where

which is very similar to the condition in FACT 6 except that there is no short-time constraint due to the fact that the moment perturbations remain small in time under the above assumptions. Finally, the simplified result (32) in Fact 7 of Sect. 3.2 is obtained by taking into account that the response is due to the forcing perturbations in linear Gaussian systems (26) so that ${\tilde{R}}^{{\textsc {m}}_i}_t=0$ so that $X^R=Y^{R,R}=0$ in (76), (77) and only $D_{\pmb {\beta },{\mathcal {I}}}$, which is independent of the truth response in the covariance, remains in (78).

$\square$

Appendix 2: Further Details of Associated with the Sufficient Conditions for Imperfect Prediction Improvement Via MME

In Sect. 3.1.1, we discussed the condition (12) for improving imperfect predictions via MME in the least-biased density representation (16). Here, we discuss the same condition in terms of general perturbations of probability densities which provides additional insight into the essential features of MME with improved prediction skill. In particular, we show that it is difficult to improve the short-term predictive skill via MME containing models with incorrect statistical initial conditions.

The formulation presented below relies on relatively weak assumptions that the truth and model densities can be written as

$$\begin{aligned} \pi ^{{\textsc {l}}}_t = \pi _{0}^{\textsc {l}}+\delta {\tilde{\pi }}_t^{{\textsc {l}}}, \qquad \pi ^{{\textsc {m}}}_t = \pi _{0}^{\textsc {m}}+\delta {\tilde{\pi }}_t^{{\textsc {m}}}, \qquad {\tilde{\pi }}_0^{{\textsc {l}}}={\tilde{\pi }}_0^{{\textsc {m}}}=0, \quad \int {\tilde{\pi }}_t^{{\textsc {l}}} {\mathrm{d}}\pmb {u}= \int {\tilde{\pi }}_t^{{\textsc {l}}}{\mathrm{d}}\pmb {u}=0, \end{aligned}$$

(79)

The above decomposition is always possible for the non-singular initial value problem; in the case of the forced response prediction from equilibrium (i.e., when $\pi _{0}^{\textsc {l}}=\pi _\mathrm{eq}^{\textsc {l}},\,\pi _{0}^{\textsc {m}}=\pi _\mathrm{eq}^{\textsc {m}}$), such a decomposition exists for $\delta \ll 1$ under the minimal assumptions of hypoelliptic noise (Hairer and Majda 2010). The possibility of estimating the evolution of statistical moments of the truth density $\pi _t$ in the case of predicting the forced response within the framework of linear response theory combined with the fluctuation–dissipation approach makes this framework particularly important in this case (see Majda et al. 2005, 2010b, a; Abramov and Majda 2007; Gritsun et al. 2008; Majda and Gershgorin 2010, 2011a, b)

Fact

Assume the decomposition (79) of the truth and model densities exists as discussed above. Then, the condition (12) for prediction improvement through MME has the following form

$$\begin{aligned} {\fancyscript{A}}_{\pmb {\beta }}\Big (\pi _{0}^{\textsc {l}},\{\pi _{0}^{{\textsc {m}}_i}\}\Big )+\delta {\fancyscript{B}}_{\pmb {\beta },{\mathcal {I}}}\Big (\pi ^{\textsc {l}},\{\pi ^{{\textsc {m}}_i}\}\Big )+\delta ^2 {\fancyscript{C}}_{\pmb {\beta },{\mathcal {I}}}\Big (\pi ^{\textsc {l}},\{\pi ^{{\textsc {m}}_i}\}\Big )+{\tilde{\varDelta }}>0, \end{aligned}$$

(80)

where

with the weights $\beta _i$ defined in (12). The following particular cases of the condition (80) for improving the predictions via the MME approach are worth noting in this general representation:

• Initial (statistical) conditions in all models of MME are consistent with the least-biased estimate of the truth; i.e., $\pi ^{{\textsc {m}}_i}_{0}=\pi _{0}^{\textsc {l}}$. In such a case, we have ${\fancyscript{A}}_{\pmb {\beta }}=0,\,{\fancyscript{B}}_{\pmb {\beta },{\mathcal {I}}}=0$ and the condition (80) for improvement in prediction via MME simplifies to

  

  (81)

  In the case of forced response predictions, perturbation of the truth density ${\tilde{\pi }}_t^{\textsc {l}}$ can be estimated from the statistics on the unperturbed equilibrium through the linear response theory and fluctuation–dissipation formulas exploiting only the unperturbed equilibrium information (Majda et al. 2005, 2010b, a; Abramov and Majda 2007; Gritsun et al. 2008; Majda and Gershgorin 2010, 2011a, b).

• Initial model densities in MME perturbed relative to the least-biased estimate of the truth; i.e., $\pi ^{{\textsc {m}}_i}_{0}=\pi _{0}^{\textsc {l}}+\epsilon \,{\tilde{\pi }}^{{\textsc {m}}_i}_{0},\,\pi ^{{\textsc {m}}_{\diamond }}_{0}=\pi _{0}^{\textsc {l}}$. In such a case, all terms in (80) are non-trivial but they can be written as

  

  (82)
  

  (83)
  

  (84)

  where $\{B_m\},\{C_m\},\,m=1,2,3$ are functions of ${\tilde{\pi }}^{{\textsc {m}}_i}_0,{\tilde{\pi }}^{{\textsc {m}}_{\diamond }}_0, {\tilde{\pi }}^{\textsc {l}}_0$ and $\epsilon$. Note that unless $\epsilon =0$ (so that $\pi ^{{\textsc {m}}_i}_{0}=\pi _{0}^{\textsc {l}}$), it is difficult to improve the prediction skill at short times within the MME framework since at $t=0$, we have ${\fancyscript{B}}_{\pmb {\beta },{\mathcal {I}}}={\fancyscript{C}}_{\pmb {\beta },{\mathcal {I}}}=0$ and ${\fancyscript{A}}_{\pmb {\beta }}<0$ in (80).