Bias-Variance-Covariance Decomposition

The bias-variance-covariance decomposition is a theoretical result underlying ensemble learning algorithms. It is an extension of the bias-variance decomposition, for linear combinations of models. The expected squared error of the ensemble $\bar{f}(\mathbf{x})$ from a target d is:

$$\begin{array}{rcl}{\mathcal{E}}_{\mathcal{D}}\{{(\bar{f}(\mathbf{x}) - d)}^{2}\}& =&{ \overline{\mathrm{bias}}}^{2} + \frac{1}{T}\overline{\mathrm{var}} +{\Biggl (1 - \frac{1}{T}\Biggr )}\overline{\mathrm{covar}}\end{array}.$$

The error is composed of the average bias of the models, plus a term involving their average variance, and a final term involving their average pairwise covariance. This shows that while a single model has a two-way bias-variance tradeoff, an ensemble is controlled by a three-way tradeoff. This ensemble tradeoff is often referred to as the accuracy-diversity dilemma for an ensemble. See ensemble learning for more details.