Bias-Variance-Covariance Decomposition

The bias-variance-covariance decomposition is a theoretical result underlying ensemble learning algorithms. It is an extension of the bias-variance decomposition, for linear combinations of models. The expected squared error of the ensemble $\bar{f}(x)$ from a target d is:

$$\displaystyle\begin{array}{rcl} \mathcal{E}_{\mathcal{D}}\{\bar{f}(x) - d)^{2}\}& =& \overline{\mathrm{bias}}^{2} + \frac{1} {T}\overline{\mathrm{var}} {}\\ & & +\left (1 - \frac{1} {T}\right )\overline{\mathrm{covar}}. {}\\ \end{array}$$

The error is composed of the average bias of the models, plus a term involving their average variance, and a final term involving their average pairwise covariance. This shows that while a single model has a two-way bias-variance tradeoff, an ensemble is controlled by a three-way tradeoff. This ensemble tradeoff is often referred to as the accuracy-diversity dilemma for an ensemble. See ensemble learning for more details.