Symmetrization Lemma

Synonyms

Basic lemma

Definition

Given a distribution P over a sample space $\mathcal{Z}$, a finite sample $\mathbf{z} = (z_{1},\ldots,z_{n})$ drawn i.i.d. from P and a function $f : \mathcal{Z}\rightarrow \mathbb{R}$ we define the shorthand $\mathbb{E}_{P}f = \mathbb{E}_{P}[f(z)]$ and $\mathbb{E}_{\mathbf{z}}f = \frac{1} {n}\sum _{i-1}^{n}f(z_{ i})$ to denote the true and empirical average of f. The symmetrization lemma is an important result in the learning theory as it allows the true average $\mathbb{E}_{P}f$ found in generalization bounds to be replaced by a second empirical average $\mathbb{E}_{\mathbf{z}^{{\prime}}}f$ taken over an independent ghost sample$\mathbf{z}^{{\prime}} = z_{1}^{{\prime}},\ldots z_{n}^{{\prime}}$ drawn i.i.d. from P. Specifically, the symmetrization lemma states that for any ε > 0 whenever n ε² ≥ 2

$$\displaystyle\begin{array}{rcl} & & P^{n}\left (\sup \limits _{ f\in \mathbb{F}}\vert \mathbb{E}_{P}f -...