Generalizing the Prediction Sum of Squares Statistic and Formula, Application to Linear Fractional Image Warp and Surface Fitting

Abstract

The prediction sum of squares statistic uses the principle of leave-one-out cross-validation in linear least squares regression. It is computationally attractive, as it can be computed non-iteratively. However, it has limitations: it does not handle coupled measurements, which should be held out simultaneously, and is specific to the principle of leave-one-out, which is known to overfit when used for selecting a model’s complexity. We propose multiple-exclusion PRESS (MEXPRESS), which generalizes PRESS to coupled measurements and other types of cross-validation, while retaining computational efficiency with the non-iterative MEXPRESS formula. Using MEXPRESS, various strategies to resolve overfitting can be efficiently implemented. The core principle is to exclude training data too ‘close’ or too ‘similar’ to the validation data. We show that this allows one to select the number of control points automatically in three cases: (i) the estimation of linear fractional warps for dense image registration from point correspondences, (ii) surface reconstruction from a dense depth-map obtained by a depth sensor and (iii) surface reconstruction from a sparse point cloud obtained by shape-from-template.

Appendix: Proof of Proposition 1

The proof of proposition 1 follows the same steps as the proof for the PRESS (Bartoli 2009). It requires the following lemma.

Lemma 1

The model estimate $\bar{\mathbf {x}}_{(\mathcal {K})}$ obtained by holding out a set of measurements with index set $\mathcal {K}\in [1,m]$ is the same as the one obtained by replacing the $\mathcal {K}$ responses by their predictions with the model $\bar{\mathbf {x}}_{(\mathcal {K})}$:

(22)

where we used the notation $\mathtt I _{(\mathcal {K})}$ for an identity matrix whose diagonal entries with index in $\mathcal {K}$ are put to zero, and $\mathtt I _{(-\mathcal {K})} \mathop {=}\limits ^{ def }\mathtt I- \mathtt I _{(\mathcal {K})}$. Formally, $\mathtt I _{(\mathcal {K})} = \text {diag}(\mathbf {d})$ with $\mathbf {d}_\mathcal {K}= 0$ and $\mathbf {d}_{-\mathcal {K}} = 1$.

Proof of lemma 1

Using the definition of ${\tilde{\mathbf {b}}}_{(\mathcal {K})}$ we have:

$$\begin{aligned} {\mathtt A}^{\dag }{\tilde{\mathbf {b}}}_{(\mathcal {K})} ={\mathtt A}^{\dag } \mathtt I _{(\mathcal {K})} \mathbf {b} + {\mathtt A}^{\dag } \mathtt I _{(-\mathcal {K})} \mathtt A \bar{\mathbf {x}}_{(\mathcal {K})} \end{aligned}$$

(23)

Because $\mathtt I _{(-\mathcal {K})} = \mathtt I- \mathtt I _{(\mathcal {K})}$ this may be expanded as:

$$\begin{aligned} {\mathtt A}^{\dag }{\tilde{\mathbf {b}}}_{(\mathcal {K})} ={\mathtt A}^{\dag } \mathtt I _{(\mathcal {K})} \mathbf {b} + {\mathtt A}^{\dag } \mathtt A \bar{\mathbf {x}}_{(\mathcal {K})} - {\mathtt A}^{\dag } \mathtt I _{(\mathcal {K})} \mathtt A \bar{\mathbf {x}}_{(\mathcal {K})}. \end{aligned}$$

(24)

The second term is simplified to ${\mathtt A}^{\dag } \mathtt A \bar{\mathbf {x}}_{(\mathcal {K})} = \bar{\mathbf {x}}_{(\mathcal {K})}$. Using the definition of $\bar{\mathbf {x}}_{(\mathcal {K})}$, the third term is simplified to ${\mathtt A}^{\dag } \mathtt I _{(\mathcal {K})} \mathtt A \bar{\mathbf {x}}_{(\mathcal {K})} = {\mathtt A}^{\dag }\mathtt I _{(\mathcal {K})}\mathbf {b}$. The overall expression is thus simplified to:

$$\begin{aligned} {\mathtt A}^{\dag }{\tilde{\mathbf {b}}}_{(\mathcal {K})} ={\mathtt A}^{\dag } \mathtt I _{(\mathcal {K})} \mathbf {b} + \bar{\mathbf {x}}_{(\mathcal {K})} - {\mathtt A}^{\dag } \mathtt I _{(\mathcal {K})} \mathbf {b} =\bar{\mathbf {x}}_{(\mathcal {K})}, \end{aligned}$$

(25)

concluding the proof. $\square$

Proof of proposition 1

The case of an empty set $\mathcal {K}$ is straightforward as then $\mathbf {e}_{(\mathcal {K})}$ is a zero-length vector. We start by rewriting the predictions in $\mathcal {K}$ by the model fitted with all measurements as $\mathtt A _\mathcal {K}\bar{\mathbf {x}} = \mathtt A _\mathcal {K}{\mathtt A}^{\dag } \mathbf {b} = \hat{\mathtt A} _\mathcal {K}\mathbf {b}$. Similarly, we rewrite the predictions in $\mathcal {K}$ by the model fitted with all but the measurements in $\mathcal {K}$ as $\mathtt A _\mathcal {K}\bar{\mathbf {x}}_{(\mathcal {K})} = \mathtt A _\mathcal {K}{\mathtt A}^{\dag } {\tilde{\mathbf {b}}}_{(\mathcal {K})} = \hat{\mathtt A} _\mathcal {K}{\tilde{\mathbf {b}}}_{(\mathcal {K})}$, where the first equality is given by Lemma 1. The difference between the two predictions is thus given by:

(26)

where the second equality is obtained by using the definition of ${\tilde{\mathbf {b}}}_{(\mathcal {K})}$. We then use $\mathtt I-\mathtt I _{(\mathcal {K})}=\mathtt I _{(-\mathcal {K})}$, leading to:

$$\begin{aligned} \mathtt A _\mathcal {K}\bar{\mathbf {x}} - \mathtt A _\mathcal {K}\bar{\mathbf {x}}_{(\mathcal {K})} =\hat{\mathtt A} _\mathcal {K}\left( \mathtt I _{(-\mathcal {K})}\mathbf {b} - \mathtt I _{(-\mathcal {K})} \mathtt A \bar{\mathbf {x}}_{(\mathcal {K})}\right) . \end{aligned}$$

(27)

By noting that $\hat{\mathtt A} _\mathcal {K}\mathtt I _{(-\mathcal {K})}\mathtt A = \hat{\mathtt A} _{\mathcal {K},\mathcal {K}}\mathtt A _\mathcal {K}$ and $\hat{\mathtt A} _\mathcal {K}\mathtt I _{(-\mathcal {K})}\mathbf {b} = \hat{\mathtt A} _{\mathcal {K},\mathcal {K}}\mathbf {b}_\mathcal {K}$ we rearrange the equation as:

$$\begin{aligned} (\mathtt I- \hat{\mathtt A} _{\mathcal {K},\mathcal {K}}) \mathtt A _\mathcal {K}\bar{\mathbf {x}}_{(\mathcal {K})} + \hat{\mathtt A} _{\mathcal {K},\mathcal {K}}\mathbf {b}_\mathcal {K}=\mathtt A _\mathcal {K}\bar{\mathbf {x}}. \end{aligned}$$

(28)

We subtract $\mathbf {b}_\mathcal {K}$ to each side of the equation:

$$\begin{aligned} \left( \mathtt I- \hat{\mathtt A} _{\mathcal {K},\mathcal {K}}\right) \left( \mathtt A _\mathcal {K}\bar{\mathbf {x}}_{(\mathcal {K})} - \mathbf {b}_\mathcal {K}\right) =\mathtt A _\mathcal {K}\bar{\mathbf {x}} - \mathbf {b}_\mathcal {K}, \end{aligned}$$

(29)

and arrive at:

$$\begin{aligned} \mathtt A _\mathcal {K}\bar{\mathbf {x}}_{(\mathcal {K})} - \mathbf {b}_\mathcal {K}={\left( \mathtt I- \hat{\mathtt A} _{\mathcal {K},\mathcal {K}}\right) }^{-1} \left( \mathtt A _\mathcal {K}\bar{\mathbf {x}} - \mathbf {b}_\mathcal {K}\right) , \end{aligned}$$

(30)

concluding the proof. $\square$