Support vector regression for polyhedral and missing data

Abstract

We introduce “Polyhedral Support Vector Regression” (PSVR), a regression model for data represented by arbitrary convex polyhedral sets. PSVR is derived as a generalization of support vector regression, in which the data is represented by individual points along input variables $X_1$, $X_2$, $\ldots$, $X_p$ and output variable Y, and extends a support vector classification model previously introduced for polyhedral data. PSVR is in essence a robust-optimization model, which defines prediction error as the largest deviation, calculated along Y, between an interpolating hyperplane and all points within a convex polyhedron; the model relies on the affine Farkas’ lemma to make this definition computationally tractable within the formulation. As an application, we consider the problem of regression with missing data, where we use convex polyhedra to model the multivariate uncertainty involving the unobserved values in a data set. For this purpose, we discuss a novel technique that builds on multiple imputation and principal component analysis to estimate convex polyhedra from missing data, and on a geometric characterization of such polyhedra to define observation-specific hyper-parameters in the PSVR model. We show that an appropriate calibration of such hyper-parameters can have a significantly beneficial impact on the model’s performance. Experiments on both synthetic and real-world data illustrate how PSVR performs competitively or better than other benchmark methods, especially on data sets with high degree of missingness.

Appendix

Proof of Theorem 1

In order to make the two $\max$ constraints in (6) feasible, the following systems of linear inequalities must be both infeasible:

$$\begin{aligned} \left\{ \begin{array}{ll} y-\varvec{w}^T\varvec{x} - w_0 > \xi _i+\epsilon _i\\ \varvec{A}_i \begin{pmatrix} \varvec{x} \\ y \end{pmatrix} \le \varvec{a}_i \end{array} \right. \end{aligned}$$

(17)

$$\begin{aligned} \left\{ \begin{array}{ll} \varvec{w}^T\varvec{x} + w_0 - y > \xi _i+\epsilon _i\\ \varvec{A}_i \begin{pmatrix} \varvec{x} \\ y \end{pmatrix} \le \varvec{a}_i \end{array} \right. \end{aligned}$$

(18)

By the affine Farkas’ lemma (Boyd and Vandenberghe 2004), alternative systems (19) and (20) must therefore be both feasible:

$$\begin{aligned} \begin{aligned}&\varvec{u}_i \ge \varvec{0} \quad \text{ and } \left\{ \begin{array}{ll} \varvec{A}_i^T \varvec{u}_i = \begin{pmatrix} -\varvec{w} \\ 1 \end{pmatrix} \\ \varvec{a}_i^T \varvec{u}_i - w_0 \le \xi _i+\epsilon _i \end{array} \right.&\text{ or }&\left\{ \begin{array}{ll} \varvec{A}_i^T \varvec{u}_i = \varvec{0}\\ \varvec{a}_i^T \varvec{u}_i < 0 \end{array} \right. \end{aligned} \end{aligned}$$

(19)

$$\begin{aligned} \begin{aligned}&\quad \varvec{v}_i \ge \varvec{0} \quad \ \text{ and } \left\{ \begin{array}{ll} \varvec{A}_i^T \varvec{v}_i = \begin{pmatrix} \varvec{w} \\ -1 \end{pmatrix} \\ \varvec{a}_i^T \varvec{v}_i + w_0 \le \xi _i+\epsilon _i \end{array} \right.&\text{ or }&\left\{ \begin{array}{ll} \varvec{A}_i^T \varvec{v}_i = \varvec{0}\\ \varvec{a}_i^T \varvec{v}_i < 0 \end{array} \right. \end{aligned} \end{aligned}$$

(20)

The feasibility of (19) and (20), and the non-emptiness of $P_i$ imply the following:

$$\begin{aligned} \left\{ \begin{array}{ll} \varvec{u}_i^T\left( \varvec{A}_i \begin{pmatrix} \varvec{x} \\ y \end{pmatrix} - \varvec{a}_i\right) \le 0\\ \varvec{v}_i^T\left( \varvec{A}_i \begin{pmatrix} \varvec{x} \\ y \end{pmatrix} - \varvec{a}_i\right) \le 0 \end{array} \right. \end{aligned}$$

(21)

Now, since both

$$\begin{aligned} \begin{aligned} \left\{ \begin{array}{ll} \varvec{A}_i^T \varvec{u}_i = \varvec{0}\\ \varvec{a}_i^T \varvec{u}_i < 0 \end{array} \right. \end{aligned} \end{aligned}$$

(22)

and

$$\begin{aligned} \left\{ \begin{array}{ll} \varvec{A}_i^T \varvec{v}_i = \varvec{0}\\ \varvec{a}_i^T \varvec{v}_i < 0 \end{array} \right. \end{aligned}$$

(23)

contradict (21), neither can hold true, which leads to formulation (7). $\square$