Models and algorithms for distributionally robust least squares problems

Abstract

We present three different robust frameworks using probabilistic ambiguity descriptions of the data in least squares problems. These probability ambiguity descriptions are given by: (1) confidence region over the first two moments; (2) bounds on the probability measure with moments constraints; (3) the Kantorovich probability distance from a given measure. For the first case, we give an equivalent formulation and show that the optimization problem can be solved using a semidefinite optimization reformulation or polynomial time algorithms. For the second case, we derive the equivalent Lagrangian problem and show that it is a convex stochastic programming problem. We further analyze three special subcases: (i) finite support; (ii) measure bounds by a reference probability measure; (iii) measure bounds by two reference probability measures with known density functions. We show that case (i) has an equivalent semidefinite programming reformulation and the sample average approximations of case (ii) and (iii) have equivalent semidefinite programming reformulations. For ambiguity description (3), we show that the finite support case can be solved by using an equivalent second order cone programming reformulation.

Appendix

In Sect. 3 we use SAA method to approximate the stochastic programming problems (3.13) and (3.18). The convergence results for the SAA method from [16] are summarized here. Consider a stochastic programming problem of the form

$$\begin{aligned} \min _\mathbf{x \in \mathcal X }\{ f(\mathbf x ):=\mathbb E _\mathbb{P }[F(\mathbf x ,\pmb {\xi })] \}, \end{aligned}$$

(6.1)

where $F(\mathbf x ,\pmb {\xi })$ is a function of variables $\mathbf x \in \mathbb R ^n$ and parameters $\pmb {\xi } \in \mathbb R ^d$. $\mathcal X \subset \mathbb R ^n$ is a given set, and $\pmb {\xi }=\pmb {\xi }(\omega )$ is a random vector. The expectation in (6.1) is taken with respect to the probability distribution of $\pmb {\xi }$ which is assumed to be known as $\mathbb P$. Denote by $\Xi \subset \mathbb R ^d$ the support of the probability distribution of $\pmb {\xi }$, that is, $\Xi$ is the smallest closed set in $\mathbb R ^d$ such that the probability of the event $\pmb {\xi } \in \mathbb R ^d \setminus \Xi$ is zero. Also denote by $\mathbb P (A)$ the probability of an event $A$. With the generated sample $\xi ^1,\ldots ,\xi ^K$, we associate the sample average function

$$\begin{aligned} \hat{f}_K(\mathbf x ):=\frac{1}{K} \sum _{i=1}^K F(\mathbf x ,\xi ^i). \end{aligned}$$

(6.2)

The stochastic programming problem (6.1) is approximated by the optimization problem

$$\begin{aligned} \min _\mathbf{x \in \mathcal X } \left\{ \hat{f}_K(\mathbf x ):=\frac{1}{K} \sum _{i=1}^K F(\mathbf x ,\xi ^i) \right\} . \end{aligned}$$

(6.3)

Let the optimal value of (6.1) be $\nu$ and it’s optimal solution set be $S$. Let $\hat{\nu }_K$ and $\hat{S}_K$ be the optimal value and the set of optimal solutions of the SAA problem (6.3). For sets $A,B \subset \mathbf R ^n$, denote $\mathrm{dist}(x,A):=\inf _{x^{\prime } \in A}\left| \left| x-x^{\prime }\right| \right|$ to be the distance from $x \in \mathbb X ^n$ to $A$, and

$$\begin{aligned} \mathbb D (A,B):=\sup _{x \in A} \mathrm{dist}(x,B). \end{aligned}$$

(6.4)

Also, define the function $(\mathbf x ,\xi ) \mapsto F(\mathbf x ,\xi )$ to be a random lower semicontinuous function if the associated epigraphical multifunction $\xi \mapsto \mathrm{epi}F(\cdot ,\xi )$ is closed valued and measurable. We say that the Law of Large Numbers (LLN) holds, for $\hat{f}_K(\mathbf x )$, pointwise if $\hat{f}_K(\mathbf x )$ converges w.p.1 to $f(\mathbf x )$, as $K \rightarrow \infty$, for any fixed $\mathbf x \in \mathbb R ^n$. The following convergence theorem ensures that a solution of SAA problem (6.2) converges to that of (6.1) as the sample size increases.

Theorem 8

[16, Chapter 6, Theorem 4.] Suppose that: (i) the integrand function $F$ is random lower semicontinuous, (ii) for almost every $\xi \in \Xi$ the function $F(\cdot ,\xi )$ is convex, (iii) the set $\mathcal X$ is closed and convex, (iv) the expected value function $f$ is lower semicontinuous and there exists a point $\hat{\mathbf{x }} \in \mathcal X$ such that $f(\mathbf x ) \le +\infty$ for all $x$ in a neighborhood of $\hat{\mathbf{x }}$, (v) the set $S$ of optimal solutions of the original problem (6.1) is nonempty and bounded, (vi) the LLN holds pointwise. Then $\hat{\nu }_K \rightarrow \nu ^*$ and $\mathbb D (\hat{S}_K,S) \rightarrow 0$ w.p.1 as $K \rightarrow \infty$.

There are also results about the exponential convergence rate of the SAA method. Please refer to [16, 19] for details of such results. We note that the assumptions in Theorem 8 are satisfied for the models (3.13) and (3.18). Hence the convergence of the solution of SAA of these problems to a solution of the original problem is guaranteed.