Variable sample size method for equality constrained optimization problems

Abstract

An equality constrained optimization problem with a deterministic objective function and constraints in the form of mathematical expectation is considered. The constraints are transformed into the Sample Average Approximation form resulting in deterministic problem. A method which combines a variable sample size procedure with line search is applied to a penalty reformulation. The method generates a sequence that converges towards first-order critical points. The final stage of the optimization procedure employs the full sample and the SAA problem is eventually solved with significantly smaller cost. Preliminary numerical results show that the proposed method can produce significant savings compared to SAA method and some heuristic sample update counterparts while generating a solution of the same quality.

Appendix

Algorithm 2

Step 0 :

Input parameters: $dm_{k}$, $\epsilon _{\delta }^{N_{k}}(x_{k})$, $x_k$, $N_k$, $N_{k}^{min}$, $\nu _{1}\in (0,1)$.

Step 1 :

Determine $N_{k+1}$

1) :

If $dm_{k}=\epsilon _{\delta }^{N_{k}}(x_{k})$ set $N_{k+1}=N_{k}$.

2) :

If $dm_{k}>\epsilon _{\delta }^{N_{k}}(x_{k})$

Starting with $N=N_{k}$, while $dm_{k}>\frac{N_k}{N}\;\epsilon _{\delta }^{N}(x_{k})$ and $N>N_{k}^{min}$, decrease N by 1 and calculate $\varepsilon _{\delta }^{N}(x_{k}).$ Set $N_{k+1}=N$.

3) :

$dm_{k}<\epsilon _{\delta }^{N_{k}}(x_{k})$

i) :

If $dm_{k}\ge \nu _{1} \epsilon _{\delta }^{N_{k}}(x_{k})$ Starting with $N=N_{k}$, while $dm_{k}<\frac{N_k}{N}\;\epsilon _{\delta }^{N}(x_{k})$ and $N<N_{max}$, increase N by 1 and calculate $\epsilon _{\delta }^{N}(x_{k})$. Set $N_{k+1}=N$.

ii) :

If $dm_{k}< \nu _{1}\epsilon _{\delta }^{N_{k}}(x_{k})$ set $N_{k+1}=N_{max}$.

Algorithm 3

We say that we have not made big enough decrease of the function $\hat{\theta }_{N_{k+1}}$ if the following inequality is true

$$\begin{aligned} \frac{\hat{\theta }_{N_{k+1}}(x_{l(k)})-\hat{\theta }_{N_{k+1}}(x_{k+1})}{k+1-l(k)}< \frac{N_{k+1}}{N_{max}}\,\epsilon _{\delta }^{N_{k+1}}(x_{k+1}), \end{aligned}$$

where l(k) is the iteration at which we started to use the sample size $N_{k+1}$ for the last time.

Step 0 :

Input parameters: $N_{k}$, $N_{k+1}$, $N_{k}^{min}$.

Step 1 :

Determine $N_{k+1}^{min}$:

1) :

If $N_{k+1}\le N_{k}$ then $N_{k+1}^{min}=N_{k}^{min}$.

2) :

If $N_{k+1}> N_{k}$ and

i) :

if $N_{k+1}$ is a sample size which has not been used so far then $N_{k+1}^{min}=N_{k}^{min}$.

ii) :

if $N_{k+1}$ is a sample size which had been used and if we have made big enough decrease of the function $\hat{\theta }_{N_{k+1}}$ since the last time we used it, then $N_{k+1}^{min}=N_{k}^{min}$.

iii) :

if $N_{k+1}$ is a sample size which had been used and if we have not made big enough decrease of the function $\hat{\theta }_{N_{k+1}}$ since the last time we used it, then $N_{k+1}^{min}=N_{k+1}$.