An Interior Stochastic Gradient Method for a Class of Non-Lipschitz Optimization Problems

Abstract

In this paper, we propose an interior stochastic gradient method for bound constrained optimization problems where the objective function is the sum of a smooth expectation of random integrands and a non-Lipshitz $\ell _p (0<p<1)$ regularizer. We first define an L-stationary point of the problem and show that an L-stationary point is a first-order stationary point and a global minimizer is an L-stationary point. Next we propose a stochastic gradient method to reduce the computational cost for the exact gradients of the smooth term in the objective function and to keep the feasibility of iterates. We show that the proposed stochastic gradient method is convergent to an $\epsilon$-approximate L-stationary point in ${\mathcal {O}}(\epsilon ^{-4})$ computational complexity, with optimality measure in $\ell _2$ norm. Finally, we conduct preliminary numerical experiments to test the proposed method and compare it with some existing method. Preliminary numerical results indicate that the proposed method is promising.

Proof of Lemma 2.2

Notice that h is level bounded. The solution set of (2.2) is nonempty and bounded.

Firstly, we assume $\alpha < {\bar{\omega }}$. For any $t^*\in {\mathcal {T}}^*_\alpha$, suppose that $t^*\ne 0$, then $t^*>0$. By the optimality condition for (2.2), we have

$$\begin{aligned} \alpha =t^*+\frac{\lambda p}{L}(t^*)^{p-1}. \end{aligned}$$

Then it derives

$$\begin{aligned} 0\ge h(t^*)-h(0)= \;&\frac{1}{2}(t^*-\alpha )^2 +\frac{\lambda }{L}(t^*)^p -\frac{1}{2}\alpha ^2\\ = \;&\frac{1}{2}(t^*)^2 - t^*\alpha + \frac{\lambda }{L}(t^*)^p \\ =\;&\frac{1}{2}(t^*)^2 - t^*(t^*+\frac{\lambda p}{L}(t^*)^{p-1}) + \frac{\lambda }{L}(t^*)^p\\ = \;&-\frac{1}{2}(t^*)^2 + \frac{\lambda }{L}(1-p)(t^*)^p \end{aligned}$$

which yields $t^* \ge (\frac{2\lambda (1-p)}{L})^{\frac{1}{2-p}}$. Then it implies from Lemma 2.1 that

$$\begin{aligned} \alpha \ge h_1\left( \left( \frac{2\lambda (1-p)}{L}\right) ^{\frac{1}{2-p}}\right) = {\bar{\omega }}. \end{aligned}$$

However, it contradict the assumption $\alpha <{\bar{\omega }}$. Hence, ${\mathcal {T}}^*_\alpha =\{0\}$.

Secondly, we assume $\alpha ={\bar{\omega }}$. By easy computations we have

$$\begin{aligned} h(0)=h\left( \left( \frac{2\lambda (1-p)}{L}\right) ^{\frac{1}{2-p}}\right) . \end{aligned}$$

(A.1)

And for any $t>0$, the following equalities hold:

$$\begin{aligned} h(t) =h(0)+ \frac{1}{2}t^2-t\alpha +\frac{\lambda }{L}t^p \, \text{ and } \, h'(t)=t +p \frac{\lambda }{L}t^{p-1}-\alpha . \end{aligned}$$

Recall that by Lemma 2.1, $h'(t)$ is monotonically decreasing in $(0,\varrho \,]$ and monotonically increasing in $[\,\varrho ,+\infty )$ with $\varrho =(\frac{\lambda p(1-p)}{L})^{\frac{1}{2-p}}$. Then we obtain

$$\begin{aligned} \lim \limits _{t\rightarrow 0{+}}h'(t)>0,\;\; \; \lim \limits _{t\rightarrow +\infty }h'(t)>0\;\;\; \mathrm{and}\;\;\; h'(\varrho ) < h'\left( \left( \frac{2\lambda (1-p)}{L}\right) ^{\frac{1}{2-p}}\right) =0. \end{aligned}$$

Thus $h'(t)$ has two roots:

$$\begin{aligned} t_1\in (0,\varrho ),\, \, t_2=\left( \frac{2\lambda (1-p)}{L}\right) ^{\frac{1}{2-p}} \in (\varrho ,+\infty ), \end{aligned}$$

$h'(t)>0$ in $(0,t_1) \cup (t_2,+\infty )$ and $h'(t)<0$ in $(t_1,t_2)$. Thus h(t) is monotonically increasing in $(0,t_1]$ and $[t_2,+\infty )$ and is monotonically decreasing in $[t_1,t_2]$. Then by (A.1) we obtain

$$\begin{aligned} h(t_1)=\max _{{t\in (0,t_1]}} h(t)\;\;\; \text{ and } \;\;\; h(t_2)=\min _{t>0} h(t). \end{aligned}$$

It thus follows that

$$\begin{aligned} h(t)>h(t_2) \quad \text{ for } \text{ any } 0<t\ne t_2, \end{aligned}$$

which together with (A.1) indicates that ${\mathcal {T}}^*_\alpha =\{0, t_2\}$ when $\alpha ={\bar{\omega }}$.

At last we assume $\alpha > {\bar{\omega }}$. Let $\bar{t}=\left( \frac{2\lambda (1-p)}{L}\right) ^{\frac{1}{2-p}}$. Then for any $t^*\in {\mathcal {T}}^*_\alpha$, it yields from $h(t^*)\le h(\bar{t})$ that

$$\begin{aligned} h(t^*)=\frac{1}{2}(t^*-\alpha )^2 + \frac{\lambda }{L}(t^*)^p\le & {} \frac{1}{2}(\bar{t}-\alpha )^2 + \frac{\lambda }{L}\bar{t}^p\\= & {} { \frac{1}{2}\alpha ^2 + \bar{t}( {\bar{\omega }}- \alpha )}\\< & {} \frac{1}{2}\alpha ^2 = h(0). \end{aligned}$$

Then $t^*\ne 0$ thus $t^*>0$. Then the optimality of $t^*$ indicates

$$\begin{aligned} \alpha =t^*+\frac{\lambda p}{L}(t^*)^{p-1}=h_1(t^*), \end{aligned}$$

(A.2)

which together with $h(t^*)< h(0)$ implies

$$\begin{aligned} t^*> \left( \frac{2\lambda (1-p)}{L}\right) ^{\frac{1}{2-p}}. \end{aligned}$$

(A.3)

Recall that by Lemma 2.1, $h_1$ is monotonically increasing on $[(\frac{\lambda p(1-p)}{L})^{\frac{1}{2-p}}, +\infty )$. Hence, $t^*$ satisfying (A.2) and (A.3) is unique. The proof is completed.