An extrapolated iteratively reweighted $\ell _1$ method with complexity analysis

Abstract

The iteratively reweighted $\ell _1$ algorithm is a widely used method for solving various regularization problems, which generally minimize a differentiable loss function combined with a convex/nonconvex regularizer to induce sparsity in the solution. However, the convergence and the complexity of iteratively reweighted $\ell _1$ algorithms is generally difficult to analyze, especially for non-Lipschitz differentiable regularizers such as $\ell _p$ norm regularization with $0<p<1$. In this paper, we propose, analyze and test a reweighted $\ell _1$ algorithm combined with the extrapolation technique under the assumption of Kurdyka-Łojasiewicz (KL) property on the proximal function of the perturbed objective. Our method does not require the Lipschitz differentiability on the regularizers nor the smoothing parameters in the weights bounded away from 0. We show the proposed algorithm converges uniquely to a stationary point of the regularization problem and has local linear convergence for KL exponent at most 1/2 and local sublinear convergence for KL exponent greater than 1/2. We also provide results on calculating the KL exponents and discuss the cases when the KL exponent is at most 1/2. Numerical experiments show the efficiency of our proposed method.

Appendix

The translation of the proof of Theorem 5.

Proof

It follows from $\nabla h(x^*)=0$ and the mean-value theorem that

$$\begin{aligned} h(x) - h(x^*) = (x-x^*)^T\nabla ^2h(x^*+t_x(x-x^*))(x-x^*),\end{aligned}$$

with $t_x \in (0,1)$. Since h is twice continuously differentiable on $\mathbb {B}(x^*, \varepsilon _0)$, there exists $L>0$ such that

$$\begin{aligned} h(x) - h(x^*) \le L\Vert x-x^*\Vert _2^2\end{aligned}$$

for any $x\in \mathbb {B}(x^*,\varepsilon _0)$. On the other hand,

$$ \begin{aligned} \nabla h(x) = & \nabla h(x) - \nabla h(x^{*} ) \\ = & \int_{0}^{1} {\nabla ^{2} } h(x + t(x - x^{*} ))(x - x^{*} )dt \\ = & \left[ {\begin{array}{*{20}c} {[\nabla ^{2} h(x^{*} + t_{{1x}} (x - x^{*} ))]_{1} } \\ \ldots \\ {\nabla ^{2} h(x^{*} + t_{{nx}} (x - x^{*} ))]_{n} } \\ \end{array} } \right](x - x^{*} ) \\ : = & A_{x} (x - x^{*} ), \\ \end{aligned} $$

where $t_{ix}\in (0,1), i=1,2, \ldots , n$.

Since $\nabla ^2 h(x^*)$ is nonsingular, there exists $0<\varepsilon _1 < \varepsilon _0$ such that for any $x\in \mathbb {B}(x^*, \varepsilon _1)$, $A_x$ is nonsingular. Therefore, $A_x^TA_x$ is positive definite on $\mathbb {B}(x^*, \varepsilon _1)$. Hence, there exists $0<\varepsilon < \varepsilon _1$ and $\sigma > 0$ satisfying $\sigma = \min _{x\in \mathbb {B}(x^*, \varepsilon )} \sigma _{\min }(A_x^TA_x)$. It then follows that

$$\begin{aligned} \Vert \nabla h(x)\Vert ^2 = (x-x^*)^TA_x^TA_x(x-x^*) \ge \sigma \Vert x-x^*\Vert ^2_2 \ge \frac{\sigma }{L}(h(x)-h(x^*)). \end{aligned}$$

This implies that there exists $\theta = \tfrac{1}{2} \in (0,1]$, $\varepsilon > 0$ and $c= \big ( \tfrac{L}{\sigma } \big )^{1/2}$ for any $x\in B(x^*, \varepsilon )$,

$$\begin{aligned} (h(x)-h(x^*))^{1-\theta }_+ \le c\Vert \nabla h(x)\Vert ,\end{aligned}$$

or, equivalently, h satisfies the KL inequality with $\theta = \tfrac{1}{2}$. $\square$