An efficient augmented Lagrangian method with applications to total variation minimization

Abstract

Based on the classic augmented Lagrangian multiplier method, we propose, analyze and test an algorithm for solving a class of equality-constrained non-smooth optimization problems (chiefly but not necessarily convex programs) with a particular structure. The algorithm effectively combines an alternating direction technique with a nonmonotone line search to minimize the augmented Lagrangian function at each iteration. We establish convergence for this algorithm, and apply it to solving problems in image reconstruction with total variation regularization. We present numerical results showing that the resulting solver, called TVAL3, is competitive with, and often outperforms, other state-of-the-art solvers in the field.

Appendix: Proof of Theorem 1

For notational simplicity, let us define

$$ \phi_k(\cdot) = \phi(\cdot, y_k) \quad \mbox{and}\quad\nabla\phi_k(\cdot) = \partial_1 \phi( \cdot, y_k). $$

(22)

The proof of the theorem relies on two lemmas. The two lemmas are modifications of their counterparts in [40]. Since our objective may contain a non-differentiable part, the key modification is to connect this non-differentiable part to the differentiable part by means of alternating minimization. Otherwise, the line of proofs follows closely that given in [40].

The first lemma presents some basic properties and established that the algorithm is well-defined.

Lemma 1

If ∇ϕ _k (x _k )^T d _k ≤0 holds for each k, then for the sequences generated by Algorithm-NADA, we have ϕ _k (x _k )≤ϕ _k−1(x _k )≤C _k for each k and {C _k } is monotonically non-increasing. Moreover, if ∇ϕ _k (x _k )^T d _k <0, a step length α _k >0 always exists so that the nonmonotone Armijo condition (11) holds.

Proof

Define real-valued function

$$D_k(t) = \frac{tC_{k-1} + \phi_{k-1}(x_k)}{t+1} \quad\mbox{for}\ t \ge0, $$

then

$$D_k'(t) = \frac{C_{k-1} - \phi_{k-1}(x_k)}{(t+1)^2} \quad\mbox{for}\ t \ge 0. $$

Due to the nonmonotone Armijo condition (11) and ∇ϕ _k (x _k )^T d _k ≤0, we have

$$C_{k-1} - \phi_{k-1}(x_k)\geq-\delta \alpha_{k-1} \nabla\phi_{k-1}(x_{k-1})^T d_{k-1} \geq0. $$

Therefore, $D_{k}'(t) \ge0$ holds for any t≥0, and then D _k is non-decreasing.

Since

$$D_k(0) = \phi_{k-1}(x_k) \quad\mbox{and}\quad D_k(\eta_{k-1} Q_{k-1}) = C_k, $$

we have

$$\phi_{k-1}(x_k) \le C_k, \quad\forall k. $$

As is defined in Algorithm-NADA,

$$y_k = \operatorname {argmin}_y \phi(x_k,y). $$

Therefore,

$$\phi(x_k, y_k) \leq\phi(x_{k}, y_{k-1}). $$

Hence, ϕ _k (x _k )≤ϕ _k−1(x _k )≤C _k holds for any k.

Furthermore,

$$C_{k+1} = \frac{\eta_k Q_k C_k + \phi_k(x_{k+1})}{Q_{k+1}} \le\frac {\eta_k Q_k C_k + C_{k+1}}{Q_{k+1}} , $$

i.e.,

$$(\eta_k Q_k +1)C_{k+1} \le\eta_k Q_k C_k + C_{k+1}, $$

that is,

$$C_{k+1}\le C_k. $$

Thus, {C _k } is monotonically non-increasing.

If C _k is replaced by ϕ _k (x _k ) in (11), the nonmonotone Armijo condition becomes the standard Armijo condition. It is well-known that α _k >0 exists for the standard Armijo condition while ∇ϕ _k (x _k )^T d _k <0 and ϕ is bounded below. Since ϕ _k (x _k )≤C _k , it follows α _k >0 exists as well for the nonmonotone Armijo condition:

$$\begin{aligned} \phi_k(x_k+\alpha_k d_k) \le C_k + \delta\alpha_k \nabla \phi_k(x_k)^T d_k. \end{aligned}$$

Now we defining the quantity A _k by

$$\begin{aligned} A_k = \frac{1}{k+1} \sum _{i=0}^k \phi_k(x_k). \end{aligned}$$

(23)

By induction, it is easy to show that C _k is bounded above by A _k . Together with the facts that C _k is also bounded below by ϕ _k (x _k ) and α _k >0 always exists, it is clear that Algorithm-NADA is well-defined. □

In the next lemma, a lower bound for the step length generated by Algorithm-NADA will be given.

Lemma 2

Assume that ∇ϕ _k (x _k )^T d _k ≤0 for all k and that Lipschitz condition (19) holds with constant L. Then

$$\begin{aligned} \alpha_k \geq\min\biggl\{ {\alpha_{\max} \over\rho}, {2(1-\delta)\over L\rho} {|\nabla\phi_k(x_k)^T d_k|\over\|d_k\| ^2} \biggr\}. \end{aligned}$$

(24)

The proof is omitted here since the proof of Lemma 2.1 in [40] is directly applicable.

With the aid of the lower bound (24), we now are ready to prove Theorem 1. We need to establish the two relationships given in (20).

Proof

First, by definition in Algorithm-NADA,

$$y_k = \operatorname {argmin}_y \phi(x_k,y). $$

Hence, it always holds true that

$$0 \in\partial_2 \phi(x_k,y_k). $$

Now it suffices to show that the limit holds true in (20). Consider the nonmonotone Armijo condition:

$$\begin{aligned} \phi_k(x_k+\alpha_k d_k) \le C_k + \delta\alpha_k \nabla \phi_k(x_k)^T d_k. \end{aligned}$$

(25)

If ρα _k <α _max, in view of the lower bound (24) on α _k in Lemma 2 and the direction assumption (18),

$$\begin{aligned} \phi_k(x_k+\alpha_k d_k) \le& C_k - \delta\frac{2(1-\delta)}{ L\rho} \frac{|\nabla\phi_k(x_k)^T d_k|^2}{\| d_k \| ^2} \\ \le& C_k - \frac{2 \delta(1-\delta)}{L\rho} \frac{c_1^2 \|\nabla\phi _k(x_k)\|^4}{c_2^2 \| \nabla\phi_k(x_k) \| ^2} \\ =& C_k - \biggl[ \frac{2\delta(1-\delta)c_1^2}{L\rho c_2^2} \biggr] \big\| \nabla \phi_k(x_k) \big\| ^2. \end{aligned}$$

On the other hand, if ρα _k ≥α _max, the lower bound (24), together with the direction assumption (18), gives

$$\begin{aligned} \phi_k(x_k+\alpha_k d_k) \le& C_k + \delta\alpha_k \nabla\phi_k(x_k)^T d_k \\ \le& C_k - \delta\alpha_k c_1 \big\| \nabla \phi_k(x_k) \big\| ^2 \\ \le& C_k - \frac{\delta\alpha_{\max} c_1}{\rho} \big\| \nabla\phi_k(x_k) \big\| ^2. \end{aligned}$$

Introducing a constant

$$\tilde{\tau} = \min\biggl\{ \frac{2\delta(1-\delta)c_1^2}{L \rho c_2^2}, \frac{\delta\alpha_{\max} c_1}{\rho} \biggr\}, $$

we can combine the above inequalities into

$$\begin{aligned} \phi_k(x_k+ \alpha_k d_k) \le C_k - \tilde{\tau} \big\| \nabla \phi_k(x_k) \big\| ^2. \end{aligned}$$

(26)

Next we show by induction that for all k

$$ \frac{1}{Q_k} \ge1-\eta_{\max}, $$

(27)

which obviously holds for k=0 given that Q ₀=1. Assume that (27) holds for k=j. Then

$$\begin{aligned} Q_{j+1} = \eta_j Q_j +1 \le \frac{\eta_j}{1-\eta_{\max}} +1 \le\frac{\eta_{\max}}{1-\eta_{\max}} +1 = \frac{1}{1-\eta_{\max}}, \end{aligned}$$

implying that (27) also holds for k=j+1. Hence, (27) holds for all k.

It follows from (26) and (27) that

$$\begin{aligned} C_k - C_{k+1} =& C_k - {\eta_k Q_k C_k + \phi_k(x_{k+1})\over Q_{k+1}} \\ =& {C_k(\eta_k Q_k + 1) - (\eta_k Q_k C_k + \phi_k(x_{k+1})) \over Q_{k+1} } \\ =& {C_k - \phi_k(x_{k+1}) \over Q_{k+1} } \\ \ge& \frac{\tilde{\tau} \|\nabla\phi_k(x_k)\|^2}{Q_{k+1}} \\ \ge& \tilde{\tau} (1-\eta_{\max}) \big\|\nabla\phi_k(x_k) \big\| ^2. \end{aligned}$$

(28)

Since ϕ is bounded below by assumption, {C _k } is also bounded below. In addition, by Lemma 1, {C _k } is monotonically non-increasing, hence convergent. Therefore, the left-hand side of (28) tends to zero, so does the right-hand side; i.e., ∥∇ϕ _k (x _k )∥→0. Finally, by definition (22),

$$\lim_{k\rightarrow0} \partial_1 \phi(x_k, y_k) = 0, $$

which completes the proof. □