Iteration-complexity of first-order penalty methods for convex programming

Abstract

This paper considers a special but broad class of convex programming problems whose feasible region is a simple compact convex set intersected with the inverse image of a closed convex cone under an affine transformation. It studies the computational complexity of quadratic penalty based methods for solving the above class of problems. An iteration of these methods, which is simply an iteration of Nesterov’s optimal method (or one of its variants) for approximately solving a smooth penalization subproblem, consists of one or two projections onto the simple convex set. Iteration-complexity bounds expressed in terms of the latter type of iterations are derived for two quadratic penalty based variants, namely: one which applies the quadratic penalty method directly to the original problem and another one which applies the latter method to a perturbation of the original problem obtained by adding a small quadratic term to its objective function.

Appendix

In this section, we prove Proposition 1.

Proof of Proposition 1

Define $\mathcal{C}:= \{(v, t) \in \mathfrak R ^m \times \mathsf{I\!\!R}: \Vert v\Vert \le t\}$ and let $\mathcal{C}^*$ denote the dual cone of $\mathcal{C}$. It is easy to see that $\mathcal{C}^* = \{(\tilde{v}, \tilde{t}) \in \mathfrak R ^m \times \mathsf{I\!\!R}: \Vert \tilde{v}\Vert \le \tilde{t}\}$. By definition of $d_\mathcal{K ^*}$ and conic duality, we have

$$\begin{aligned} d_{{\mathcal{K}^*}}(u)&= \inf _{\tilde{k}, \tilde{t}} \{\tilde{t}: \Vert u-\tilde{k}\Vert \le \tilde{t}, \, \, \tilde{k} \in {\mathcal{K}^*}\}\\&= \inf _{\tilde{k}, \tilde{t}, \tilde{v}} \{\tilde{t}: \tilde{v} + \tilde{k} = u, \,\, (\tilde{v}, \tilde{t}) \in \mathcal{C}^*, \,\, \tilde{k} \in {\mathcal{K}^*}\} \\&= \sup _{(v,k,t)} \{ \langle u, y \rangle : t = 1, \, y + v = 0, \, y + k = 0, \, (v,t) \in \mathcal{C}, \, k \in \mathcal K \} \\&= \sup \{ \langle u, y \rangle : (-y , 1) \in \mathcal{C}, \, \, - y \in \mathcal K \} \ = \ \sup \{ \langle u, y \rangle : y \in (- \mathcal K ) \cap B(0,1) \}. \end{aligned}$$

Statement (a) follows from the above identity and the definition of the support function of a set (see Sect. 1.1).

To show statement (b), let $u \in \mathfrak R ^m$ and $\lambda \in \mathcal K$ be given and assume without loss of generality that $\lambda \ne 0$. Now noting that $-\lambda /\Vert \lambda \Vert \in C := (-\mathcal K ) \cap B(0,1)$, we conclude from the above identity that $d_\mathcal K (u) \ge \langle u, -\lambda /\Vert \lambda \Vert \rangle$, or equivalently, $\langle u, \lambda \rangle \ge - \Vert \lambda \Vert \, d_\mathcal K (u)$. $\square$