Skip to main content
Log in

A primal majorized semismooth Newton-CG augmented Lagrangian method for large-scale linearly constrained convex programming

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

In this paper, we propose a primal majorized semismooth Newton-CG augmented Lagrangian method for large-scale linearly constrained convex programming problems, especially for some difficult problems. The basic idea of this method is to apply the majorized semismooth Newton-CG augmented Lagrangian method to the primal convex problem. And we take two special nonlinear semidefinite programming problems as examples to illustrate the algorithm. Furthermore, we establish the global convergence and the iteration complexity of the algorithm. Numerical experiments demonstrate that our method works very well for the testing problems, especially for many ill-conditioned ones.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Rockafellar, R.T.: Convex Analysis. Princeton University Press, Princeton (1970)

    Book  MATH  Google Scholar 

  2. Rockfellar, R.T., Wets, R.J.B.: Variational Analysis. Springer, Berlin (1998)

    Book  Google Scholar 

  3. Moreau, J.J.: Décomposition orthogonale d’un espace hilbertien selon deux cones mutuellement polaires. CR Acad. Sci. 255, 238–240 (1962)

    MATH  Google Scholar 

  4. Rockafellar, R.T.: Monotone operators and the proximal point algorithm. SIAM J. Control Optim. 14, 877–898 (1976)

    Article  MATH  MathSciNet  Google Scholar 

  5. Rockafellar, R.T.: Augmented Lagrangains and applications of the proximal point algorithm in convex programming. Math. Oper. Res. 1, 97–116 (1976)

    Article  MATH  MathSciNet  Google Scholar 

  6. Lan, G., Monteiro, R.: Iteration-complexity of first-order augmented Lagrangian methods for convex programming. Math. Program. 155, 511–547 (2016)

    Article  MATH  MathSciNet  Google Scholar 

  7. Yamashita, H., Yabe, H.: A survey of numerical methods for nonlinear semidefinite programming. J. Oper. Res. Soc. Jpn. 58, 24–60 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  8. Zhao, X.-Y.: A semismooth Newton-CG augmented Lagrangian method for large scale linear and convex quadratic SDPs, Ph.D. Thesis, National University of Singapore (2009)

  9. Zhao, X.-Y., Sun, D., Toh, K.-C.: A Newton-CG augmented Lagrangian method for semidefinite programming. SIAM J. Optim. 20, 1737–1765 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  10. Sun, D., Sun, J., Zhang, L.: The rate of convergence of the augmented Lagrangian method for nonlinear semidefinite programming. Math. Program. 114, 349–391 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  11. Alizadeh, F., Haeberly, J.P.A., Overton, O.L.: Complementarity and nondegeneracy in semidefinite programming. Math. Program. 77, 111–128 (1997)

    MATH  MathSciNet  Google Scholar 

  12. Qi, H., Sun, D.: A quadratically convergent Newton method for computing the nearest correlation matrix. SIAM J. Matrix Anal. Appl. 28, 360–385 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  13. Li, M., Sun, D., Toh, K.-C.: A majorized ADMM with indefinite proximal terms for linearly constrained convex composite optimization. SIAM J. Optim. 26, 922–950 (2016)

    Article  MATH  MathSciNet  Google Scholar 

  14. Liu, Y., Sun, D., Toh, K.-C.: An implementable proximal point algorithmic framework for nuclear norm minimization. Math. Program. 133, 399–436 (2012)

    Article  MATH  MathSciNet  Google Scholar 

  15. Qi, H., Sun, D.: An augmented Lagrangian dual approach for the H-weighted nearest correlation matrix problem. IMA J. Numer. Anal. 31, 491–511 (2011)

    Article  MATH  MathSciNet  Google Scholar 

  16. Wang, C., Sun, D., Toh, K.-C.: Solving log-determinant optimization problems by a Newton-CG primal proximal point algorithm. SIAM J. Optim. 20, 2994–3013 (2010)

    Article  MATH  MathSciNet  Google Scholar 

  17. Yang, L., Sun, D., Toh, K.-C.: SDPNAL+: a majorized semismooth Newton-CG augmented Lagrangian method for semidefinite programming with nonnegative constraints. Math. Program. Comput. 7, 331–366 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  18. Devolder, O., Glineur, F., Nesterov, Y.: First-order methods of smooth convex optimization with inexact oracle. Math. Program. 146, 37–75 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  19. Nedelcu, V., Necoara, I., Tran-Dinh, Q.: Computational complexity of inexact gradient augmented Lagrangian methods: application to constrained MPC. SIAM J. Control Optim. 52, 3109–3134 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  20. Chen, L., Sun, D., Toh, K.-C.: An efficient inexact symmetric Gauss–Seidel based majorized ADMM for high-dimensional convex composite conic programming. Math. Program. 161, 237–270 (2017)

    Article  MATH  MathSciNet  Google Scholar 

  21. Fazel, M., Pong, T.-K., Sun, D., Tseng, P.: Hankel matrix rank minimization with applications to system identification and realization. SIAM J. Matrix Anal. Appl. 34, 946–977 (2013)

    Article  MATH  MathSciNet  Google Scholar 

  22. Ouyang, Y., Chen, Y., Lan, G., Pasiliao, E.: An accelerated linearized alternating direction method of multipliers. SIAM J. Imaging Sci. 8, 644–681 (2015)

    Article  MATH  MathSciNet  Google Scholar 

  23. Li, X., Sun, D., Toh, K.-C.: QSDPNAL: a two-phase proximal augmented Lagrangian method for convex quadratic semidefinite programming (2015). arXiv:1512.08872v1, [math.OC]

  24. Toh, K.-C.: An inexact primal-dual path following algorithm for convex quadratic SDP. Math. Program. 112, 221–254 (2008)

    Article  MATH  MathSciNet  Google Scholar 

  25. Toh, K.-C., Tütüncü, R.H., Todd, M.J.: Inexact primal-dual path-following algorithms for a special class of convex quadratic SDP and related problems. Pac. J. Optim. 3, 135–164 (2007)

    MATH  MathSciNet  Google Scholar 

  26. Fortin, M., Glowinski, R.: On decomposition-coordination methods using an augmented Lagrangian. In: Fortin, M., Glowinski, R. (eds.) Augmented Lagrangian Methods: Applications to the Solution of Boundary Problems, pp. 97–146. NorthCHolland, Amsterdam (1983)

    Chapter  Google Scholar 

  27. Wang, C.: On how to solve large-scale log-determinant optimization problems. Comput. Optim. Appl. 64, 489–511 (2016)

    Article  MATH  MathSciNet  Google Scholar 

  28. Natsoulis, G., Pearson, C.I., Gollub, J., Eynon, B.P., Ferng, J., Nair, R., Idury, R., Lee, M.D., Fielden, M.R., Brennan, R.J., Roter, A.H., Jarnagin, K.: The liver pharmacological and xenobiotic gene response repertoire. Mol. Syst. Biol. 175, 1–12 (2008)

    Google Scholar 

  29. Hughes, T.R., Marton, M.J., Jones, A.R., Roberts, C.J., Stoughton, R., Armour, C.D., Bennett, H.A., Coffey, E., Dai, H., He, Y.D., Kidd, M.J., King, A.M., Meyer, M.R., Slade, D., Lum, P.Y., Stepaniants, S.B., Shoemaker, D.D., Gachotte, D., Chakraburtty, K., Simon, J., Bard, M., Friend, S.H.: Functional discovery via a compendium of expression profiles. Cell 102, 109–126 (2000)

    Article  Google Scholar 

Download references

Acknowledgements

We sincerely thank Professor Defeng Sun at the National University of Singapore who gave us many helpful comments and suggestions, which improved the quality of this paper greatly. We also thank the editor and the anonymous referees for their many suggestions that greatly help improve the manuscript.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Peipei Tang.

Additional information

Chengjing Wang was supported by the National Natural Science Foundation of China under Grant 11201382, the Youth Fund of Humanities and Social Sciences of the Ministry of Education under grant 12YJC910008. Peipei Tang was partially supported by the Natural Science Foundation of Zhejiang Province of China under Grant LQ13A010006, the Science & Technology Development Project of Hangzhou, China (No. 20162013A08).

Appendix

Appendix

In this section, we prove Proposition 3.2, Theorem 3.1, and Lemma 3.1. Firstly, we prove Proposition 3.2.

Proof

(Proof of proposition 3.2) From (6), we know that

$$\begin{aligned} \nabla f(x^{k+1})= & {} \mathcal{A}^*y^k - \sigma \mathcal{A}^*(\mathcal{A}(x^{k+1})-b)) + \sigma (x^k-x^{k+1})\nonumber \\&\quad + \Pi _{K^*}(z^k-\sigma x^k) + d^k. \end{aligned}$$
(32)

And from the MNAL method, we have

$$\begin{aligned} y^{k+1}= & {} y^k-\tau \sigma (\mathcal{A}(x^{k+1})-b),\nonumber \\ z^{k+1}= & {} z^k-\tau \sigma (x^{k+1}-s^{k+1})=z^k-\tau (\sigma x^{k+1}-\Pi _{K}(\sigma x^{k+1}-z^k))\\= & {} z^k+\tau (\Pi _{K^*}(z^k-\sigma x^{k+1})-z^k).\nonumber \end{aligned}$$
(33)

Then (32) and (33) imply that

$$\begin{aligned} \nabla f(x^{k+1})= & {} \mathcal{A}^*(y^{k+1}-(1-\tau )\sigma (\mathcal{A}(x^{k+1})-b))+ \sigma (x^k-x^{k+1})\nonumber \\&\quad + \Pi _{K^*}(z^k-\sigma x^k) + d^k. \end{aligned}$$
(34)

Meanwhile, from the KKT condition (1), we have

$$\begin{aligned} \nabla f(\overline{x}) - \mathcal{A}^*\overline{y} - \overline{z}=0. \end{aligned}$$
(35)

By (34), (35) and (36),

we obtain that

$$\begin{aligned} \langle \mathcal{A}^*(y^{k+1}-(1-\tau )\sigma (\mathcal{A}(x^{k+1})-b)+ \sigma (x^k-x^{k+1})\\ + \Pi _{K^*}(z^k-\sigma x^k) + d^k - \mathcal{A}^*\overline{y} - \overline{z}, \, x^{k+1}-\overline{x}\rangle \\ \ge \Vert x^{k+1}_{e}\Vert ^2_{\Sigma _{f}}. \end{aligned}$$

Furthermore, we obtain that

$$\begin{aligned}&\langle y^{k+1}_{e}, \, \mathcal{A}(x^{k+1}-\overline{x})\rangle -(1-\tau )\sigma \langle \mathcal{A}(x^{k+1})-b, \, \mathcal{A}(x^{k+1})-\mathcal{A}(\overline{x})\rangle \\&\quad +\sigma \langle x^{k}_{e}{-}x^{k+1}_{e}, \, x^{k+1}_{e}\rangle {+} \langle \Pi _{K^*}(z^k{-}\sigma x^k)-\overline{z}, \, x^{k+1}_{e}\rangle + \langle d^k, \, x^{k+1}_{e}\rangle \ge \Vert x^{k+1}_{e}\Vert ^2_{\Sigma _{f}}, \end{aligned}$$

i.e.,

$$\begin{aligned}&\frac{1}{\tau \sigma }\langle y^{k+1}_{e}, \, y^{k}_{e}-y^{k+1}_{e}\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2+\sigma \langle x^{k}_{e}-x^{k+1}_{e}, \, x^{k+1}_{e}\rangle \nonumber \\&\qquad +\langle \Pi _{K^*}(z^k-\sigma x^k)-\overline{z}, \, x^{k+1}_{e}\rangle + \langle d^k, \, x^{k+1}_{e}\rangle \ge \Vert x^{k+1}_{e}\Vert ^2_{\Sigma _{f}}. \end{aligned}$$
(36)

Here we note that the term

$$\begin{aligned} \langle \Pi _{K^*}(z^k-\sigma x^k)-\overline{z}, \, x^{k+1}_{e}\rangle= & {} \langle \Pi _{K^*}(z^k-\sigma x^k)-\Pi _{K^*}(z^k-\sigma x^{k+1}), \, x^{k+1}_{e}\rangle \nonumber \\&+ \langle \Pi _{K^*}(z^k-\sigma x^{k+1})-\overline{z}, \, x^{k+1}_{e}\rangle \nonumber \\= & {} \langle \Pi _{K^*}(z^k-\sigma x^k)-\Pi _{K^*}(z^k-\sigma x^{k+1}), \, x^{k+1}_{e}\rangle \nonumber \\&+ \langle z^{k+1}+(1-\tau )(\Pi _{K^*}(z^k-\sigma x^{k+1})-z^k)\nonumber \\&-\overline{z}, \, x^{k+1}_{e}\rangle . \end{aligned}$$
(37)

Since

$$\begin{aligned} x^{k+1}_{e}= & {} \frac{1}{\sigma }(\sigma x^{k+1}-z^k-\sigma \overline{x}+z^k)\\= & {} \frac{1}{\sigma }(\Pi _{K}(\sigma x^{k+1}-z^k)-\Pi _{K^*}(z^k-\sigma x^{k+1})-\sigma \overline{x}+z^k)\\= & {} \frac{1}{\sigma }(\Pi _{K}(\sigma x^{k+1}-z^k)-\sigma \overline{x}) + \frac{1}{\tau \sigma }(z^k-z^{k+1}), \end{aligned}$$

where the third equality is according to (33), thus

$$\begin{aligned}&\langle \Pi _{K^*}(z^k-\sigma x^{k+1})-\overline{z}, \, x^{k+1}_{e}\rangle \nonumber \\&= \frac{1}{\tau \sigma }\langle z^{k+1}_{e}, \, z^{k}_{e}-z^{k+1}_{e}\rangle \nonumber \\&\quad -\frac{1-\tau }{\tau \sigma }\langle \Pi _{K^*}(z^k-\sigma x^{k+1})-z^k, \, z^{k+1}-z^{k}\rangle \nonumber \\&\quad + \frac{1}{\sigma }\langle \Pi _{K}(\sigma x^{k+1}-z^k)-\sigma \overline{x}, \, \Pi _{K^*}(z^k-\sigma x^{k+1})-\overline{z}\rangle \nonumber \\&=\frac{1}{\tau \sigma }\langle z^{k+1}_{e}, \, z^{k}_{e}-z^{k+1}_{e}\rangle - \frac{1-\tau }{\sigma }\Vert \Pi _{K^*}(z^k-\sigma x^{k+1})-z^k\Vert ^2\nonumber \\&\quad -\frac{1}{\sigma }\langle \Pi _{K}(\sigma x^{k+1}-z^k), \, \overline{z}\rangle -\langle \Pi _{K^*}(z^k-\sigma x^{k+1}), \, \overline{x}\rangle . \end{aligned}$$
(38)

By substituting (38) into (37), and then substituting (37) into (36), we obtain that

$$\begin{aligned}&\frac{1}{\tau \sigma }\langle y^{k+1}_{e}, \, y^{k}_{e}-y^{k+1}_{e}\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\\&\qquad +\sigma \langle x^{k}_{e}-x^{k+1}_{e}, \, x^{k+1}_{e}\rangle +\frac{1}{\tau \sigma }\langle z^{k+1}_{e}, \, z^{k}_{e}-z^{k+1}_{e}\rangle \\&\qquad -\frac{1-\tau }{\sigma }\Vert \Pi _{K^*}(z^k-\sigma x^{k+1})-z^k\Vert ^2+\langle \Pi _{K^*}(z^k-\sigma x^k)\\&\qquad -\Pi _{K^*}(z^k-\sigma x^{k+1}), \, x^{k+1}_{e}\rangle + \langle d^k, \, x^{k+1}_{e}\rangle \ge \Vert x^{k+1}_{e}\Vert ^2_{\Sigma _{f}}\\&\qquad +\langle \Pi _{K^*}(z^k-\sigma x^{k+1}), \, \overline{x}\rangle +\frac{1}{\sigma }\langle \Pi _{K}(\sigma x^{k+1}-z^k), \, \overline{z}\rangle . \end{aligned}$$

Using the elementary relations \(\langle u, \, v\rangle =\frac{1}{2}(\Vert u\Vert ^2+\Vert v\Vert ^2-\Vert u-v\Vert ^2)=\frac{1}{2}(\Vert u+v\Vert ^2-\Vert u\Vert ^2-\Vert v\Vert ^2)\), we further obtain the following inequality

$$\begin{aligned}&\frac{1}{2\tau \sigma }\Vert y^{k}_{e}\Vert ^2-\frac{1}{2\tau \sigma }\Vert y^{k+1}_{e}\Vert ^2\\&\qquad -\frac{1}{2\tau \sigma }\Vert y^{k}_{e}-y^{k+1}_{e}\Vert ^2-(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2+\frac{\sigma }{2}\Vert x^{k}_{e}\Vert ^2-\frac{\sigma }{2}\Vert x^{k+1}_{e}\Vert ^2\\&\qquad -\frac{\sigma }{2}\Vert x^{k}_{e}-x^{k+1}_{e}\Vert ^2+\frac{1}{2\tau \sigma }\Vert z^{k}_{e}\Vert ^2\\&\qquad -\frac{1}{2\tau \sigma }\Vert z^{k+1}_{e}\Vert ^2-\frac{1}{2\tau \sigma }\Vert z^{k}_{e}-z^{k+1}_{e}\Vert ^2-\frac{1-\tau }{\sigma }\Vert \Pi _{K^*}(z^k-\sigma x^{k+1})-z^k\Vert ^2\\&\qquad +\frac{\sigma }{2}\Vert x^{k}_{e}-x^{k+1}_{e}\Vert ^2+\frac{\sigma }{2}\Vert x^{k+1}_{e}\Vert ^2+\langle d^k, \, x^{k+1}_{e}\rangle \\&\ge \Vert x^{k+1}_{e}\Vert ^2_{\Sigma _{f}}+\langle \Pi _{K^*}(z^k-\sigma x^{k+1}), \, \overline{x}\rangle +\frac{1}{\sigma }\langle \Pi _{K}(\sigma x^{k+1}-z^k), \, \overline{z}\rangle , \end{aligned}$$

then we have

$$\begin{aligned}&\frac{1}{\tau \sigma }\Vert y^{k}_{e}\Vert ^2-\frac{1}{\tau \sigma }\Vert y^{k+1}_{e}\Vert ^2\nonumber \\&\qquad -(2-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2+\sigma \Vert x^{k}_{e}\Vert ^2-\sigma \Vert x^{k+1}_{e}\Vert ^2+\frac{1}{\tau \sigma }\Vert z^{k}_{e}\Vert ^2-\frac{1}{\tau \sigma }\Vert z^{k+1}_{e}\Vert ^2\nonumber \\&\qquad -\frac{2-\tau }{\sigma }\Vert \Pi _{K^*}(z^k-\sigma x^{k+1})-z^k\Vert ^2+\sigma \Vert x^{k+1}_{e}\Vert ^2+2\langle d^k, \, x^{k+1}_{e}\rangle \nonumber \\&\ge 2\Vert x^{k+1}_{e}\Vert ^2_{\Sigma _{f}}+2\langle \Pi _{K^*}(z^k-\sigma x^{k+1}), \, \overline{x}\rangle +\frac{2}{\sigma }\langle \Pi _{K}(\sigma x^{k+1}-z^k), \, \overline{z}\rangle . \end{aligned}$$
(39)

Finally, we reformulate (39) to get the required inequality (20). \(\square \)

Secondly, we prove Theorem 3.1.

Proof (Proof of Theorem 3.1)

By Proposition 3.3, we obtain the following relation

$$\begin{aligned} \frac{1}{\tau \sigma }\Vert \overline{y}^{k+1}_{e}\Vert ^2+\frac{1}{\tau \sigma }\Vert \overline{z}^{k+1}_{e}\Vert ^2+\sigma \Vert \overline{x}^{k+1}_{e}\Vert ^2\le & {} \frac{1}{\tau \sigma }\Vert y^{k}_{e}\Vert ^2+\frac{1}{\tau \sigma }\Vert z^{k}_{e}\Vert ^2+\sigma \Vert x^{k}_{e}\Vert ^2\nonumber \\ \end{aligned}$$
(40)

Now we define the sequences \(\{\xi ^k\}\) and \(\{\overline{\xi }^k\}\) with

$$\begin{aligned} \xi ^k:= & {} \left( \frac{1}{\sqrt{\tau \sigma }}y^k_e,\frac{1}{\sqrt{\tau \sigma }}z^k_e,\sqrt{\sigma } x^k_e\right) ,\\ \overline{\xi }^k:= & {} \left( \frac{1}{\sqrt{\tau \sigma }}\overline{y}^k_e,\frac{1}{\sqrt{\tau \sigma }}\overline{z}^k_e,\sqrt{\sigma } \overline{x}^k_e\right) . \end{aligned}$$

Thus by (40), we get

$$\begin{aligned} \Vert \overline{\xi }^{k+1}\Vert\le & {} \Vert \xi ^{k}\Vert . \end{aligned}$$

Furthermore, we have

$$\begin{aligned} \Vert \xi ^{k+1}\Vert\le & {} \Vert \xi ^{k}\Vert +\Vert \overline{\xi }^{k+1}-\xi ^{k+1}\Vert . \end{aligned}$$
(41)

Next we estimate the bound of the last term in (41). First,

$$\begin{aligned} \Vert \overline{\xi }^{k+1}-\xi ^{k+1}\Vert ^2= & {} \frac{1}{\tau \sigma }\Vert \overline{y}^{k+1}-y^{k+1}\Vert ^2+\frac{1}{\tau \sigma }\Vert \overline{z}^{k+1}-z^{k+1}\Vert ^2\nonumber \\&\quad +\sigma \Vert \overline{x}^{k+1}-x^{k+1}\Vert ^2, \end{aligned}$$
(42)

and

$$\begin{aligned} \overline{y}^{k+1}-y^{k+1}= & {} y^k-\tau \sigma (\mathcal{A}(\overline{x}^{k+1}-b))-(y^k-\tau \sigma (\mathcal{A}(x^{k+1}-b)))\nonumber \\= & {} \tau \sigma \mathcal{A}(x^{k+1}-\overline{x}^{k+1}) \end{aligned}$$
(43)
$$\begin{aligned} \overline{z}^{k+1}-z^{k+1}= & {} z^k{+}\tau (\Pi _{K^*}(z^k{-}\sigma \overline{x}^{k+1}){-}z^k)-(z^k+\tau (\Pi _{K^*}(z^k-\sigma x^{k+1})-z^k))\nonumber \\= & {} \tau (\Pi _{K^*}(z^k-\sigma \overline{x}^{k+1})-\Pi _{K^*}(z^k-\sigma x^{k+1})). \end{aligned}$$
(44)

Then (42)–(44) imply that

$$\begin{aligned} \Vert \overline{\xi }^{k+1}{-}\xi ^{k+1}\Vert ^2\le & {} \tau \sigma \Vert \mathcal{A}(\overline{x}^{k+1}{-}x^{k+1})\Vert ^2{+}\tau \sigma \Vert \overline{x}^{k+1}{-}x^{k+1}\Vert ^2{+}\sigma \Vert \overline{x}^{k+1}-x^{k+1}\Vert ^2\nonumber \\= & {} \Vert \overline{x}^{k+1}-x^{k+1}\Vert ^2_{(\tau +1)\sigma \mathcal{I}+\tau \sigma \mathcal{A}^*\mathcal{A}}\le \varrho \varepsilon _k^2, \end{aligned}$$

where \(\varrho :=\Vert (\tau +1)\sigma \mathcal{I}+\tau \sigma \mathcal{A}^*\mathcal{A}\Vert /\underline{\lambda }\). Hence

$$\begin{aligned} \Vert \overline{\xi }^{k+1}-\xi ^{k+1}\Vert\le & {} \sqrt{\varrho }\varepsilon _k, \end{aligned}$$

then

$$\begin{aligned} \Vert \xi ^{k+1}\Vert\le & {} \Vert \xi ^{k}\Vert +\sqrt{\varrho }\varepsilon _k. \end{aligned}$$
(45)

From (45), we have

$$\begin{aligned} \Vert \xi ^{k+1}\Vert\le & {} \Vert \xi ^{1}\Vert +\sqrt{\varrho }\mathcal{E}. \end{aligned}$$
(46)

We can see from (46) that the sequences \(\{\xi ^k\}\) and \(\{\overline{\xi }^k\}\) are bounded. From the definition of \(\{\xi ^k\}\), we see that \(\{x^k\}\), \(\{y^k\}\) and \(\{z^k\}\) are also bounded.

Since the sequence \(\{(x^k,y^k,z^k)\}\) is bounded, it has a subsequence \(\{(x^{k_i},y^{k_i},z^{k_i})\}\) which converges to an accumulation point \((x^{\infty },y^{\infty },z^{\infty })\). Now we show that \((x^{\infty },y^{\infty },z^{\infty })\) is a KKT point. By Proposition 3.3, we have that

$$\begin{aligned}&\sum _{k=1}^{\infty }\left( \Vert \overline{x}^{k+1}_e\Vert ^2_{2\Sigma _f+(2-\tau )\sigma \mathcal{A}^*\mathcal{A}-\sigma \mathcal{I}}+\frac{2-\tau }{\sigma }\Vert \Pi _{K^*}(z^k-\sigma \overline{x}^{k+1})-z^k\Vert ^2\right) \\&\qquad \le \sum _{k=1}^{\infty }\left( (\phi _k(\overline{x},\overline{y},\overline{z})-\phi _{k+1}(\overline{x},\overline{y},\overline{z}))+(\phi _{k+1}(\overline{x},\overline{y},\overline{z})-\overline{\phi }_{k+1}(\overline{x},\overline{y},\overline{z}))\right) \\&\qquad \le \phi _1(\overline{x},\overline{y},\overline{z})+\sum _{k=1}^{\infty }\Vert \xi ^{k+1}-\overline{\xi }^{k+1}\Vert (\Vert \xi ^{k+1}\Vert +\Vert \overline{\xi }^{k+1}\Vert )\\&\qquad \le \phi _1(\overline{x},\overline{y},\overline{z})+\sqrt{\varrho }\mathcal{E}(\Vert \xi ^{k+1}\Vert +\Vert \overline{\xi }^{k+1}\Vert )<\infty , \end{aligned}$$

where we have used the inequality that \(\phi _{k+1}(\overline{x},\overline{y},\overline{z})-\overline{\phi }_{k+1}(\overline{x},\overline{y},\overline{z})=\Vert \xi ^{k+1}\Vert ^2-\Vert \overline{\xi }^{k+1}\Vert ^2\le \Vert \xi ^{k+1}-\overline{\xi }^{k+1}\Vert (\Vert \xi ^{k+1}\Vert +\Vert \overline{\xi }^{k+1}\Vert )\).

From the summability of the sequences \(\{\Vert \overline{x}^{k+1}_e\Vert ^2_{2\Sigma _f+(2-\tau )\sigma \mathcal{A}^*\mathcal{A}-\sigma \mathcal{I}}\}\) and \(\{\Vert \Pi _{K^*}(z^k-\sigma \overline{x}^{k+1})-z^k\Vert ^2\}\), we have that

$$\begin{aligned} \lim _{k\rightarrow \infty }\left( \Vert \overline{x}^{k+1}_e\Vert ^2_{2\Sigma _f+(2-\tau )\sigma \mathcal{A}^*\mathcal{A}-\sigma \mathcal{I}}+\frac{2-\tau }{\sigma }\Vert \Pi _{K^*}(z^k-\sigma \overline{x}^{k+1})-z^k\Vert ^2\right) =0.\qquad \end{aligned}$$
(47)

By (47) and (22), we obtain that

$$\begin{aligned} \lim _{k\rightarrow \infty }\Vert \overline{x}^{k+1}_e\Vert =0=\lim _{k\rightarrow \infty }\Vert \Pi _{K^*}(z^k-\sigma \overline{x}^{k+1})-z^k\Vert . \end{aligned}$$
(48)

From (48), we get

$$\begin{aligned} \lim _{k\rightarrow \infty }\Vert \overline{x}^{k+1}-\overline{x}^{k}\Vert= & {} 0. \end{aligned}$$

In addition to \(\Vert \overline{x}^{k+1}-x^{k+1}\Vert \le \varepsilon _k/\sqrt{\underline{\lambda }}\), we can get

$$\begin{aligned} \lim _{k\rightarrow \infty }\Vert x^{k+1}-x^{k}\Vert =0=\lim _{k\rightarrow \infty }\Vert x^{k+1}-\overline{x}\Vert , \end{aligned}$$
(49)

and

$$\begin{aligned} \lim _{k\rightarrow \infty }\Vert \Pi _{K^*}(z^k-\sigma x^{k+1})-z^k\Vert= & {} 0. \end{aligned}$$
(50)

Based on (11) and (49), we obtain that

$$\begin{aligned} \lim _{k\rightarrow \infty }\Vert \mathcal{A}(x^{k+1})-b\Vert =\lim _{k\rightarrow \infty }\Vert y^{k+1}-y^{k}\Vert =0. \end{aligned}$$
(51)

From (48), (49), (50), and (51), we see that by taking limits on both sides of (34) along the subsequence \(\{x^{k_i},y^{k_i},z^{k_i}\}\), we obtain that

$$\begin{aligned} \nabla f(x^{\infty }) - \mathcal{A}^*y^{\infty } - z^{\infty }=0. \end{aligned}$$

Moreover, from (51) and (50), we also obtain that

$$\begin{aligned} \mathcal{A}(x^{\infty })-b=0,\ x^{\infty }-\frac{1}{\sigma }\Pi _{K}(\sigma x^{\infty }-z^{\infty })=0. \end{aligned}$$

So \((x^{\infty },y^{\infty },z^{\infty })\) is a KKT point. Without loss of generality, we can assume \((x^{\infty },y^{\infty },z^{\infty })=(\overline{x},\overline{y},\overline{z})\). From (45), we have for any \(k\ge k_i\),

$$\begin{aligned} \Vert \xi ^{k+1}\Vert\le & {} \Vert \xi ^{k_i}\Vert +\sum _{j=k_i}^{k}\sqrt{\varrho }\varepsilon _j. \end{aligned}$$

Since \(\lim _{k_i\rightarrow \infty }\Vert \xi ^{k_i}\Vert =0\) and \(\{\varepsilon _k\}\) is summable, we have that \(\lim _{k\rightarrow \infty }\Vert \xi ^k\Vert =0\). Finally, making use of the definition of \(\xi ^k\), we get

$$\begin{aligned} \lim _{k\rightarrow \infty }x^{k}=x^{\infty }=\overline{x},\ \lim _{k\rightarrow \infty }y^{k}=y^{\infty }=\overline{y},\ \text {and}\ \lim _{k\rightarrow \infty }z^{k}=x^{\infty }=\overline{z}. \end{aligned}$$

That is, we have proved the convergence of the sequence \(\{x^k,y^k,z^k\}\) to an optimal solution. \(\square \)

Finally, we present the proof of Lemma 3.1.

Proof

(Proof of Lemma 3.1) In combination of (32) and (11), we have

$$\begin{aligned} \nabla f(x^{k+1})= & {} \mathcal{A}^*[y^{k+1}-(1-\tau )\sigma (\mathcal{A}(x^{k+1})-b)]\nonumber \\&\quad -\sigma (x^{k+1}-x^{k})+\Pi _{K^*}(z^{k}-\sigma x^k)+d^k. \end{aligned}$$
(52)

By the convexity of \(f(\cdot )\), (52), (11) and (12), we conclude that

$$\begin{aligned}&Q(\overline{x},\overline{s},y,z;w^{k+1}) = [f(x^{k+1})-\langle y, \, \mathcal{A}(x^{k+1})-b\rangle -\langle z, \, x^{k+1}-s^{k+1}\rangle ]-[f(\overline{x})\nonumber \\&\quad -\langle y^{k+1}, \, \mathcal{A}(\overline{x})-b\rangle -\langle z^{k+1}, \, \overline{x}-\overline{s}\rangle ]\nonumber \\&\le \langle \nabla f(x^{k+1}), \, x^{k+1}-\overline{x}\rangle -\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}\nonumber \\&\quad -\langle y, \, \mathcal{A}(x^{k+1})-b\rangle -\langle z, \, x^{k+1}-s^{k+1}\rangle \nonumber \\&= \langle \mathcal{A}^*[y^{k+1}-(1-\tau )\sigma (\mathcal{A}(x^{k+1})-b)]-\sigma (x^{k+1}-x^{k})+\Pi _{K^*}(z^{k}-\sigma x^k)\nonumber \\&\quad +d^k, x^{k+1}-\overline{x}\rangle -\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}-\langle y, \, \mathcal{A}(x^{k+1})-b\rangle -\langle z, \, x^{k+1}-s^{k+1}\rangle \nonumber \\&= \langle y^{k+1}-y, \, \mathcal{A}(x^{k+1})-b\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\nonumber \\&\quad +\sigma \langle x^{k}-x^{k+1}, \, x^{k+1}-\overline{x}\rangle \nonumber \\&\quad +\langle \Pi _{K^*}(z^{k}-\sigma x^k)-\Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle \nonumber \\&\quad +\langle \Pi _{K^*}(z^{k}-\sigma x^{k+1}), x^{k+1}-\overline{x}\rangle -\langle z, \, x^{k+1}-s^{k+1}\rangle +\langle d^k, \, x^{k+1}-\overline{x}\rangle \nonumber \\&\quad -\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}\nonumber \\&= \frac{1}{\tau \sigma }\langle y^{k}-y^{k+1}, \, y^{k+1}-y\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\nonumber \\&\quad +\sigma \langle x^{k}-x^{k+1}, \, x^{k+1}-\overline{x}\rangle \nonumber \\&\quad +\langle \Pi _{K^*}(z^{k}-\sigma x^k)-\Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle -\langle z, \, x^{k+1}-s^{k+1}\rangle \nonumber \\&\quad + \langle \Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, s^{k+1}-\overline{x}-\frac{1}{\tau \sigma }(z^{k+1}-z^k)\rangle +\langle d^k, \, x^{k+1}-\overline{x}\rangle \nonumber \\&\quad -\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f} \end{aligned}$$

By taking advantage of the fact that \(\langle \Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, s^{k+1}-\overline{x}\rangle \le 0\), we have

$$\begin{aligned} Q(\overline{x},\overline{s},y,z;w^{k+1})\le & {} \frac{1}{\tau \sigma }\langle y^{k}-y^{k+1}, \, y^{k+1}-y\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\nonumber \\&+\sigma \langle x^{k}-x^{k+1}, \, x^{k+1}-\overline{x}\rangle +\langle \Pi _{K^*}(z^{k}-\sigma x^k)\nonumber \\&-\Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle -\langle z, \, x^{k+1}-s^{k+1}\rangle \nonumber \\&- \frac{1}{\tau \sigma }\langle \Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, z^{k+1}-z^k\rangle +\langle d^k, \, x^{k+1}-\overline{x}\rangle \nonumber \\&-\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}\nonumber \\\le & {} \frac{1}{\tau \sigma }\langle y^{k}-y^{k+1}, \, y^{k+1}-y\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\nonumber \\&+\sigma \langle x^{k}-x^{k+1}, \, x^{k+1}-\overline{x}\rangle \nonumber \\&+\langle \Pi _{K^*}(z^{k}-\sigma x^k)-\Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle \nonumber \\&-\frac{1}{\tau \sigma }\langle \Pi _{K^*}(z^{k}-\sigma x^{k+1})-z, \, z^{k+1}-z^k\rangle \nonumber \\&+\langle d^k, \, x^{k+1}-\overline{x}\rangle -\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}\nonumber \\= & {} \frac{1}{\tau \sigma }\langle y^{k}-y^{k+1}, \, y^{k+1}-y\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\nonumber \\&+\sigma \langle x^{k}-x^{k+1}, \, x^{k+1}-\overline{x}\rangle \nonumber \\&+\langle \Pi _{K^*}(z^{k}-\sigma x^k)-\Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle \nonumber \\&-\frac{1}{\tau \sigma }\langle z^{k+1}+(1-\tau )(\Pi _{K^*}(z^{k}\nonumber \\&-\sigma x^{k+1})-z^k)-z, \, z^{k+1}-z^k\rangle +\langle d^k, \, x^{k+1}-\overline{x}\rangle \nonumber \\&-\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}\nonumber \\= & {} \frac{1}{\tau \sigma }\langle y^{k}-y^{k+1}, \, y^{k+1}-y\rangle -(1-\tau )\sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2\nonumber \\&+\sigma \langle x^{k}-x^{k+1}, \, x^{k+1}-\overline{x}\rangle \nonumber \\&+\langle \Pi _{K^*}(z^{k}-\sigma x^k)-\Pi _{K^*}(z^{k}\nonumber \\&-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle -\frac{1}{\tau \sigma }\langle z^{k+1}-z, \, z^{k+1}-z^k\rangle \nonumber \\&-\frac{1-\tau }{\tau \sigma }\langle \Pi _{K^*}(z^{k}\nonumber \\&-\sigma x^{k+1}){-}z^k, \, \tau (\Pi _{K^*}(z^{k}{-}\sigma x^{k+1})-z^k)\rangle +\langle d^k, \, x^{k+1}-\overline{x}\rangle \nonumber \\&-\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f} \end{aligned}$$

Making use of the fact that \(\langle \Pi _{K^*}(z^{k}-\sigma x^k)-\Pi _{K^*}(z^{k}-\sigma x^{k+1}), \, x^{k+1}-\overline{x}\rangle \le \sigma \Vert x^{k+1}-x^{k}\Vert \Vert x^{k+1}-\overline{x}\Vert \le \frac{\sigma }{2}(\Vert x^{k+1}-x^{k}\Vert ^2+\Vert x^{k+1}-\overline{x}\Vert ^2)\), we have

$$\begin{aligned} Q(\overline{x},\overline{s},y,z;w^{k+1})\le & {} \frac{1}{2\tau \sigma }\Vert y^k-y\Vert ^2-\frac{1}{2\tau \sigma }\Vert y^{k+1}-y\Vert ^2\nonumber \\&-(1-\frac{\tau }{2}) \sigma \Vert \mathcal{A}(x^{k+1})-b\Vert ^2+\frac{\sigma }{2}\Vert x^k-\overline{x}\Vert ^2\nonumber \\&-\frac{1}{2}\Vert x^{k+1}-\overline{x}\Vert ^2_{\Sigma _f}+\frac{1}{2\tau \sigma }\Vert z^k-z\Vert ^2-\frac{1}{2\tau \sigma }\Vert z^{k+1}-z\Vert ^2\nonumber \\&-\frac{1-\frac{1}{2}\tau }{\sigma }\Vert \Pi _{K^*}(z^{k}-\sigma x^{k+1})-z^k\Vert ^2+\langle d^k, \, x^{k+1}-\overline{x}\rangle \nonumber \\\le & {} \frac{1}{2\tau \sigma }\Vert y^k-y\Vert ^2-\frac{1}{2\tau \sigma }\Vert y^{k+1}-y\Vert ^2+\frac{1}{2\tau \sigma }\Vert z^k-z\Vert ^2\nonumber \\&-\frac{1}{2\tau \sigma }\Vert z^{k+1}-z\Vert ^2\nonumber \\&+\frac{\sigma }{2}\Vert x^k-\overline{x}\Vert ^2-\frac{\sigma }{2}\Vert x^{k+1}-\overline{x}\Vert ^2-\frac{1}{2}\Vert x^{k+1}\nonumber \\&-\overline{x}\Vert ^2_{\Sigma _f+(2-\tau )\sigma \mathcal{A}^*\mathcal{A}-\sigma \mathcal{I}} +\langle d^k, \, x^{k+1}-\overline{x}\rangle . \end{aligned}$$
(53)

Summing on both sides of the inequality (53), we could get the conclusion (25) as below

$$\begin{aligned}&\sum _{i=1}^{k}Q(\overline{x},\overline{s},y,z;w^{i+1})\nonumber \\&\le \sum _{i=1}^{k} \Big \{\frac{1}{2\tau \sigma }(\Vert y^i-y\Vert ^2-\Vert y^{i+1}-y\Vert ^2)+\frac{1}{2\tau \sigma }(\Vert z^i-z\Vert ^2-\Vert z^{i+1}-z\Vert ^2)\nonumber \\&\quad +\frac{\sigma }{2}(\Vert x^i-\overline{x}\Vert ^2-\Vert x^{i+1}-\overline{x}\Vert ^2)-\frac{1}{2}\Vert x^{i+1}-\overline{x}\Vert ^2_{\Sigma _f+(2-\tau )\sigma \mathcal{A}^*\mathcal{A}-\sigma \mathcal{I}}\Big \} \nonumber \\&\quad +\sum _{i=1}^{k}\langle d^i, \, x^{i+1}-\overline{x}\rangle \nonumber \\&\le \frac{1}{2\tau \sigma }(\Vert y^1-y\Vert ^2-\Vert y^{k+1}-y\Vert ^2) + \frac{1}{2\tau \sigma }(\Vert z^1-z\Vert ^2-\Vert z^{k+1}-z\Vert ^2) \nonumber \\&\quad +\frac{\sigma }{2}(\Vert x^1-\overline{x}\Vert ^2 - \Vert x^{k+1}-\overline{x}\Vert ^2)-\sum _{i=1}^{k}\frac{1}{2}\Vert x^{i+1}-\overline{x}\Vert ^2_{\Sigma _f+(2-\tau )\sigma \mathcal{A}^*\mathcal{A}-\sigma \mathcal{I}} \nonumber \\&\quad + \sum _{i=1}^{k}\langle d^i, \, x^{i+1}-\overline{x}\rangle . \end{aligned}$$
(54)

The result (27) follows immediately from (54) and the condition (26). \(\square \)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, C., Tang, P. A primal majorized semismooth Newton-CG augmented Lagrangian method for large-scale linearly constrained convex programming. Comput Optim Appl 68, 503–532 (2017). https://doi.org/10.1007/s10589-017-9930-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-017-9930-9

Keywords

Navigation