Skip to main content
SpringerLink
Log in

Smoothing projected Barzilai–Borwein method for constrained non-Lipschitz optimization

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

We present a smoothing projected Barzilai–Borwein (SPBB) algorithm for solving a class of minimization problems on a closed convex set, where the objective function is nonsmooth nonconvex, perhaps even non-Lipschitz. At each iteration, the SPBB algorithm applies the projected gradient strategy that alternately uses the two Barzilai–Borwein stepsizes to the smooth approximation of the original problem. Nonmonotone scheme is adopted to ensure global convergence. Under mild conditions, we prove convergence of the SPBB algorithm to a scaled stationary point of the original problem. When the objective function is locally Lipschitz continuous, we consider a general constrained optimization problem and show that any accumulation point generated by the SPBB algorithm is a stationary point associated with the smoothing function used in the algorithm. Numerical experiments on \(\ell _2\)-\(\ell _p\) problems, image restoration problems, and stochastic linear complementarity problems show that the SPBB algorithm is promising.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. Barzilai, J., Borwein, J.M.: Two-point step size gradient methods. IMA J. Numer. Anal. 8, 141–148 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  2. Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM J. Imaging Sci. 2(1), 183–202 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  3. Bertsekas, D.P.: Nonlinear Programming, 2nd edn. Athena Scientific, Belmont (1999)

    MATH  Google Scholar 

  4. Bian, W., Chen, X.: Smoothing neural network for constrained non-Lipschitz optimization with applications. IEEE Trans. Neural Netw. Lean. Syst. 23(3), 399–411 (2012)

    Article  Google Scholar 

  5. Bian, W., Chen, X.: Worst-case complexity of smoothing quadratic regularization methods for non-Lipschitzian optimization. SIAM J. Optim. 23(3), 1718–1741 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  6. Bian, W., Chen, X.: Neural network for nonsmooth, nonconvex constrained minimization via smooth approximation. IEEE Trans. Neural Netw. Lean. Syst. 25(3), 545–556 (2014)

    Article  Google Scholar 

  7. Bian, W., Chen, X., Ye, Y.: Complexity analysis of interior point algorithms for non-Lipschitz and nonconvex minimization. Math. Program. (2014). doi:10.1007/s10107-014-0753-5

  8. Birgin, E.G., Martínez, J.M., Raydan, M.: Nonmonotone spectral projected gradient methods on convex sets. SIAM J. Optim. 10(4), 1196–1211 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  9. Burke, J.V., Lewis, A.S., Overton, M.L.: A robust gradient sampling algorithm for nonsmooth, nonconvex optimization. SIAM J. Optim. 15(3), 751–779 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  10. Calamai, P.H., Moré, J.J.: Projected gradient methods for linearly constrained problems. Math. Program. 39, 93–116 (1987)

    Article  MathSciNet  MATH  Google Scholar 

  11. Candès, E.J., Romberg, J.: Quantitative robust uncertainty principles and optimally sparse decompositions. Found. Comput. Math. 6(2), 227–254 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  12. Candès, E.J., Romberg, J., Tao, T.: Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Trans. Inf. Theory 52(2), 489–509 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  13. Candès, E.J., Tao, T.: Near-optimal signal recovery from random projections: Universal encoding strategies. IEEE Trans. Inf. Theory 52(12), 5406–5425 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  14. Candès, E.J., Wakin, M.B., Boyd, S.P.: Enhancing sparsity by reweighted \(\ell _1\) minimization. J. Fourier Anal. Appl. 14(5–6), 877–905 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  15. Cartis, C., Gould, N.I., Toint, P.L.: On the evaluation complexity of composite function minimization with applications to nonconvex nonlinear programming. SIAM J. Optim. 21(4), 1721–1739 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  16. Chartrand, R.: Exact reconstruction of sparse signals via nonconvex minimization. IEEE Signal Process. Lett. 14(10), 707–710 (2007)

    Article  Google Scholar 

  17. Chartrand, R.: Nonconvex regularization for shape preservation. In: IEEE International Conference on Image Processing (ICIP) (2007)

  18. Chartrand, R.: Fast algorithms for nonconvex compressive sensing: MRI reconstruction from very few data. In: IEEE International Symposium on Biomedical Imaging (ISBI) (2009)

  19. Chartrand, R., Yin, W.: Iteratively reweighted algorithms for compressive sensing. In: IEEE international conference on Acoustics, speech, and signal processing (ICASSP) (2008)

  20. Chen, B., Chen, X.: A global and local superlinear continuation-smoothing method for \(P_0\) and \(R_0\) NCP or monotone NCP. SIAM J. Optim. 9(3), 624–645 (1999)

    Article  MathSciNet  Google Scholar 

  21. Chen, C., Mangasarian, O.L.: A class of smoothing functions for nonlinear and mixed complementarity problems. Comput. Optim. Appl. 5(2), 97–138 (1996)

    Article  MathSciNet  MATH  Google Scholar 

  22. Chen, S.S., Donoho, D.L., Saunders, M.A.: Atomic decomposition by basis pursuit. SIAM J. Sci. Comput. 20(1), 33–61 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  23. Chen, X.: Smoothing methods for nonsmooth, nonconvex minimization. Math. Program. 134, 71–99 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  24. Chen, X., Fukushima, M.: Expected residual minimization method for stochastic linear complementarity problems. Math. Oper. Res. 30(4), 1022–1038 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  25. Chen, X., Ge, D., Wang, Z., Ye, Y.: Complexity of unconstrained \(l_2\)-\(l_p\) minimization. Math. Program. 143, 371–383 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  26. Chen, X., Ng, M.K., Zhang, C.: Non-Lipschitz \(\ell _{p}\)-regularization and box constrained model for image restoration. IEEE Trans. Signal Process. 21(12), 4709–4721 (2012)

    MathSciNet  Google Scholar 

  27. Chen, X., Niu, L., Yuan, Y.: Optimality conditions and smoothing trust region Newton method for non-Lipschitz optimization. SIAM J. Optim. 23, 1528–1552 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  28. Chen, X., Xu, F., Ye, Y.: Lower bound theory of nonzero entries in solutions of \(\ell _2\)-\(\ell _p\) minimization. SIAM J. Sci. Comput. 32(5), 2832–2852 (2010)

    Article  MathSciNet  Google Scholar 

  29. Chen, X., Zhang, C., Fukushima, M.: Robust solution of monotone stochastic linear complementarity problems. Math. Program. 117, 51–80 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  30. Chen, X., Zhou, W.: Smoothing nonlinear conjugate gradient method for image restoration using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 3(4), 765–790 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  31. Clarke, F.H.: Optimization and Nonsmooth Analysis. Wiley, New York (1983)

    MATH  Google Scholar 

  32. Curtis, F.E., Overton, M.L.: A sequential quadratic programming algorithm for nonconvex, nonsmooth constrained optimization. SIAM J. Optim. 22(2), 474–500 (2012)

    Article  MathSciNet  MATH  Google Scholar 

  33. Dai, Y.H., Fletcher, R.: Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer. Math. 100(1), 21–47 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  34. Dai, Y.H., Hager, W.W., Schittkowski, K., Zhang, H.: The cyclic Barzilai–Borwein method for unconstrained optimization. IMA J. Numer. Anal. 26, 604–627 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  35. Dai, Y.H., Liao, L.Z.: \(R\)-linear convergence of the Barzilai and Borwein gradient method. IMA J. Numer. Anal. 22, 1–10 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  36. Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  37. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  38. Fang, H., Chen, X., Fukushima, M.: Stochastic \(R_0\) matrix linear complementarity problems. SIAM J. Optim. 18(2), 482–506 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  39. Figueiredo, M.A., Nowak, R.D., Wright, S.J.: Gradient projection for sparse reconstruction: application to compressed sensing and other inverse problems. IEEE J. Sel. Top. Signal Process. 1(4), 586–597 (2007)

    Article  Google Scholar 

  40. Fletcher, R.: On the Barzilai–Borwein method. Optimization and Control with Applications. Springer, New York (2005)

    Google Scholar 

  41. Garmanjani, R., Vicente, L.N.: Smoothing and worst-case complexity for direct-search methods in nonsmooth optimization. IMA J. Numer. Anal. 33, 1008–1028 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  42. Ge, D., Jiang, X., Ye, Y.: A note on the complexity of \(l_p\) minimization. Math. Program. 129, 285–299 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  43. Grippo, L., Sciandrone, M.: Nonmonotone globalization techniques for the Barzilai–Borwein gradient method. Comput. Optim. Appl. 23(2), 143–169 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  44. Huang, Y., Liu, H.: On the rate of convergence of projected Barzilai–Borwein methods. Optim. Method Softw. 30(4), 880–892 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  45. Huang, Y., Liu, H., Cong, W.: A note on the smoothing quadratic regularization method for non-Lipschitz optimization. Numer. Algorithm 69(4), 863–874 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  46. Huang, Y., Liu, H., Yu, T.: Smoothing projected cyclic Barzilai–Borwein method for stochastic linear complementarity problems. Int. J. Comput. Math. (2014). doi:10.1080/00207160.2015.1040780

  47. Huang, Y., Liu, H., Zhou, S.: A Barzilai–Borwein type method for stochastic linear complementarity problems. Numer. Algorithm 67(3), 477–489 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  48. Huang, Y., Liu, H., Zhou, S.: A Barzilai–Borwein type method for minimizing composite functions. Numer. Algorithm 69(4), 819–838 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  49. Huang, Y., Liu, H., Zhou, S.: Quadratic regularization projected Barzilai–Borwein method for nonnegative matrix factorization. Data Min. Knowl. Disc. 29(6), 1665–1684 (2015)

    Article  MathSciNet  Google Scholar 

  50. Huang, Y., Liu, H., Zhou, S.: An efficient monotone projected Barzilai–Borwein method for nonnegative matrix factorization. Appl. Math. Lett. 45, 12–17 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  51. Jiang, B., Dai, Y.H.: Feasible Barzilai–Borwein-like methods for extreme symmetric eigenvalue problems. Optim. Method Softw. 28(4), 756–784 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  52. Jiang, B., Dai, Y.H.: A framework of constraint preserving update schemes for optimization on Stiefel manifold. Math. Program. 153(2), 535–575 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  53. Kleywegt, A.J., Shapiro, A., Homem-De-Mello, T.: The sample average approximation method for stochastic discrete optimization. SIAM J. Optim. 12(2), 479–502 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  54. Lin, G.H., Fukushima, M.: Stochastic equilibrium problems and stochastic mathematical programs with equilibrium constraints: a survey. Pac. J. Optim. 6(3), 455–482 (2010)

    MathSciNet  MATH  Google Scholar 

  55. Liu, H., Huang, Y., Li, X.: New reformulation and feasible semismooth Newton method for a class of stochastic linear complementarity problems. Appl. Math. Comput. 217, 9723–9740 (2011)

    MathSciNet  MATH  Google Scholar 

  56. Liu, H., Huang, Y., Li, X.: Partial projected Newton method for a class of stochastic linear complementarity problems. Numer. Algorithm 58(4), 593–618 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  57. Liu, Y.F., Dai, Y.H., Ma, S.: Joint power and admission control: non-convex \(l_q\) approximation and an effective polynomial time deflation approach. IEEE Trans. Signal Process. 63(14), 3641–3656 (2015)

    Article  MathSciNet  Google Scholar 

  58. Liu, Y.F., Ma, S., Dai, Y.H., Zhang, S.: A smoothing SQP framework for a class of composite \(l_q\) minimization over polyhedron. Math. Program. (2015). doi:10.1007/s10107-015-0939-5

  59. Lu, Z.: Iterative reweighted minimization methods for \(\ell _p\) regularized unconstrained nonlinear programming. Math. Program. 147, 277–307 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  60. Mourad, N., Reilly, J.P.: Minimizing nonconvex functions for sparse vector reconstruction. IEEE Trans. Signal Process. 58(7), 3485–3496 (2010)

    Article  MathSciNet  Google Scholar 

  61. Nesterov, Yu.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  62. Nesterov, Yu.: Modified Gauss–Newton scheme with worst case guarantees for global performance. Optim. Method Softw. 22(3), 469–483 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  63. Nesterov, Yu.: Gradient methods for minimizing composite functions. Math. Program. 140(1), 125–161 (2013)

    Article  MathSciNet  MATH  Google Scholar 

  64. Nikolova, M., Ng, M.K., Tam, C.P.: Fast nonconvex nonsmooth minimization methods for image restoration and reconstruction. IEEE Trans. Image Process. 19(12), 3073–3088 (2010)

    Article  MathSciNet  Google Scholar 

  65. Nikolova, M., Ng, M.K., Zhang, S., Ching, W.K.: Efficient reconstruction of piecewise constant images using nonsmooth nonconvex minimization. SIAM J. Imaging Sci. 1(1), 2–25 (2008)

    Article  MathSciNet  MATH  Google Scholar 

  66. Raydan, M.: The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J. Optim. 7(1), 26–33 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  67. Tibshirani, R.: Regression shrinkage and selection via the lasso. J. R. Stat. Soc. B 58, 267–288 (1996)

    MathSciNet  MATH  Google Scholar 

  68. Toint, P.H.L.: Global convergence of a class of trust region methods for nonconvex minimization in Hilbert space. MA J. Numer. Anal. 8, 231–252 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  69. Wright, S.J., Nowak, R.D., Figueiredo, M.A.: Sparse reconstruction by separable approximation. IEEE Trans. Signal Process. 57(7), 2479–2493 (2009)

    Article  MathSciNet  Google Scholar 

  70. Zhang, C., Chen, X.: Smoothing projected gradient method and its application to stochastic linear complementarity problems. SIAM J. Optim. 20(2), 627–649 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  71. Zhou, G.L., Caccetta, L.: Feasible semismooth Newton method for a class of stochastic linear complementarity problems. J. Optim. Theory Appl. 139(2), 379–392 (2008)

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the associate editor and the anonymous reviewers for their insightful and constructive comments, which help to enrich the content and improve the presentation of this paper. We are very grateful to Professor Zhaosong Lu of Simon Fraser University for providing us the code of finding the parameter of the algorithm in [59] for numerical test. We also thanks Dr. Yu Han of Shenzhen University for his help in numerical experiments on image restoration problems. This work was supported by the National Natural Science Foundation of China (NNSFC) under Grant no. 61072144 and no. 61179040 and the Fundamental Research Funds for the Central Universities no. K50513100007

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xidian University, Xi’an, 710126, People’s Republic of China

    Yakui Huang & Hongwei Liu

  2. LSEC, ICMSEC, Academy of Mathematics and Systems Science, Chinese Academy of Sciences, Beijing, 100190, China

    Yakui Huang

Authors
  1. Yakui Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hongwei Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yakui Huang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, Y., Liu, H. Smoothing projected Barzilai–Borwein method for constrained non-Lipschitz optimization. Comput Optim Appl 65, 671–698 (2016). https://doi.org/10.1007/s10589-016-9854-9

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-016-9854-9

Keywords

Search

Navigation