Skip to main content
SpringerLink
Log in

Learning an Alternating Bergman Network for Non-convex and Non-smooth Optimization Problems

  • Conference paper
  • First Online:
Intelligence Science and Big Data Engineering (IScIDE 2017)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 10559))

  • 2297 Accesses

Abstract

Recently, non-convex and non-smooth problems have received considerable interests in the fields of image processing and machine learning. The proposed conventional algorithms rely on carefully designed initializations, and the parameters can not be tuned adaptively during iterations with corresponding to various real-world data. To settle these problems, we propose an alternating Bregman network (ABN), which discriminatively learns all the parameters from training pairs and then is directly applied to test data without additional operations. Specifically, parameters of ABN are adaptively learnt from training data to force the objective value drop rapidly toward the optimal and then obtain a desired solution in practice. Furthermore, the basis algorithm of ABN is an alternating method with Bregman modification (AMBM), which solves each subproblem with a designated Bregman distance. This AMBM is more general and flexible than previous approaches; at the same time it is proved to receive the best convergence result for general non-convex and non-smooth optimization problems. Thus, our proposed ABN is an efficient and converged algorithm which rapidly converges to desired solutions in practice. We applied ABN to sparse coding problem with \(\ell _0\) penalty and the experimental results verify the efficiency of our proposed algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The definition of KŁ function will be introduced afterwards.

References

  1. Berry, M.W., Brown, M., Langvill, A.N., Pauca, V.P., Plemmons, R.J.: Algorithms and applications for approximate nonnegative matrix factorization. Comput. Stat. Data Anal. 52(1), 155–173 (2007)

    Article  MathSciNet  Google Scholar 

  2. Ding, C., Li, T., Peng, W., Park, H.: Orthogonal nonnegative matrix t-factorizations for clustering. In: ACM SIGKDD (2006)

    Google Scholar 

  3. Zuo, W., Meng, D., Zhang, L., Feng, X., Zhang, D.: A generalized iterated shrinkage algorithm for non-convex sparse coding. In: ICCV, pp. 217–224 (2013)

    Google Scholar 

  4. Sandler, R., Lindenbaum, M.: Nonnegative matrix factorization with earth movers distance metric for image analysis. IEEE TPAMI 33(8), 1590–1602 (2011)

    Article  Google Scholar 

  5. Wang, Z., Ling, Q., Huang, T.S.: Learning deep \(\ell _0\) encoders. In: AAAI

    Google Scholar 

  6. Gong, P., Zhang, C., Lu, Z., Huang, J.Z., Ye, J.: A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization problems. In: ICML (2013)

    Google Scholar 

  7. Lu, C., Tang, J., Yan, S., Lin, Z.: Nonconvex nonsmooth low rank minimization via iteratively reweighted nuclear norm. IEEE TIP 25(2), 829–839 (2016)

    MathSciNet  Google Scholar 

  8. Cai, D., He, X., Han, J., Huang, T.S.: Graph regularized non-negative matrix factorization for data representation. IEEE TPAMI 33(8), 1548–1560 (2010)

    Google Scholar 

  9. Benetos, E., Kotropoulos, C.: Non-negative tensor factorization applied to music genre classification. IEEE TASLP 18(8), 1955–1967 (2010)

    Google Scholar 

  10. Jia, S., Qian, Y.: Constrained nonnegative matrix factorization for hyperspectral unmixing. IEEE TGRS 47(1), 161–173 (2009)

    MATH  Google Scholar 

  11. Peng, X., Lu, C., Yi, Z., Tang, H.: Connections between nuclear-norm and frobenius-norm-based representations. IEEE TNNLS

    Google Scholar 

  12. Deng, Y., Bao, F., Dai, Q.: A unified view of nonconvex heuristic approach for low-rank and sparse structure learning. In: Handbook of Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing

    Google Scholar 

  13. Yuan, G., Ghanem, B.: A proximal alternating direction method for semi-definite rank minimization. In: AAAI (2016)

    Google Scholar 

  14. Wang, Y., Liu, R., Song, X., Su, Z.: Linearized alternating direction method with penalization for nonconvex and nonsmooth optimization. In: AAAI (2016)

    Google Scholar 

  15. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: NIPS, pp. 556–562 (2001)

    Google Scholar 

  16. Wang, Y.X., Zhang, Y.J.: Nonnegative matrix factorization: a comprehensive review. IEEE TKDE 25(6), 1336–1353 (2013)

    Google Scholar 

  17. Shi, J., Ren, X., Dai, G., Wang, J.: A non-convex relaxation approach to sparse dictionary learning. In: CVPR (2011)

    Google Scholar 

  18. Zhang, C.H.: Nearly unbiased variable selection under minimax concave penalty. Ann. Stat. 38(2), 894–942 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  19. Attouch, H., Bolte, J.: On the convergence of the proximal algorithm for nonsmooth functions involving analytic features. MP 116(1–2), 5–16 (2009)

    MathSciNet  MATH  Google Scholar 

  20. Attouch, H., Bolte, J., Redont, P., Soubeyran, A.: Proximal alternating minimization and projection methods for nonconvex problems: an approach based on the kurdyka-lojasiewicz inequality. Math. Oper. Res. 35(2), 438–457 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  21. Bolte, J., Sabach, S., Teboulle, M.: Proximal alternating linearized minimization for nonconvex and nonsmooth problems. MP 146(1–2), 459–494 (2014)

    MathSciNet  MATH  Google Scholar 

  22. Xu, Y., Yin, W.: A globally convergent algorithm for nonconvex optimization based on block coordinate update, arXiv preprint arXiv:1410.1386

  23. Li, H., Lin, Z.: Accelerated proximal gradient methods for nonconvex programming. In: NIPS (2015)

    Google Scholar 

  24. Frankel, P., Garrigos, G., Peypouquet, J.: Splitting methods with variable metric for kurdyka-łojasiewicz functions and general convergence rates. J. Optim. Theory Appl. 165(3), 874–900 (2015)

    Article  MathSciNet  MATH  Google Scholar 

  25. Elad, M., Aharon, M.: Image denoising via sparse and redundant representations over learned dictionaries. IEEE TIP 15(12), 3736–3745 (2007)

    MathSciNet  Google Scholar 

  26. Pinghua, G., Zhang, C., Lu, Z., Huang, J., Jieping, Y.: A general iterative shrinkage and thresholding algorithm for non-convex regularized optimization porblems. In: ICML (2013)

    Google Scholar 

  27. Zuo, W., Ren, D., Gu, S., Lin, L.: Discriminative learning of iteration-wise priors for blind deconvolution. In: CVPR (2015)

    Google Scholar 

  28. Gregor, K., Lecun, Y.: Learning fast approximations of sparse coding. In: ICML (2010)

    Google Scholar 

  29. Foucart, S., Lai, M.-J.: Sparsest solutions of underdetermined linear systems via \(\ell _q\)-minimization for \(0< q \le 1\). ACHA 26(3), 395–407 (2009)

    MATH  Google Scholar 

  30. Fan, J., Li, R.: Variable selection via nonconcave penalized likelihood and its oracle properties. J. Am. Stat. Assoc. 96(456), 1348–1360 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  31. Banerjee, A., Merugu, S., Dhillon, I.S., Ghosh, J.: Clustering with bregman divergences. JMLR 6(4), 1705–1749 (2005)

    MathSciNet  MATH  Google Scholar 

  32. Fischer, A.: Quantization and clustering with bregman divergences. J. Multivar. Anal. 101(9), 2207–2221 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  33. Xu, L., Lu, C., Xu, Y., Jia, J.: Image smoothing via \(l_{0}\) gradient minimization. ACM TOG 30(6), 174 (2011)

    Google Scholar 

  34. Kang, Y., Zhang, Z., Li, W.: On the global convergence of majorization minimization algorithms for nonconvex optimization problems, arXiv preprint arXiv:1504.07791

  35. Nocedal, J., Wright, S.: Numerical Optimization. Springer Science & Business Media, New York (2006). doi:10.1007/978-0-387-40065-5

    MATH  Google Scholar 

  36. Sra, S., Nowozin, S., Wright, S.J.: Optimization for Machine Learning. MIT Press, Cambridge (2011)

    Google Scholar 

  37. Dempe, S.: Foundations of Bilevel Programming. Nonconvex Optimization & Its Applications, vol. 61

    Google Scholar 

  38. Ochs, P., Ranftl, R., Brox, T., Pock, T.: Bilevel optimization with nonsmooth lower level problems. In: Aujol, J.-F., Nikolova, M., Papadakis, N. (eds.) SSVM 2015. LNCS, vol. 9087, pp. 654–665. Springer, Cham (2015). doi:10.1007/978-3-319-18461-6_52

    Google Scholar 

  39. Schmidt, U., Roth, S.: Shrinkage fields for effective image restoration. In: CVPR (2014)

    Google Scholar 

Download references

Acknowledgements

Risheng Liu is supported by the National Natural Science Foundation of China (Nos. 61672125, 61300086, 61572096, 61432003 and 61632019), the Fundamental Research Funds for the Central Universities (DUT2017TB02) and the Hong Kong Scholar Program (No. XJ2015008). Zhixun Su is supported by National Natural Science Foundation of China (No. 61572099) and National Science and Technology Major Project (No. 2014ZX04001011).

Author information

Authors and Affiliations

  1. School of Mathematical Science, Dalian University of Technology, Dalian, China

    Yiyang Wang & Zhixun Su

  2. DUT-RU International School of Information Science and Engineering, Dalian University of Technology, Dalian, China

    Risheng Liu

  3. Shenzhen Key Laboratory of Media Security, Shenzhen University, Shenzhen, China

    Risheng Liu

  4. Key Laboratory for Ubiquitous Network and Service Software of Liaoning Province, Dalian, China

    Risheng Liu

  5. National Engineering Research Center of Digital Life, Guangzhou, China

    Zhixun Su

Authors
  1. Yiyang Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Risheng Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zhixun Su
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Risheng Liu .

Editor information

Editors and Affiliations

  1. Dalian University of Technology, Dalian, China

    Yi Sun

  2. Dalian University of Technology, Dalian, China

    Huchuan Lu

  3. Dalian University of Technology, Dalian, China

    Lihe Zhang

  4. Nanjing University of Science and Technology, Nanjing, China

    Jian Yang

  5. Beijing Institute of Technology, Beijing, China

    Hua Huang

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Wang, Y., Liu, R., Su, Z. (2017). Learning an Alternating Bergman Network for Non-convex and Non-smooth Optimization Problems. In: Sun, Y., Lu, H., Zhang, L., Yang, J., Huang, H. (eds) Intelligence Science and Big Data Engineering. IScIDE 2017. Lecture Notes in Computer Science(), vol 10559. Springer, Cham. https://doi.org/10.1007/978-3-319-67777-4_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-67777-4_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-67776-7

  • Online ISBN: 978-3-319-67777-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation