Skip to main content
Springer Nature Link
Log in

Quadratic regularization projected Barzilai–Borwein method for nonnegative matrix factorization

  • Published:
Data Mining and Knowledge Discovery Aims and scope Submit manuscript

Abstract

In this paper, based on the alternating nonnegative least squares framework, we present a new efficient method for nonnegative matrix factorization that uses a quadratic regularization projected Barzilai–Borwein (QRPBB) method to solve the subproblems. At each iteration, the QRPBB method first generates a point by solving a strongly convex quadratic minimization problem, which has a simple closed-form solution that is inexpensive to calculate, and then applies a projected Barzilai–Borwein method to update the solution of NMF. Global convergence result is established under mild conditions. Numerical comparisons of methods on both synthetic and real-world datasets show that the proposed method is efficient.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. http://www.csie.ntu.edu.tw/~cjlin/nmf/index.html.

  2. http://homepages.umflint.edu/~lxhan/software.html.

  3. http://www.cl.cam.ac.uk/research/dtg/attarchive/facedatabase.html.

  4. http://cbcl.mit.edu/software-datasets/FaceData2.html.

  5. Both Reuters-21578 corpus and TDT-2 corpus in MATLAB format are available at http://www.cad.zju.edu.cn/home/dengcai/Data/TextData.html.

References

  • Barzilai J, Borwein JM (1988) Two-point step size gradient methods. IMA J Numer Anal 8(1):141–148

    Article  MathSciNet  MATH  Google Scholar 

  • Berry MW, Browne M, Langville AN, Pauca VP, Plemmons RJ (2007) Algorithms and applications for approximate nonnegative matrix factorization. Computat Statist Data Anal 52(1):155–173

    Article  MathSciNet  MATH  Google Scholar 

  • Bertsekas DP (1999) Nonlinear programming, 2nd edn. Athena Scientific, Belmont

    MATH  Google Scholar 

  • Birgin EG, Martínez JM, Raydan M (2000) Nonmonotone spectral projected gradient methods on convex sets. SIAM J Optim 10(4):1196–1211

    Article  MathSciNet  MATH  Google Scholar 

  • Birgin EG, Martínez JM, Raydan M (2012) Spectral projected gradient methods: Review and perspectives. http://www.ime.usp.br/~egbirgin/

  • Bonettini S, Zanella R, Zanni L (2009) A scaled gradient projection method for constrained image deblurring. Inverse Probl 25(1):015002

    Article  MathSciNet  Google Scholar 

  • Cai D, He X, Han J (2005) Document clustering using locality preserving indexing. IEEE Trans Knowl Data Eng 17(12):1624–1637

    Article  Google Scholar 

  • Cai D, He X, Han J, Huang TS (2011) Graph regularized nonnegative matrix factorization for data representation. IEEE Trans Pattern Anal Mach Intell 33(8):1548–1560

    Article  Google Scholar 

  • Cichocki A, Zdunek R, Amari SI (2006) New algorithms for non-negative matrix factorization in applications to blind source separation. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pp 621–624

  • Cichocki A, Zdunek R, Phan AH, Amari SI (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, Chichester

    Book  Google Scholar 

  • Cieri C, Graff D, Liberman M, Martey N, Strassel S (1999) The TDT-2 text and speech corpus. In: Proceedings of the DARPA Broadcast News Workshop, pp 57–60

  • Cores D, Escalante R, González-Lima M, Jimenez O (2009) On the use of the spectral projected gradient method for support vector machines. Comp Optim Appl 28(3):327–364

    MATH  Google Scholar 

  • Dai YH, Zhang H (2001) Adaptive two-point stepsize gradient algorithm. Numer Algor 27(4):377–385

    Article  MathSciNet  MATH  Google Scholar 

  • Dai YH, Liao LZ (2002) \(R\)-linear convergence of the Barzilai and Borwein gradient method. IMA J Numer Anal 22(1):1–10

    Article  MathSciNet  MATH  Google Scholar 

  • Dai YH, Fletcher R (2005) Projected Barzilai-Borwein methods for large-scale box-constrained quadratic programming. Numer Math 100(1):21–47

    Article  MathSciNet  MATH  Google Scholar 

  • Dai YH, Hager WW, Schittkowski K, Zhang H (2006) The cyclic Barzilai–Borwein method for unconstrained optimization. IMA J Numer Anal 26(3):604–627

    Article  MathSciNet  MATH  Google Scholar 

  • Figueiredo MA, Nowak RD, Wright SJ (2007) Gradient projection for sparse reconstruction: Application to compressed sensing and other inverse problems. IEEE J Sel Top Signal Process 1(4):586–597

    Article  Google Scholar 

  • Fletcher R (2005) On the Barzilai–Borwein method. Optimization and control with applications, applied optimization. Springer, New York

    Google Scholar 

  • Gong P, Zhang C (2012) Efficient nonnegative matrix factorization via projected Newton method. Pattern Recognit 45(9):3557–3565

    Article  MATH  Google Scholar 

  • Grippo L, Sciandrone M (2000) On the convergence of the block nonlinear Gauss–Seidel method under convex constraints. Oper Res Lett 26(3):127–136

    Article  MathSciNet  MATH  Google Scholar 

  • Grippo L, Sciandrone M (2002) Nonmonotone globalization techniques for the Barzilai–Borwein gradient method. Comput Optim Appl 23(2):143–169

    Article  MathSciNet  MATH  Google Scholar 

  • Guan N, Tao D, Luo Z, Yuan B (2012) NeNMF: an optimal gradient method for nonnegative matrix factorization. IEEE Trans Signal Process 60(6):2882–2898

    Article  MathSciNet  Google Scholar 

  • Han L, Neumann M, Prasad U (2009) Alternating projected Barzilai-Borwein methods for nonnegative matrix factorization. Electron Trans Numer Anal 36(6):54–82

    MathSciNet  MATH  Google Scholar 

  • Hoyer PO (2004) Non-negative matrix factorization with sparseness constraints. J Mach Learn Res 5:1457–1469

    MathSciNet  MATH  Google Scholar 

  • Huang Y, Liu H, Zhou S (2013) A Barzilai–Borwein type method for stochastic linear complementarity problems. Numer Algor. doi:10.1007/s11075-013-9803-y

  • Kim H, Park H (2008) Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM J Matrix Anal Appl 30(2):713–730

    Article  MathSciNet  MATH  Google Scholar 

  • Kim D, Sra S, Dhillon IS (2007) Fast Newton-type methods for the least squares nonnegative matrix approximation problem. In: Proceedings of the 2007 SIAM International Conference on Data Mining, pp 343–354

  • Lee DD, Seung HS (2001) Algorithms for nonnegative matrix factorization. In: Advances in Neural Information Processing Systems, pp 556–562

  • Lee DD, Seung HS (1999) Learning the parts of objects by non-negative matrix factorization. Nature 401(6755):788–791

    Article  Google Scholar 

  • Lewis DD, Yang Y, Rose TG, Li F (2004) RCV1: a new benchmark collection for text categorization research. J Mach Learn Res 5:361–397

    Google Scholar 

  • Li SZ, Hou XW, Zhang HJ, Cheng QS (2001) Learning spatially localized, parts-based representation. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp 207–212

  • Lin CJ (2007) Projected gradient methods for nonnegative matrix factorization. Neural Comput 19(10):2756–2779

    Article  MathSciNet  MATH  Google Scholar 

  • Nesterov Y (1983) A method of solving a convex programming problem with convergence rate \(O(1/k^2)\). In: Soviet Mathematics Doklady, pp 372–376

  • Paatero P, Tapper U (1994) Positive matrix factorization: a non-negative factor model with optimal utilization of error estimates of data values. Environmetrics 5(2):111–126

    Article  Google Scholar 

  • Pascual-Montano A, Carazo JM, Kochi K, Lehmann D, Pascual-Marqui R (2006) Nonsmooth nonnegative matrix factorization (nsNMF). IEEE Trans Pattern Anal Mach Intell 28(3):403–415

    Article  Google Scholar 

  • Pauca VP (2004) Text mining using nonnegative matrix factorizations. In: Proceedings of the 2004 SIAM International Conference on Data Mining, pp 22–24

  • Raydan M (1997) The Barzilai and Borwein gradient method for the large scale unconstrained minimization problem. SIAM J Optim 7(1):26–33

    Article  MathSciNet  MATH  Google Scholar 

  • Shahnaz F, Berry MW, Pauca VP, Plemmons RJ (2006) Document clustering using nonnegative matrix factorization. J Inf Process Manage 42(2):373–386

    Article  MATH  Google Scholar 

  • Vavasis S (2009) On the complexity of nonnegative matrix factorization. SIAM J Optim 20(3):1364–1377

    Article  MathSciNet  MATH  Google Scholar 

  • Xu W, Liu X, Gong Y (2003) Document clustering based on non-negative matrix factorization. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pp 267–273

  • Zdunek R, Cichocki A (2006) Non-negative matrix factorization with quasi-Newton optimization. In: Proceedings of the 8th Artificial Intelligence and Soft Computing, pp 870–879

  • Zhang H, Hager WW (2004) A nonmonotone line search technique and its application to unconstrained optimization. SIAM J Optim 14(4):1043–1056

    Article  MathSciNet  MATH  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the handling associate editor Professor Kristian Kersting and the anonymous referees for their constructive comments and useful suggestions. This work is supported by the National Natural Science Foundation of China (NNSFC) under Grant No. 61072144 and No. 61179040 and the Fundamental Research Funds for the Central Universities No. K50513100007.

Author information

Authors and Affiliations

  1. School of Mathematics and Statistics, Xidian University, Xi’an, 710071, People’s Republic of China

    Yakui Huang, Hongwei Liu & Shuisheng Zhou

Authors
  1. Yakui Huang
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Hongwei Liu
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Shuisheng Zhou
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Yakui Huang.

Additional information

Responsible editor: Kristian Kersting.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, Y., Liu, H. & Zhou, S. Quadratic regularization projected Barzilai–Borwein method for nonnegative matrix factorization. Data Min Knowl Disc 29, 1665–1684 (2015). https://doi.org/10.1007/s10618-014-0390-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10618-014-0390-x

Keywords