Skip to main content
SpringerLink
Log in

Global convergence of modified multiplicative updates for nonnegative matrix factorization

  • Published:
Computational Optimization and Applications Aims and scope Submit manuscript

Abstract

Nonnegative matrix factorization (NMF) is the problem of approximating a given nonnegative matrix by the product of two nonnegative matrices. The multiplicative updates proposed by Lee and Seung are widely used as efficient computational methods for NMF. However, the global convergence of these updates is not formally guaranteed because they are not defined for all pairs of nonnegative matrices. In this paper, we consider slightly modified versions of the original multiplicative updates and study their global convergence properties. The only difference between the modified updates and the original ones is that the former do not allow variables to take values less than a user-specified positive constant. Using Zangwill’s global convergence theorem, we prove that any sequence of solutions generated by either of those modified updates has at least one convergent subsequence and the limit of any convergent subsequence is a stationary point of the corresponding optimization problem. Furthermore, we propose algorithms based on the modified updates that always stop within a finite number of iterations.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Algorithm 1
Algorithm 2

Similar content being viewed by others

Notes

  1. Although it is not explicitly written in their original paper [17] which of \(H^{k}\) and \(H^{k+1}\) is used for the computation of \(W^{k+1}\), we consider the latter case throughout this paper as in [18].

  2. The conditions (7)–(12) are derived by eliminating Lagrange multipliers in the original KKT conditions.

  3. The conditions (19)–(24) are derived by eliminating Lagrange multipliers in the original KKT conditions.

  4. Although \(A\) is assumed to be a point-to-set mapping in the original version of Zangwill’s global convergence theorem, we consider in this paper its special case where \(A\) is a point-to-point mapping.

References

  1. Badeau, R., Bertin, N., Vincent, E.: Stability analysis of multiplicative update algorithms and application to nonnegative matrix factorization. IEEE Trans. Neural Netw. 21(12), 1869–1881 (2010)

    Article  Google Scholar 

  2. Berry, M.W., Browne, M.: Email surveillance using non-negative matrix factorization. Comput. Math. Organ. Theory 11, 249–264 (2005)

    Article  MATH  Google Scholar 

  3. Berry, M.W., Browne, M., Langville, A.N., Pauca, V.P., Plemmons, R.J.: Algorithms and applications for approximate nonnegative matrix factorization. Comput. Stat. Data Anal. 52, 155–173 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  4. Boyd, S., Vandenberghe, L.: Convex Optimization. Cambridge University Press, Cambridge (2004)

    Book  MATH  Google Scholar 

  5. Brunet, J.P., Tamayo, P., Golub, T.R., Mesirov, J.P.: Metagenes and molecular pattern discovery using matrix factorization. Proc. Natl. Acad. Sci. USA 101(12), 4164–4169 (2004)

    Article  Google Scholar 

  6. Cai, D., He, X., Han, J., Huang, T.S.: Graph regularized nonnegative matrix factorization for data representation. IEEE Trans. Pattern Anal. Mach. Intell. 33(8), 1548–1560 (2011)

    Article  Google Scholar 

  7. Cichocki, A., Zdunek, R., Phan, A.H., Amari, S.I.: Nonnegative Matrix and Tensor Factorizations. Wiley, West Sussex (2009)

    Book  Google Scholar 

  8. Csiszár, I., Tusnády, G.: Information geometry and alternating minimization procedures. Statistics and Decisions, Supplemental Issue, 205–237 (1984)

  9. Févotte, C., Bertin, N., Durrieu, J.L.: Nonnegative matrix factorization with the Itakura-Saito divergence: with application to music analysis. Neural Comput. 21(3), 793–830 (2009)

    Article  MATH  Google Scholar 

  10. Finesso, L., Spreij, P.: Nonnegative matrix factorization and I-divergence alternating minimization. Linear Algebra Appl. 416, 270–287 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  11. Gillis, N.: Nonnegative matrix factorization: Complexity, algorithms and applications. Ph.D. thesis, Université Catholique de Louvain, Louvain-la-Neuve (2011)

  12. Gillis, N., Glineur, F.: Nonnegative factorization and the maximum edge biclique problem (2008). ArXiv e-prints

  13. Gillis, N., Glineur, F.: Accelerated multiplicative updates and hierarchical ALS algorithms for nonnegative matrix factorization. Neural Comput. 24(4), 1085–1105 (2012)

    Article  MathSciNet  Google Scholar 

  14. Hibi, R., Takahashi, N.: A modified multiplicative update algorithm for Euclidean distance-based nonnegative matrix factorization and its global convergence. In: Proceedings of 18th International Conference on Neural Information Processing, Part-II, pp. 655–662 (2011)

    Google Scholar 

  15. Holzapfel, A., Stylianou, Y.: Musical genre classification using nonnegative matrix factorization-based features. IEEE Trans. Audio Speech Lang. Process. 16(2), 424–434 (2008)

    Article  Google Scholar 

  16. Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401, 788–792 (1999)

    Article  Google Scholar 

  17. Lee, D.D., Seung, H.S.: Algorithms for non-negative matrix factorization. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 556–562 (2001)

    Google Scholar 

  18. Lin, C.J.: On the convergence of multiplicative update algorithms for nonnegative matrix factorization. IEEE Trans. Neural Netw. 18(6), 1589–1596 (2007)

    Article  Google Scholar 

  19. Lin, C.J.: Projected gradient methods for non-negative matrix factorization. Neural Comput. 19(10), 2756–2779 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  20. Lu, W., Sun, W., Lu, H.: Robust watermarking based on DWT and nonnegative matrix factorization. Comput. Electr. Eng. 35, 183–188 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  21. Sha, F., Min, Y., Saul, L.K., Lee, D.D.: Multiplicative updates for nonnegative quadratic programming. Neural Comput. 19, 2004–2031 (2007)

    Article  MATH  MathSciNet  Google Scholar 

  22. Shahnaz, F., Berry, M.W., Pauca, V.P., Plemmons, R.J.: Document clustering using nonnegative matrix factorization. Inf. Process. Manag. 42, 373–386 (2006)

    Article  MATH  Google Scholar 

  23. Takahashi, N., Nishi, T.: Global convergence of decomposition learning methods for support vector machines. IEEE Trans. Neural Netw. 17(6), 1362–1369 (2006)

    Article  Google Scholar 

  24. Vavasis, S.A.: On the complexity of nonnegative matrix factorization. SIAM J. Optim. 20(3), 1364–1377 (2009)

    Article  MATH  MathSciNet  Google Scholar 

  25. Wang, R.S., Zhang, S., Wang, Y., Zhang, X.S., Chen, L.: Clustering complex networks and biological networks by nonnegative matrix factorization with various similarity measures. Neurocomputing 72, 134–141 (2008)

    Article  Google Scholar 

  26. Wu, C.F.J.: On the convergence properties of the EM algorithm. Ann. Stat. 11(1), 95–103 (1983)

    Article  MATH  Google Scholar 

  27. Yang, Z., Oja, E.: Unified development of multiplicative algorithm for linear and quadratic nonnegative matrix factorization. IEEE Trans. Neural Netw. 22(12), 1878–1891 (2011)

    Article  Google Scholar 

  28. Zangwill, W.I.: Nonlinear Programming: a Unified Approach. Prentice Hall, Englewood Cliffs (1969)

    MATH  Google Scholar 

Download references

Acknowledgements

This work was partially supported by JSPS KAKENHI Grant Numbers 24560076 and 23310104, and by the project “R&D for cyber-attack predictions and rapid response technology by means of international cooperation” of the Ministry of Internal Affairs and Communications, Japan.

Author information

Author notes

  1. Norikazu Takahashi

    Present address: Graduate School of Natural Science and Technology, Okayama University, 3-1-1 Tsushima-naka, Kita-ku, Okayama, 700-8530, Japan

  2. Ryota Hibi

    Present address: Mitsubishi UFJ Morgan Stanley Securities Co., Ltd., Tokyo, Japan

Authors and Affiliations

  1. Department of Informatics, Kyushu University, 744 Motooka, Nishi-ku, Fukuoka, 819–0395, Japan

    Norikazu Takahashi & Ryota Hibi

  2. Institute of Systems, Information Technologies and Nanotechnologies, 2-1-22 Momochihama, Sawara-ku, Fukuoka, 814-0001, Japan

    Norikazu Takahashi

Authors
  1. Norikazu Takahashi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryota Hibi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Norikazu Takahashi.

Additional information

Part of this paper was presented at 18th International Conference on Neural Information Processing, Shanghai, China, in November 2011 [14].

Rights and permissions

Reprints and permissions

About this article

Cite this article

Takahashi, N., Hibi, R. Global convergence of modified multiplicative updates for nonnegative matrix factorization. Comput Optim Appl 57, 417–440 (2014). https://doi.org/10.1007/s10589-013-9593-0

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10589-013-9593-0

Keywords

Search

Navigation