Skip to main content
SpringerLink
Log in

Seeking an appropriate alternative least squares algorithm for nonnegative tensor factorizations

A novel recursive solution for nonnegative quadratic programming and NTF

  • ICONIP2010
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Alternative least squares (ALS) algorithm is considered as a “work-horse” algorithm for general tensor factorizations. A common form of this algorithm for nonnegative tensor factorizations (NTF) is always combined with a nonlinear projection (rectifier) to enforce nonnegative entries during the estimation. Such simple modification often provides acceptable results for general data. However, this does not establish an appropriate ALS algorithm for NTF. This kind of ALS algorithm often converges slowly, or cannot converge to the desired solution, especially for collinear data. To this end, in this paper, we reinvestigate the nonnegative quadratic programming, propose a recursive method for solving this problem. Then, we formulate a novel ALS algorithm for NTF. The validity and high performance of the proposed algorithm has been confirmed for difficult benchmarks, and also in an application of object classification, and analysis of EEG signals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Abadir KM, Magnus JR (2005) Matrix algebra (econometric exercises). Cambridge University Press, URL http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/0521822890

  2. Albright R, Cox J, Duling D, Langville AN, Meyer CD (2006) Algorithms, initializations, and convergence for the nonnegative matrix factorization. Tech. rep., NCSU Technical Report Math 81706, URL http://www.wfu.edu/~plemmons/papers.htm

  3. Cichocki A, Phan AH (2009) Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE 92-A(3):708–721 (invited paper)

  4. Cichocki A, Zdunek R (2006) Multilayer nonnegative matrix factorization. Electron Lett 42(16):947–948

    Article  Google Scholar 

  5. Cichocki A, Zdunek R, Choi S, Plemmons R, Amari S (2007) Nonnegative tensor factorization using Alpha and Beta divergencies. In: Proceedings of IEEE international conference on acoustics, speech, and signal processing (ICASSP07), vol III, Honolulu, pp 1393–1396

  6. Cichocki A, Washizawa Y, Rutkowski T, Bakardjian H, Phan AH, Choi S, Lee H, Zhao Q, Zhang L, Li Y (2008) Noninvasive BCIs: multiway signal-processing array decompositions. Computer 41(10):34–42. doi:10.1109/MC.2008.431. http://www.ieeecomputersociety.org/

    Google Scholar 

  7. Cichocki A, Zdunek R, Phan AH, Amari S (2009) Nonnegative matrix and tensor factorizations: applications to exploratory multi-way data analysis and blind source separation. Wiley, Chichester

    Google Scholar 

  8. Gillis N, Glineur F (2008) Nonnegative factorization and the maximum edge biclique problem. CORE Discussion Papers 2008064, Universit catholique de Louvain, Center for Operations Research and Econometrics (CORE), URL http://econpapers.repec.org/RePEc:cor:louvco:2008064

  9. Hancewicz T, Wang JH (2005) Discriminant image resolution: a novel multivariate image analysis method utilizing a spatial classification constraint in addition to bilinear nonnegativity. Chemometr Intell Lab Syst 77:18–31

    Article  Google Scholar 

  10. Harshman R (1970) Foundations of the PARAFAC procedure: models and conditions for an explanatory multimodal factor analysis. UCLA Working Papers Phonetics 16:1–84

    Google Scholar 

  11. Higham DJ, Higham NJ (2005) MATLAB guide, 2nd edn. SIAM

  12. Higham NJ (2002) Accuracy and stability of numerical algorithms, 2nd edn. SIAM, Philadelphia, xxx+680 pp. ISBN: 0-89871-521-0

  13. Horn RA, Johnson CR (1990) Matrix analysis. Cambridge University Press, Cambridge

    MATH  Google Scholar 

  14. Kim YD, Choi S (2007) Nonnegative Tucker decomposition. In: Proceedings of conference on computer vision and pattern recognition (CVPR-2007), Minneapolis, Minnesota

  15. Kivinen J, Warmuth M (1997) Exponentiated gradient versus gradient descent for linear predictors. Inform Comput 132

  16. Kolda T, Bader B (2009) Tensor decompositions and applications. SIAM Rev 51(3):455–500

    Google Scholar 

  17. Langville AN, Meyer CD, Albright R (2006) Initializations for the nonnegative matrix factorization. In: Proceedings of the twelfth ACM SIGKDD international conference on knowledge discovery and data mining, Philadelphia

  18. Lee D, Seung H (1999) Learning of the parts of objects by non-negative matrix factorization. Nature 401:788–791

    Article  Google Scholar 

  19. Lee H, Kim YD, Cichocki A, Choi S (2007) Nonnegative tensor factorization for continuous EEG classification. Int J Neural Syst 17(4):1–13

    Article  Google Scholar 

  20. Lee H, Yoo J, Choi S (2010) Semi-supervised nonnegative matrix factorization. IEEE Signal Process Lett 18

  21. Lin CJ (2007) Projected gradient methods for non-negative matrix factorization. Neural Comput 19(10):2756–2779

    Article  MathSciNet  MATH  Google Scholar 

  22. Mørup M, Hansen L, Arnfred S (2006) ERPWAVELAB a toolbox for multi-channel analysis of time-frequency transformed event related potentials. J Neurosci Methods 161(2):361–368

    Article  Google Scholar 

  23. Mørup M, Hansen L, Parnas J, Arnfred S (2006) Decomposing the time-frequency representation of EEG using non-negative matrix and multi-way factorization. Tech. rep., URL http://www2.imm.dtu.dk/pubdb/p.php?4144

  24. Nion D, De Lathauwer L (2008) An enhanced line search scheme for complex-valued tensor decompositions. Application in DS-CDMA. Signal Process 88(3):749–755

    Article  MathSciNet  MATH  Google Scholar 

  25. NVIDIA (2008) NVIDIA CUDA Programming Guide 2.0. NVIDIA, Santa Clara

  26. Paatero P (1997) Least-squares formulation of robust nonnegative factor analysis. Chemometr Intell Lab Syst 37:23–35

    Article  Google Scholar 

  27. Paatero P (1997) A weighted non-negative least squares algorithm for three-way PARAFAC factor analysis. Chemometr Intell Lab Syst 38(2):223–242

    Article  Google Scholar 

  28. Phan AH, Cichocki A (2008) Multi-way nonnegative tensor factorization using fast hierarchical alternating least squares algorithm (HALS). In: Proceedings of The 2008 international symposium on nonlinear theory and its applications, Budapest

  29. Phan AH, Cichocki A (2011) PARAFAC algorithms for large-scale problems. Adaptive incremental learning in neural networks; learning algorithm and mathematic modelling selected papers from the international conference on neural information processing - ICONIP 2009. Neurocomputing 74(11):1970–1984. doi:10.1016/j.neucom.2010.06.030. http://www.sciencedirect.com/science/article/pii/S0925231211000415

  30. Rajih M, Comon P, Harshman RA (2008) Enhanced line search: a novel method to accelerate PARAFAC. SIAM J Matrix Anal Appl 30(3):1128–1147

    Article  MathSciNet  Google Scholar 

  31. Sha F, Saul LK, Lee DD (2002) Multiplicative updates for nonnegative quadratic programming in support vector machines. In: Advances in neural information processing systems 15. MIT Press, pp 1041–1048

  32. Sha F, Lin Y, Saul LK, Lee DD (2007) Multiplicative updates for nonnegative quadratic programming. Neural Comput 19

  33. Shashua A, Hazan T (2005) Non-negative tensor factorization with applications to statistics and computer vision. In: Proceedings of the 22-th international conference on machine learning, Bonn

  34. Wang J, Hopke P, Hancewicz T, Zhang SL (2003) Application of modified alternating least squares regression to spectroscopic image analysis. Analytica Chimica Acta 476:93–109

    Article  Google Scholar 

  35. Zdunek R, Cichocki A (2007) Nonnegative matrix factorization with constrained second-order optimization. Signal Process 87:1904–1916

    Article  MATH  Google Scholar 

  36. Zdunek R, Cichocki A (2008) Nonnegative matrix factorization with quadratic programming. Neurocomputing 71(10–12):2309–2320

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Lab for Advanced Brain Signal Processing, BSI, RIKEN, Wako, Japan

    Anh Huy Phan & Andrzej Cichocki

Authors
  1. Anh Huy Phan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrzej Cichocki
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anh Huy Phan.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Phan, A.H., Cichocki, A. Seeking an appropriate alternative least squares algorithm for nonnegative tensor factorizations. Neural Comput & Applic 21, 623–637 (2012). https://doi.org/10.1007/s00521-011-0652-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-011-0652-0

Keywords

Search

Navigation