Skip to main content

Advertisement

SpringerLink
Log in

NeuralCP: Bayesian Multiway Data Analysis with Neural Tensor Decomposition

  • Published:
Cognitive Computation Aims and scope Submit manuscript

Abstract

Multiway data are widely observed in neuroscience, health informatics, food science, etc. Tensor decomposition is an important technique for capturing high-order interactions among such multiway data. Classical tensor decomposition methods, such as the Tucker decomposition and the CANDECOMP/PARAFAC (CP), assume that the complex interactions among objects are multi-linear and thus insufficient to represent nonlinear relationships in data. To effectively model the complex nonlinear relationships of a tensor, we design a neural model joining neural networks with the Bayesian tensor decomposition, in which the high-order interactions are captured by neural networks. By taking advantages of the nonlinear modeling provided by the neural networks and the uncertainty modeling provided by Bayesian models, we replace the multi-linear product in traditional Bayesian tensor decomposition with a more flexible neural function (i.e., a multi-layer perceptron) whose parameters can be learned from data. Our model can be efficiently optimized with stochastic gradient descent. Accordingly, it is scalable to large real-world tensor. We conducted experiments on both synthetic data and real-world chemometrics tensor data. Experimental results have demonstrated that the proposed model can achieve significantly higher prediction performance than the state-of-the-art tensor decomposition approaches. The proposed nonlinear tensor decomposition method, i.e., NeuralCP, has been demonstrated to obtain promising prediction results on many multi-way data.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. In this paper, \([N]\) denotes the set \(\{1,2,3,...,N\}\), where N is a positive integer.

  2. Theoretically, the weak upper bound of a three-order tensor \(\mathcal {A}\in \mathbb {R}^{I\times J\times K}\) is \(\min \{IJ,JK,KI\}\) [19]. For our datasets, the upper bounds are relatively high. Therefore, we just empirically search the rank from 2 to 20.

  3. The height of the box is the likely range of RMSE variation (distance between the RMSE of the first quartile and the third quartile). The blue line in the box is the median value of RMSE. And the two black bars on the top and bottom of the box represent the maximum and minimum values.

References

  1. Beckmann CF, Smith SM. Tensorial extensions of independent component analysis for multisubject fMRI analysis. Neuroimage 2005;25(1):294–311.

    Article  CAS  Google Scholar 

  2. Blei DM, Kucukelbir A, Mcauliffe JD. Variational inference: a review for statisticians. J Am Stat Assoc 2017;112(518):859–77.

    Article  CAS  Google Scholar 

  3. Bro R. PARAFAC tutorial and applications. Chemom Intell Lab Syst 1997;38(2):149–71.

    Article  CAS  Google Scholar 

  4. Bro R. Exploratory study of sugar production using fluorescence spectroscopy and multi-way analysis. Chemom Intell Lab Syst 1999;46(2):133–47.

    Article  CAS  Google Scholar 

  5. Carroll JD, Chang JJ. Analysis of individual differences in multidimensional scaling via an n-way generalization of “eckart-young” decomposition. Psychometrika 1970;35(3):283–319.

    Article  Google Scholar 

  6. Chen S, Lyu MR, King I, Xu Z. 2014. Exact and stable recovery of pairwise interaction tensors. Adv Neural Inf Process Syst, 1691–99.

  7. Chi EC, Kolda TG. On tensors, sparsity, and nonnegative factorizations. SIAM J Matrix Anal Appl 2012; 33(4):1272–99.

    Article  Google Scholar 

  8. Chu W, Ghahramani Z. Probabilistic models for incomplete multi-dimensional arrays. In: AISTATS, p. 89–96. 2009.

  9. Cohen N, Sharir O, Shashua A. On the expressive power of deep learning: a tensor analysis. In: Conference on learning theory. 2016.

  10. Comon P, Luciani X, De Almeida AL. Tensor decompositions, alternating least squares and other tales. J Chemom 2009;23 (7-8):393–405.

    Article  CAS  Google Scholar 

  11. De Lathauwer L, De Moor B, Vandewalle J. A multilinear singular value decomposition. SIAM J Matrix Anal Appl 2000;21(4):1253–78.

    Article  Google Scholar 

  12. De Vos M, Vergult A, De Lathauwer L, De Clercq W, Van Huffel S, Dupont P, Palmini A, Van Paesschen W. Canonical decomposition of ictal scalp eeg reliably detects the seizure onset zone. Neuroimage 2007; 37(3):844–54.

    Article  CAS  Google Scholar 

  13. Dunson DB, Xing C. Nonparametric bayes modeling of multivariate categorical data. J Am Stat Assoc 2009; 104(487):1042–51.

    Article  CAS  Google Scholar 

  14. Harshman RA. 1970. Foundations of the parafac procedure. Models and conditions for an “explanatory” multi-modal factor analysis.

  15. Kapteyn A, Neudecker H, Wansbeek T. An approach ton-mode components analysis. Psychometrika 1986; 51(2):269–75.

    Article  Google Scholar 

  16. Kingma D, Ba J. 2015. Adam: a method for stochastic optimization. In: International conference on learning representation.

  17. Kingma DP, Welling M. Auto-encoding variational Bayes. arXiv:1312.6114. 2013.

  18. Kingma DP, Welling M. 2014. Stochastic gradient vb and the variational auto-encoder. In: International conference on learning representation.

  19. Kolda TG, Bader BW. Tensor decompositions and applications. SIAM Rev 2009;51(3):455–500.

    Article  Google Scholar 

  20. Kruskal JB. Three-way arrays: rank and uniqueness of trilinear decompositions, with application to arithmetic complexity and statistics. Linear Algebra Appl 1977;18(2):95–138.

    Article  Google Scholar 

  21. Lawrence ND. 2006. Gaussian process latent variable model. Technical Report CS-06-03.

  22. Li G, Xu Z, Wang L, Ye J, King I, Lyu M. Simple and efficient parallelization for probabilistic temporal tensor factorization. In: 2017 International joint conference on neural networks (IJCNN). IEEE; 2017. p. 1–8.

  23. Li G, Ye J, Yang H, Chen D, Yan S, Xu Z. 2017. BT-nets: simplifying deep neural networks via block term decomposition. ArXiv e-prints.

  24. Maehara T, Hayashi K, Kawarabayashi K. 2016. Expected tensor decomposition with stochastic gradient descent. In: Thirtieth AAAI conference on artificial intelligence.

  25. Mørup M, Hansen LK. Automatic relevance determination for multi-way models. J Chemometr 2009;23 (7-8):352–63.

    Article  Google Scholar 

  26. Rai P, Hu C, Harding M, Carin L. Scalable probabilistic tensor factorization for binary and count data. In: Proceedings of the 24th international conference on artificial intelligence. AAAI Press; 2015. p. 3770–76.

  27. Rai P, Wang Y, Carin L. 2015. Leveraging features and networks for probabilistic tensor decomposition. In: Twenty-Ninth AAAI conference on artificial intelligence. 2015. p. 2942–48.

  28. Rai P, Wang Y, Guo S, Chen G, Dunson DB, Carin L. Scalable bayesian low-rank decomposition of incomplete multiway tensors. In: ICML. 2014. p. 1800–08.

  29. Shashua A, Hazan T. Non-negative tensor factorization with applications to statistics and computer vision. In: Proceedings of the 22nd international conference on machine learning. ACM; 2005. p. 792–99.

  30. Socher R, Chen D, Manning CD, Ng AY. Reasoning with neural tensor networks for knowledge base completion. In: NIPS. 2013; p. 926–34.

  31. Sun R. Moral judgment, human motivation, and neural networks. Cogn Comput 2013;5(4):566–579. https://doi.org/10.1007/s12559-012-9181-0 https://doi.org/10.1007/s12559-012-9181-0.

    Article  Google Scholar 

  32. Tucker LR. Some mathematical notes on three-mode factor analysis. Psychometrika 1966;31(3):279–311.

    Article  CAS  Google Scholar 

  33. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E. Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cogn Comput 2017;9(5):597–610. https://doi.org/10.1007/s12559-017-9472-6.

    Article  Google Scholar 

  34. Xiong L, Chen X, Huang TK, Schneider JG, Carbonell JG. Temporal collaborative filtering with bayesian probabilistic tensor factorization. In: Siam international conference on data mining, SDM 2010, April 29 - May 1, 2010, Columbus, Ohio, Usa, p. 211–22.

  35. Xu Z, Yan F, Qi Y. Bayesian nonparametric models for multiway data analysis. IEEE Trans Pattern Anal Mach Intell 2015;37(2):475–487.

    Article  Google Scholar 

  36. Xu Z, Yan F, Qi YA. Infinite tucker decomposition: nonparametric bayesian models for multiway data analysis. In: Proceedings of the 29th international conference on machine learning, ICML 2012, Edinburgh, Scotland, UK, June 26 - July 1, 2012.

  37. Ye J, Wang L, Li G, Chen D, Zhe S, Chu X, Xu Z. 2018. Learning compact recurrent neural networks with block-term tensor decomposition. In: Proceedings of 2018 IEEE international conference on computer vision and pattern recognition.

  38. Yoshii K, Tomioka R, Mochihashi D, Goto M. 2013. Infinite positive semidefinite tensor factorization for source separation of mixture signals. In: ICML (3), p. 576–84.

  39. Zhang X, Song S, Wu C. Robust bayesian classification with incomplete data. Cogn Comput 2013;5(2): 170–187. https://doi.org/10.1007/s12559-012-9188-6 https://doi.org/10.1007/s12559-012-9188-6.

    Article  Google Scholar 

  40. Zhao Q, Zhang L, Cichocki A. Bayesian CP factorization of incomplete tensors with automatic rank determination. IEEE Trans Pattern Anal Mach Intell 2015;37(9):1751–1763.

    Article  Google Scholar 

  41. Zhe S, Qi Y, Park Y, Xu Z, Molloy I, Chari S. Dintucker: scaling up gaussian process models on large multidimensional arrays. In: Thirtieth AAAI conference on artificial intelligence. 2016. p. 2386–92.

  42. Zhe S, Xu Z, Chu X, Qi YA, Park Y. 2015. Scalable nonparametric multiway data analysis. In: AISTATS.

  43. Zhe S, Zhang K, Wang P, Lee KC, Xu Z, Qi Y, Ghahramani Z. Distributed flexible nonlinear tensor factorization. In: Advances in neural information processing systems. 2016. p. 928–36.

Download references

Funding

Bin Liu, Lirong He, and Zenglin Xu were supported by the Natural Science Foundation of China (61572111, G05QNQR004), a 985 Project of UESTC (No.A1098531023601041) and a Fundamental Research Fund for the Central Universities of China (No. A03017023701012). Yingming Li was supported by Natural Science Foundation of China (No. 61702448).

Author information

Authors and Affiliations

  1. Center of Statistical Research Southwestern University of Finance and Economics, Chengdu, Sichuan, China

    Bin Liu

  2. SMILE Lab, School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, Sichuan, China

    Lirong He & Zenglin Xu

  3. College of Information Science and Electronic Engineering, Zhejiang University, Hangzhou, Zhejiang, China

    Yingming Li

  4. School of Computing, University of Utah, Salt Lake City, UT, USA

    Shandian Zhe

Authors
  1. Bin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lirong He
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yingming Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shandian Zhe
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zenglin Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zenglin Xu.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Informed Consent

Informed consent was not required as no humans or animals were involved.

Human and Animal Rights

This article does not contain any studies with human participants or animals performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liu, B., He, L., Li, Y. et al. NeuralCP: Bayesian Multiway Data Analysis with Neural Tensor Decomposition. Cogn Comput 10, 1051–1061 (2018). https://doi.org/10.1007/s12559-018-9587-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12559-018-9587-4

Keywords

Search

Navigation