research-article

Tensorizing Restricted Boltzmann Machine

Authors:

Michael Antolovich,

Baocai YinAuthors Info & Claims

ACM Transactions on Knowledge Discovery from Data (TKDD), Volume 13, Issue 3

Article No.: 30, Pages 1 - 16

https://doi.org/10.1145/3321517

Published: 07 June 2019 Publication History

Abstract

Restricted Boltzmann machine (RBM) is a famous model for feature extraction and can be used as an initializer for neural networks. When applying the classic RBM to multidimensional data such as 2D/3D tensors, one needs to vectorize such as high-order data. Vectorizing will result in dimensional disaster and valuable spatial information loss. As RBM is a model with fully connected layers, it requires a large amount of memory. Therefore, it is difficult to use RBM with high-order data on low-end devices. In this article, to utilize classic RBM on tensorial data directly, we propose a new tensorial RBM model parameterized by the tensor train format (TTRBM). In this model, both visible and hidden variables are in tensorial form, which are connected by a parameter matrix in tensor train format. The biggest advantage of the proposed model is that TTRBM can obtain comparable performance compared with the classic RBM with much fewer model parameters and faster training process. To demonstrate the advantages of TTRBM, we conduct three real-world applications, face reconstruction, handwritten digit recognition, and image super-resolution in the experiments.

References

[1]

N. Andrew. 2011. Sparse autoencoder. CS294A Lecture notes, 72.

[2]

Y. Bengio. 2009. Learning deep architectures for AI. Foundations and Trends in Machine Learning 2, 1 (2009), 1--127.

Digital Library

[3]

C. M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer.

Digital Library

[4]

N. Cai, Z. Su, Z. Lin, H. Wang, Z. Yang, and B. Ling. 2017. Blind inpainting using the fully convolutional neural network. Visual Computer 33, 2 (2017), 249--261.

Digital Library

[5]

C. Chen, H. Tong, L. Xie, L. Ying, and Q. He. 2017. Cross-dependency inference in multi-layered networks: A collaborative filtering perspective. ACM Transactions on Knowledge Discovery from Data 2016 Special Issue 11, 4, Article 42 (2017), 1--26.

[6]

S. He, H. Chen, Z. Zhu, D. G. Ward, H. J. Cooper, M. R. Viant, J. K. Heath, and X. Yao. 2015. Robust twin boosting for feature selection from high-dimensional omics data with label noise. Information Sciences 219, 10 (2015), 1--18.

Digital Library

[7]

G. E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural Computation 14, 8 (2002), 1771--1800.

Digital Library

[8]

G. E. Hinton, S. Osindero, and Y. W. Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 1527--1554.

Digital Library

[9]

G. E. Hinton and R. Salakhutdinov. 2009. Replicated softmax: An undirected topic model. In Proceedings of the Advances in Neural Information Processing Systems. 1607--1614.

Digital Library

[10]

L. K. Kim. 2018. DeepX: Deep learning accelerator for restricted boltzmann machine artificial neural networks. IEEE Transactions on Neural Networks and Learning Systems 29, 5 (2018), 1441--1453.

[11]

T. G. Kolda and B. W. Bader. 2009. Tensor decompositions and applications. SIAM Review 51, 3 (2009), 455--500.

Digital Library

[12]

A. Krizhevsky and G. E. Hinton. 2010. Factored 3-way restricted Boltzmann machines for modeling natural images. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 621--628.

[13]

S. Lee, Y. Kim, J. Lee, K. Kim, K. Lee, and J. Noh. 2014. Depth manipulation using disparity histogram analysis for stereoscopic 3D. Visual Computer 30, 4 (2014), 455--465.

Digital Library

[14]

Q. Liu, C. Zhang, Q. Guo, H. Xu, and Y. Zhou. 2016. Adaptive sparse coding on PCA dictionary for image denoising. Visual Computer 32, 4 (2016), 535--549.

Digital Library

[15]

T. D. Nguyen, T. Tran, D. Phung, and S. Venkatesh. 2015. Tensor-variate restricted Boltzmann machines. In Proceedings of the Association for the Advancement of Artificial Intelligence. 2887--2893.

Digital Library

[16]

A. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov. 2015. Tensorizing neural networks. In Proceedings of the Advances in Neural Information Processing Systems. 442--450.

[17]

I. V. Oseledets. 2011. Tensor-train decomposition. SIAM Journal on Scientific Computing 33, 5 (2011), 2295--2317.

Digital Library

[18]

E. Papalexakis, C. Faloutsos, and N. D. Sidiropoulos. 2015. ParCube: Sparse parallelizable CANDECOMP-PARAFAC tensor decomposition. ACM Transactions on Knowledge Discovery from Data 10, 1 (2015), 3:1--3:25.

[19]

G. Qi, Y. Sun, J. Gao, Y. Hu, and J. Li. 2016. Matrix variate restricted Boltzmann machine. In Proceedings of the IEEE International Joint Conference on Neural Networks. 389--395.

[20]

R. Socher, D. Chen, C. Manning, and N. Andrew. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of the Advances in Neural Information Processing Systems. 926--934.

[21]

S. Soundarajan and J. E. Hopcroft. 2015. Use of local group information to identify communities in networks. ACM Transactions on Knowledge Discovery from Data 9, 3 (2015), 1--27.

Digital Library

[22]

J. Tang, X. Shu, Z. Li, Y. Jiang, and Q. Tian. 2018. Social anchor-unit graph regularized tensor completion for large-scale image retagging. arxiv:1804.04397.

[23]

J. Tang, X. Shu, G. Qi, Z. Li, M. Wang, S. Yan, and R. Jain. 2017. Tri-clustered tensor completion for social-aware image tag refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 8 (2017), 1662--1674.

Digital Library

[24]

T. Tieleman. 2008. Training restricted Boltzmann machines using approximations to the likelihood gradient. In Proceedings of the International Conference on Machine Learning. 1064--1071.

Digital Library

[25]

W. K. Wong, Z. Lai, Y. Xu, J. Wen, and C. P. Ho. 2015. Joint tensor feature analysis for visual object recognition. IEEE Transactions on Cybernetics 45, 11 (2015), 2425--2436.

[26]

Y. Wu, S. C. Hoi, T. Mei, and N. Yu. 2017. Large-scale online feature selection for ultra-high dimensional sparse data. ACM Transactions on Knowledge Discovery from Data 11, 4 (2017), 48:1--48:22.

Digital Library

[27]

J. Yang, J. Wright, T. S. Huang, and Y. Ma. 2010. Image super-resolution via sparse representation. IEEE Transactions on Image Processing 19, 11 (2010), 2861--2873.

Digital Library

[28]

M. Yang, P. Zhu, F. Liu, and L Shen. 2015. Joint representation and pattern learning for robust face recognition. Neurocomputing 168 (2015), 70--80.

Digital Library

[29]

H. Zhang, Y. Yang, H. Luan, S. Yang, and T. Chua. 2014. Start from scratch: Towards automatically identifying, modeling, and naming visual attributes. In Proceedings of the ACM International Conference on Multimedia. 187--196.

[30]

J. Zhang, Y. Han, and J. Jiang. 2015. Tensor rank selection for multimedia analysis. Journal of Visual Communication and Image Representation 30, 7 (2015), 376--392.

Digital Library

[31]

Z. Zhu, S. Jia, S. He, Y. Sun, Z. Ji, and L. Shen. 2015. Three-dimensional Gabor feature extraction for hyperspectral imagery classification using a memetic framework. Information Sciences 298, 20 (2015), 274--287.

Digital Library

Cited By

Li HLi ZLi KRellermeyer JChen LLi K(2021)SGD_Tucker: A Novel Stochastic Optimization Strategy for Scalable Parallel Sparse Tucker DecompositionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.3047460(1-1)Online publication date: 2021
https://doi.org/10.1109/TPDS.2020.3047460
Zhang SYang LFeng JWei WCui ZXie XYan P(2021)A tensor-network-based big data fusion framework for Cyber–Physical–Social Systems (CPSS)Information Fusion10.1016/j.inffus.2021.05.01476:C(337-354)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1016/j.inffus.2021.05.014
Gao JLi PChen ZZhang J(2020)A Survey on Deep Learning for Multimodal Data FusionNeural Computation10.1162/neco_a_0127332:5(829-864)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1162/neco_a_01273

Index Terms

Tensorizing Restricted Boltzmann Machine
1. Networks
  1. Network architectures
    1. Network design principles
      1. Layering

Recommendations

Tikhonov-Type regularization for restricted boltzmann machines
ICANN'12: Proceedings of the 22nd international conference on Artificial Neural Networks and Machine Learning - Volume Part I

In this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both ...
Learning Informative Features from Restricted Boltzmann Machines

In recent years deep learning paradigm achieved important empirical success in a number of practical applications such as object recognition, speech recognition and natural language processing. A lot of effort has been put on understanding theoretical ...
An overview on Restricted Boltzmann Machines

The Restricted Boltzmann Machine (RBM) has aroused wide interest in machine learning fields during the past decade. This review aims to report the recent developments in theoretical research and applications of the RBM. We first give an overview of the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Knowledge Discovery from Data

ACM Transactions on Knowledge Discovery from Data Volume 13, Issue 3

June 2019

261 pages

ISSN:1556-4681

EISSN:1556-472X

DOI:10.1145/3331063

Editors:
Charu Aggarwal
IBM T. J. Watson Research, USA
,
Xindong Wu
University of Louisiana at Lafayette, USA

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 June 2019

Accepted: 01 March 2019

Revised: 01 January 2019

Received: 01 October 2017

Published in TKDD Volume 13, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

University of Sydney Business School ARC Bridging Fund (2017)
Beijing Natural Science Foundation
National Natural Science Foundation of China
Beijing Postdoctoral Research Foundation, China Postdoctoral Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
296
Total Downloads

Downloads (Last 12 months)18
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li HLi ZLi KRellermeyer JChen LLi K(2021)SGD_Tucker: A Novel Stochastic Optimization Strategy for Scalable Parallel Sparse Tucker DecompositionIEEE Transactions on Parallel and Distributed Systems10.1109/TPDS.2020.3047460(1-1)Online publication date: 2021
https://doi.org/10.1109/TPDS.2020.3047460
Zhang SYang LFeng JWei WCui ZXie XYan P(2021)A tensor-network-based big data fusion framework for Cyber–Physical–Social Systems (CPSS)Information Fusion10.1016/j.inffus.2021.05.01476:C(337-354)Online publication date: 1-Dec-2021
https://dl.acm.org/doi/10.1016/j.inffus.2021.05.014
Gao JLi PChen ZZhang J(2020)A Survey on Deep Learning for Multimodal Data FusionNeural Computation10.1162/neco_a_0127332:5(829-864)Online publication date: 1-May-2020
https://dl.acm.org/doi/10.1162/neco_a_01273

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Issue’s Table of Contents