Fujiao Ju
Abstract
Restricted Boltzmann machine (RBM) is a famous model for feature extraction and can be used as an initializer for neural networks. When applying the classic RBM to multidimensional data such as 2D/3D tensors, one needs to vectorize such as high-order data. Vectorizing will result in dimensional disaster and valuable spatial information loss. As RBM is a model with fully connected layers, it requires a large amount of memory. Therefore, it is difficult to use RBM with high-order data on low-end devices. In this article, to utilize classic RBM on tensorial data directly, we propose a new tensorial RBM model parameterized by the tensor train format (TTRBM). In this model, both visible and hidden variables are in tensorial form, which are connected by a parameter matrix in tensor train format. The biggest advantage of the proposed model is that TTRBM can obtain comparable performance compared with the classic RBM with much fewer model parameters and faster training process. To demonstrate the advantages of TTRBM, we conduct three real-world applications, face reconstruction, handwritten digit recognition, and image super-resolution in the experiments.
- N. Andrew. 2011. Sparse autoencoder. CS294A Lecture notes, 72.Google Scholar
- Y. Bengio. 2009. Learning deep architectures for AI. Foundations and Trends in Machine Learning 2, 1 (2009), 1--127.Google ScholarDigital Library
- C. M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer. Google ScholarDigital Library
- N. Cai, Z. Su, Z. Lin, H. Wang, Z. Yang, and B. Ling. 2017. Blind inpainting using the fully convolutional neural network. Visual Computer 33, 2 (2017), 249--261. Google ScholarDigital Library
- C. Chen, H. Tong, L. Xie, L. Ying, and Q. He. 2017. Cross-dependency inference in multi-layered networks: A collaborative filtering perspective. ACM Transactions on Knowledge Discovery from Data 2016 Special Issue 11, 4, Article 42 (2017), 1--26.Google Scholar
- S. He, H. Chen, Z. Zhu, D. G. Ward, H. J. Cooper, M. R. Viant, J. K. Heath, and X. Yao. 2015. Robust twin boosting for feature selection from high-dimensional omics data with label noise. Information Sciences 219, 10 (2015), 1--18. Google ScholarDigital Library
- G. E. Hinton. 2002. Training products of experts by minimizing contrastive divergence. Neural Computation 14, 8 (2002), 1771--1800. Google ScholarDigital Library
- G. E. Hinton, S. Osindero, and Y. W. Teh. 2006. A fast learning algorithm for deep belief nets. Neural Computation 18, 7 (2006), 1527--1554. Google ScholarDigital Library
- G. E. Hinton and R. Salakhutdinov. 2009. Replicated softmax: An undirected topic model. In Proceedings of the Advances in Neural Information Processing Systems. 1607--1614. Google ScholarDigital Library
- L. K. Kim. 2018. DeepX: Deep learning accelerator for restricted boltzmann machine artificial neural networks. IEEE Transactions on Neural Networks and Learning Systems 29, 5 (2018), 1441--1453.Google ScholarCross Ref
- T. G. Kolda and B. W. Bader. 2009. Tensor decompositions and applications. SIAM Review 51, 3 (2009), 455--500. Google ScholarDigital Library
- A. Krizhevsky and G. E. Hinton. 2010. Factored 3-way restricted Boltzmann machines for modeling natural images. In Proceedings of the International Conference on Artificial Intelligence and Statistics. 621--628.Google Scholar
- S. Lee, Y. Kim, J. Lee, K. Kim, K. Lee, and J. Noh. 2014. Depth manipulation using disparity histogram analysis for stereoscopic 3D. Visual Computer 30, 4 (2014), 455--465. Google ScholarDigital Library
- Q. Liu, C. Zhang, Q. Guo, H. Xu, and Y. Zhou. 2016. Adaptive sparse coding on PCA dictionary for image denoising. Visual Computer 32, 4 (2016), 535--549. Google ScholarDigital Library
- T. D. Nguyen, T. Tran, D. Phung, and S. Venkatesh. 2015. Tensor-variate restricted Boltzmann machines. In Proceedings of the Association for the Advancement of Artificial Intelligence. 2887--2893. Google ScholarDigital Library
- A. Novikov, D. Podoprikhin, A. Osokin, and D. P. Vetrov. 2015. Tensorizing neural networks. In Proceedings of the Advances in Neural Information Processing Systems. 442--450.Google Scholar
- I. V. Oseledets. 2011. Tensor-train decomposition. SIAM Journal on Scientific Computing 33, 5 (2011), 2295--2317. Google ScholarDigital Library
- E. Papalexakis, C. Faloutsos, and N. D. Sidiropoulos. 2015. ParCube: Sparse parallelizable CANDECOMP-PARAFAC tensor decomposition. ACM Transactions on Knowledge Discovery from Data 10, 1 (2015), 3:1--3:25.Google Scholar
- G. Qi, Y. Sun, J. Gao, Y. Hu, and J. Li. 2016. Matrix variate restricted Boltzmann machine. In Proceedings of the IEEE International Joint Conference on Neural Networks. 389--395.Google Scholar
- R. Socher, D. Chen, C. Manning, and N. Andrew. 2013. Reasoning with neural tensor networks for knowledge base completion. In Proceedings of the Advances in Neural Information Processing Systems. 926--934.Google Scholar
- S. Soundarajan and J. E. Hopcroft. 2015. Use of local group information to identify communities in networks. ACM Transactions on Knowledge Discovery from Data 9, 3 (2015), 1--27. Google ScholarDigital Library
- J. Tang, X. Shu, Z. Li, Y. Jiang, and Q. Tian. 2018. Social anchor-unit graph regularized tensor completion for large-scale image retagging. arxiv:1804.04397.Google Scholar
- J. Tang, X. Shu, G. Qi, Z. Li, M. Wang, S. Yan, and R. Jain. 2017. Tri-clustered tensor completion for social-aware image tag refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence 39, 8 (2017), 1662--1674.Google ScholarDigital Library
- T. Tieleman. 2008. Training restricted Boltzmann machines using approximations to the likelihood gradient. In Proceedings of the International Conference on Machine Learning. 1064--1071. Google ScholarDigital Library
- W. K. Wong, Z. Lai, Y. Xu, J. Wen, and C. P. Ho. 2015. Joint tensor feature analysis for visual object recognition. IEEE Transactions on Cybernetics 45, 11 (2015), 2425--2436.Google ScholarCross Ref
- Y. Wu, S. C. Hoi, T. Mei, and N. Yu. 2017. Large-scale online feature selection for ultra-high dimensional sparse data. ACM Transactions on Knowledge Discovery from Data 11, 4 (2017), 48:1--48:22. Google ScholarDigital Library
- J. Yang, J. Wright, T. S. Huang, and Y. Ma. 2010. Image super-resolution via sparse representation. IEEE Transactions on Image Processing 19, 11 (2010), 2861--2873. Google ScholarDigital Library
- M. Yang, P. Zhu, F. Liu, and L Shen. 2015. Joint representation and pattern learning for robust face recognition. Neurocomputing 168 (2015), 70--80. Google ScholarDigital Library
- H. Zhang, Y. Yang, H. Luan, S. Yang, and T. Chua. 2014. Start from scratch: Towards automatically identifying, modeling, and naming visual attributes. In Proceedings of the ACM International Conference on Multimedia. 187--196.Google Scholar
- J. Zhang, Y. Han, and J. Jiang. 2015. Tensor rank selection for multimedia analysis. Journal of Visual Communication and Image Representation 30, 7 (2015), 376--392.Google ScholarDigital Library
- Z. Zhu, S. Jia, S. He, Y. Sun, Z. Ji, and L. Shen. 2015. Three-dimensional Gabor feature extraction for hyperspectral imagery classification using a memetic framework. Information Sciences 298, 20 (2015), 274--287. Google ScholarDigital Library
Index Terms
- Tensorizing Restricted Boltzmann Machine
Recommendations
Tikhonov-Type regularization for restricted boltzmann machines
ICANN'12: Proceedings of the 22nd international conference on Artificial Neural Networks and Machine Learning - Volume Part IIn this paper, we study a Tikhonov-type regularization for restricted Boltzmann machines (RBM). We present two alternative formulations of the Tikhonov-type regularization which encourage an RBM to learn a smoother probability distribution. Both ...
Learning Informative Features from Restricted Boltzmann Machines
In recent years deep learning paradigm achieved important empirical success in a number of practical applications such as object recognition, speech recognition and natural language processing. A lot of effort has been put on understanding theoretical ...
An overview on Restricted Boltzmann Machines
The Restricted Boltzmann Machine (RBM) has aroused wide interest in machine learning fields during the past decade. This review aims to report the recent developments in theoretical research and applications of the RBM. We first give an overview of the ...
Comments