On the Quantitative Analysis of Sparse RBMs

Abstract

With the development of deep neural networks, the model of restricted Boltzmann machine(RBM) has gradually become one of the essential aspects in deep learning researches. Because of the presence of the partition function, it is intractable to get the model selection, control the complexity, and learn an exact maximum likelihood in RBM model. A kind of effective measure is approximate inference that adopts annealing importance sampling(AIS) scheme only to evaluate a RBM’s performance. At present, there is little quantitative analysis on discrepancies generated by different RBM models. So we focus on the innovation research on some quantitative evaluation of the generalization performance of all kinds of sparse RBM models, including the classical sparse RBM(SpRBM) and the log sum sparse RBM(LogSumRBM). We discuss the influence and efficiency of the AIS strategy for these sparse RBMs’ estimations. Particularly, we confirm that the LogSumRBM is the optimal model in RBM and sparse RBMs for its smaller deviations in the assessment results regardless of the training MNIST data and the test, which provides a guarantee on some theories and experience in the choice of the deep learning models in the future.