Skip to main content
SpringerLink
Log in

On Chow-Liu Forest Based Regularization of Deep Belief Networks

  • Conference paper
  • First Online:
Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions (ICANN 2019)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 11731))

Included in the following conference series:

Abstract

In this paper we introduce a methodology for the simple integration of almost-independence information on the visible (input) variables of the restricted Boltzmann machines (RBM) into the weight decay regularization of the contrastive divergence and stochastic gradient descent algorithm. After identifying almost independent clusters of the input coordinates by Chow-Liu tree and forest estimation, the RBM regularization strategy is constructed. We show an example of a sparse two hidden layer Deep Belief Net (DBN) applied on the MNIST data classification problem. The performance is quantified by estimating misclassification rate and measure of manifold disentanglement. Approach is benchmarked to the full model.

This work was developed in Fraunhofer Cluster of Excellence “Cognitive Internet Technologies”.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bach, F.R., Jordan, M.I.: Beyond independent components: trees and clusters. J. Mach. Learn. Res. 4, 1205–1233 (2003)

    MathSciNet  MATH  Google Scholar 

  2. Brahma, P.P., Wu, D., She, Y.: Why deep learning works: a manifold disentanglement perspective. IEEE Trans. Neural Netw. Learn. Syst. 27(10), 1997–2007 (2016). https://doi.org/10.1109/TNNLS.2015.2496947

    Article  MathSciNet  Google Scholar 

  3. Chow, C., Liu, C.: Approximating discrete probability distributions with dependence trees. IEEE Trans. Inf. Theory 14, 462–467 (1968)

    Article  Google Scholar 

  4. Cover, T.M., Thomas, J.A.: Elements of Information Theory. Wiley Interscience (2006). https://doi.org/10.1002/047174882X

    Book  Google Scholar 

  5. Dasgupta, S.: The Sample Complexity of Learning Fixed-Structure Bayesian Networks. Kluwer Academic Publishers, Boston (1999). https://doi.org/10.1023/A:1007417612269

    Article  Google Scholar 

  6. Fischer, A., Igel, C.: An introduction to restricted Boltzmann machines. In: Alvarez, L., Mejail, M., Gomez, L., Jacobo, J. (eds.) CIARP 2012. LNCS, vol. 7441, pp. 14–36. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-33275-3_2

    Chapter  Google Scholar 

  7. Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8) (2002). https://doi.org/10.1162/089976602760128018

    Article  Google Scholar 

  8. Hinton, G.E.: A practical guide to training restricted Boltzmann machines. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 599–619. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_32

    Chapter  Google Scholar 

  9. Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 27–54 (2006). https://doi.org/10.1162/neco.2006.18.7.1527

    Article  MathSciNet  MATH  Google Scholar 

  10. Koller, D., Friedman, N.: Probabilistic Graphical Models, Principles and Techniques. MIT Press, Cambridge (2009). https://doi.org/10.1017/S0269888910000275

    Article  Google Scholar 

  11. Koller, D., Friedman, N., Getoor, L., Taskar, B.: Graphical Models in a Nutshell. MIT Press, Cambridge (2007). https://doi.org/10.1.1.146.2935

    Google Scholar 

  12. Meek, C.: Finding a path is harder than finding a tree. J. Artif. Intell. Res. 15 (2001). https://doi.org/10.1613/jair.914

    Article  MathSciNet  Google Scholar 

  13. Pinsker, M.S.: On estimation of information via variation. Probl. Inf. Transm. 41(2), 71–75 (2005). https://doi.org/10.1007/s11122-005-0012-8

    Article  MathSciNet  MATH  Google Scholar 

  14. Salakhutdinov, R., Mnih, A., Hinton, G.E.: Restricted Boltzmann machines for collaborative filtering. In: International Conference on Machine Learning 24 (2007). https://doi.org/10.1145/1273496.1273596

  15. Tan, V.Y.F., Anandkumar, A., Willsky, A.: Learning high-dimensional markov forest distributions: analysis of error rates. J. Mach. Learn. Res. 12, 1617–1653 (2011)

    MathSciNet  MATH  Google Scholar 

  16. West, D.B.: Introduction to Graph Theory, 2nd edn. Prentice Hall, Upper Saddle River (2001)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Fraunhofer ITWM, Kaiserslautern, Germany

    Alex Sarishvili & Andreas Wirsen

  2. Fraunhofer-Chalmers Centre, Göteborg, Sweden

    Mats Jirstrand

  3. Fraunhofer Center for Machine Learning, Munich, Germany

    Alex Sarishvili, Andreas Wirsen & Mats Jirstrand

Authors
  1. Alex Sarishvili
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andreas Wirsen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mats Jirstrand
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alex Sarishvili .

Editor information

Editors and Affiliations

  1. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Igor V. Tetko

  2. Institute of Computer Science, Czech Academy of Sciences, Prague 8, Czech Republic

    Věra Kůrková

  3. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Pavel Karpov

  4. Helmholtz Zentrum München - Deutsches Forschungszentrum für Gesundheit und Umwelt (GmbH), Neuherberg, Germany

    Fabian Theis

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Sarishvili, A., Wirsen, A., Jirstrand, M. (2019). On Chow-Liu Forest Based Regularization of Deep Belief Networks. In: Tetko, I., Kůrková, V., Karpov, P., Theis, F. (eds) Artificial Neural Networks and Machine Learning – ICANN 2019: Workshop and Special Sessions. ICANN 2019. Lecture Notes in Computer Science(), vol 11731. Springer, Cham. https://doi.org/10.1007/978-3-030-30493-5_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-30493-5_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-30492-8

  • Online ISBN: 978-3-030-30493-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation