Skip to main content
SpringerLink
Log in

Sparse Restricted Boltzmann Machine Based on Multiobjective Optimization

  • Conference paper
  • First Online:
Simulated Evolution and Learning (SEAL 2017)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 10593))

Included in the following conference series:

Abstract

This article proposes an efficient method for Restricted Boltzmann Machine (RBM) to learn sparse feature. Deep learning algorithms are used more and more often. The Deep Belief Network (DBN) model, which is composed of RBM, is considered as one of the most effective deep learning algorithms. RBM or auto-encoder (AE) is the basic model to build deep networks. However, RBM may produce redundant features without any constraints, then much improved RBM were proposed by added a regularization term to control sparsity of hidden units. Most of the proposed algorithms need a parameter to control the sparseness of the code. In this paper, we proposed a multiobjective optimization model to avoid user-defined constant that is a trade-off between the regularization term and the reconstruction error based on SR-RBM. We employ evolutionary algorithm to optimize the distortion function and the sparsity of hidden units simultaneously. Experimental results show that our novel approach can learn useful sparse feature without a user-define constant and it performs better than other feature learning models.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Felleman, D.J.: Distributed hierarchical processing in the primate cerebral cortex. Cereb. Cortex 1(1), 1–47 (1991)

    Article  Google Scholar 

  2. Lee, T.S.: Hierarchical Bayesian inference in the visual cortex. Opt. Soc. Am. A, Opt. Image Sci. Vis. 20(7), 1434–1448 (2003)

    Article  Google Scholar 

  3. Morris, G.: Anatomical funneling, sparse connectivity and redundancy reduction in the neural networks of the basal ganglia. Physiol.-Paris 97(4–6), 581–589 (2003)

    Article  Google Scholar 

  4. Kruger, N.: Deep hierarchies in the primate visual cortex: What can we learn for computer vision? IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1847–1871 (2013)

    Article  Google Scholar 

  5. Arel, I.: Deep machine learning—A new frontier in artificial intelligence research [research frontier]. IEEE Comput. Intell. Mag. 5(4), 13–18 (2010)

    Article  Google Scholar 

  6. Hou, W.: Blind image quality assessment via deep learning. IEEE Trans. Neural Netw. Learn. Syst. 26(6), 1275–1286 (2015)

    Article  MathSciNet  Google Scholar 

  7. Silver, D.: Mastering the game of Go with deep neural networks and tree search. Nature 529(7587), 484–489 (2015)

    Article  Google Scholar 

  8. Xu, Y.: A regression approach to speech enhancement based on deep neural networks. IEEE/ACM Trans. Audio Speech Lang. Process. 23(1), 7–19 (2015)

    Article  MathSciNet  Google Scholar 

  9. Krizhevsky, A., Sutskever, I.: ImageNet classification with deep convolutional neural networks. In: International Conference on Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates Inc. (2012)

    Google Scholar 

  10. Karpathy, A., Toderici, G.: Large-scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1725–1732. IEEE (2014)

    Google Scholar 

  11. Sainath, T.N.: Deep convolutional neural networks for large-scale speech tasks. Neural Netw. 64, 39–48 (2015)

    Article  Google Scholar 

  12. Hinton, G.E., Sejnowski, T.J.: Learning and relearning in Boltzmann machines. In: Parallel distributed processing: explorations in the microstructure of cognition, vol. 1. MIT Press, pp. 45–76 (1986)

    Google Scholar 

  13. Fischer, A., Igel, C.: An introduction to restricted Boltzmann machines. In: Alvarez, L., Mejail, M., Gomez, L., Jacobo, J. (eds.) CIARP 2012. LNCS, vol. 7441, pp. 14–36. Springer, Heidelberg (2012). doi:10.1007/978-3-642-33275-3_2

    Chapter  Google Scholar 

  14. Makhzani, A., Frey, B.: K-sparse autoencoders. Comput. Sci. (2013)

    Google Scholar 

  15. Rolfe, J.T.: Discriminative recurrent sparse auto-encoders. Comput. Sci. (2013)

    Google Scholar 

  16. Ranzato, M.: Sparse feature learning for deep belief networks. In: Proceedings of Advances in Neural Information Processing Systems, Vancouver, BC, Canada, pp. 1185–1192December (2007)

    Google Scholar 

  17. Lee, H.: Sparse deep belief net model for visual area V2. Adv. Neural. Inf. Process. Syst. 20, 873–880 (2007)

    Google Scholar 

  18. Ji, N.-N.: A sparse-response deep belief network based on rate distortion theory. Pattern Recognit. 47(9), 3179–3191 (2014)

    Article  Google Scholar 

  19. Luo, H.: Sparse Group Restricted Boltzmann Machines. Statistics (2010)

    Google Scholar 

  20. Keyvanrad, M.A., Homayounpour, M.M.: Normal sparse deep belief network. In: International Joint Conference on Neural Networks, pp. 1–7 (2015)

    Google Scholar 

  21. Olshausen, B.A.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583), 607–609 (1996)

    Article  Google Scholar 

  22. Li, L.: An evolutionary multiobjective approach to sparse reconstruction. IEEE Trans. Evol. Comput. 8(16), 827–845 (2014)

    Google Scholar 

  23. Chen, W.: Response surface and multiobjective optimization methodology for the design of compliant interconnects. IEEE Trans. Compon. Packag. Manuf. Technol. 4(11), 1769–1777 (2014)

    Article  Google Scholar 

  24. Garcia-Piquer, A.: Large-scale experimental evaluation of cluster representations for multiobjective evolutionary clustering. IEEE Trans. Evol. Comput. 18(1), 36–53 (2014)

    Article  Google Scholar 

  25. Svenson, J., Santner, T.: Multiobjective Optimization of Expensive-to-Evaluate Deterministic Computer Simulator Models. Elsevier Science Publishers B. V, Amsterdam (2016)

    Google Scholar 

  26. Asrari, A.: Pareto dominance-based multiobjective optimization method for distribution network reconfiguration. IEEE Trans. Smart Grid 7(3), 1401–1410 (2016)

    Article  Google Scholar 

  27. Csirmaz, L.: Using multiobjective optimization to map the entropy region. Comput. Optim. Appl. 1, 1–23 (2016)

    MathSciNet  MATH  Google Scholar 

  28. Branke, J.: Interactive evolutionary multiobjective optimization driven by robust ordinal regression. Bull. Pol. Acad. Sci. Tech. Sci. 58(3), 347–358 (2016)

    MathSciNet  MATH  Google Scholar 

  29. Gong, M.: A multiobjective sparse feature learning model for deep neural networks. IEEE Trans. Neural Netw. Learn. Syst. 26(12), 3263–3277 (2015)

    Article  MathSciNet  Google Scholar 

  30. Li, Y.: Dynamic-context cooperative quantum-behaved particle swarm optimization based on multilevel thresholding applied to medical image segmentation. Inf. Sci. 294, 408–422 (2015)

    Article  MathSciNet  Google Scholar 

  31. Cover, T.M.: Elements of Information Theory (Wiley Series in Telecommunications and Signal Processing). Wiley, Hoboken (2006)

    Google Scholar 

  32. Zhang, Q.: MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Trans. Evol. Comput. 11(6), 712–731 (2008)

    Article  Google Scholar 

  33. Hillermeier, C.: Nonlinear Multiobjective Optimization. Birkhaüser Verlag, Basel (2001)

    Book  MATH  Google Scholar 

  34. Venske, S.M.S., Gonçalves, R.A.: ADEMO/D: adaptive differential evolution for multiobjective problems. In: Brazilian Symposium on Neural Networks, pp. 226–231. IEEE (2012)

    Google Scholar 

  35. Rachmawati, L.: Multiobjective evolutionary algorithm with controllable focus on the knees of the Pareto front. IEEE Trans. Evol. Comput. 13(4), 810–824 (2009)

    Article  Google Scholar 

  36. Vincent, P.: Stacked denoising autoencoders: learning useful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res. 11(12), 3371–3408 (2010)

    MathSciNet  MATH  Google Scholar 

  37. Davis, G.: Adaptive nonlinear approximations. Ph.D. dissertation, Department of Mathematics Courant Institute Mathematical Science, New York University, New York (1994)

    Google Scholar 

  38. Ng, A.Y.: Feature selection, L1 vs. L2 regularization, and rotational invariance. In: Proceedings of 21st International Conference Machine Learning, Banff, AB, Canada, pp. 379–387 (2004)

    Google Scholar 

  39. Lee, H., Battle, A.: Efficient sparse coding algorithms. In: Proceedings of Advances in Neural Information Processing Systems, Vancouver, BC, Canada, pp. 801–808 (2006)

    Google Scholar 

  40. Qin, A.K., Suganthan, P.N.: Self-adaptive differential evolution algorithm for numerical optimization. In: IEEE Congress on Evolutionary Computation, vol. 2, pp. 1785–1791 (2005)

    Google Scholar 

  41. Branke, J., Deb, K., Dierolf, H., Osswald, M.: Finding knees in multi-objective optimization. In: Yao, X., et al. (eds.) PPSN 2004. LNCS, vol. 3242, pp. 722–731. Springer, Heidelberg (2004). doi:10.1007/978-3-540-30217-9_73

    Chapter  Google Scholar 

  42. The MNIST Database of Handwirtten Digits. http://yann.lecun.com/exdb/mnist/

  43. Learning Multiple Layers of Reatires from Tiny Images. Alex Krizhevsky (2009). http://www.cs.toronto.edu/~kriz/cifar.html/

  44. Miettinen, K.: Nonlinear Multiobjective Optimization. Kluwer, Norwell (1999)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, Joint International Research Laboratory of Intelligent Perception and Computation, Xidian University, Xi’an, 710071, Shaanxi Province, China

    Yangyang Li, Xiaoyu Bai, Xiaoxu Liang & Licheng Jiao

Authors
  1. Yangyang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiaoyu Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaoxu Liang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Licheng Jiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yangyang Li .

Editor information

Editors and Affiliations

  1. Southern University of Science and Technology, Shenzhen, China

    Yuhui Shi

  2. City University of Hong Kong, Hong Kong, Kowloon, Hong Kong

    Kay Chen Tan

  3. Victoria University of Wellington, Wellington, Wellington, New Zealand

    Mengjie Zhang

  4. Southern University of Science and Technology, Shenzhen, China

    Ke Tang

  5. RMIT University, Melbourne, Victoria, Australia

    Xiaodong Li

  6. City University of Hong Kong, Kowloon Tong, Hong Kong

    Qingfu Zhang

  7. Peking University, Beijing, China

    Ying Tan

  8. University of Leipzig, Leipzig, Germany

    Martin Middendorf

  9. University of Surrey, Guildford, Surrey, United Kingdom

    Yaochu Jin

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this paper

Cite this paper

Li, Y., Bai, X., Liang, X., Jiao, L. (2017). Sparse Restricted Boltzmann Machine Based on Multiobjective Optimization. In: Shi, Y., et al. Simulated Evolution and Learning. SEAL 2017. Lecture Notes in Computer Science(), vol 10593. Springer, Cham. https://doi.org/10.1007/978-3-319-68759-9_73

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-68759-9_73

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-68758-2

  • Online ISBN: 978-3-319-68759-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation