Skip to main content

Advertisement

Springer Nature Link
Log in

Analysis of Different Sparsity Methods in Constrained RBM for Sparse Representation in Cognitive Robotic Perception

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Cognitive robotic systems nowadays are intensively involving learning algorithms to achieve highly adaptive and intelligent behaviors, including actuation, sensing, perception and adaptive control. Deep learning has emerged as an effective approach in image-based robotic perception and actions. Towards cognitive robotic perception based on deep learning, this paper focuses the Constrained Restricted Boltzmann Machine (RBM) on visual images for sparse feature representation. Inspired by sparse coding, the sparse constraints are performed on the hidden layer of RBM to obtain sparse and effective feature representation from perceived visual images. The RBM with Sparse Constraint (RBMSC) is proposed with a generalized optimization problem, where the constraints are applied on the probability density of hidden units directly to obtain more sparse representation. This paper presents three novel RBM variants, namely L 1-RBM, L 2-RBM, and L 1/2-RBM constrained by L 1-norm, L 2-norm, and L 1/2-norm on RBM, respectively. A Deep Belief Network with two hidden layers is built for comparison between each RBM variants. The experiments on MNIST database (Mixed National Institute of Standards and Technology database) show that the L 1/2-RBM can obtain more sparse representation than RBM, L 1-RBM, L 2-RBM, and Sparse-RBM (SRBM) in terms of sparseness metric. For further verification, the proposed methods are still tested on MNIST Variations dataset. The recognition results from perceived images in MNIST and MNIST Variations demonstrate that our proposed constrained RBM variants are feasible for object cognitive and perception, and the proposed L 1/2-RBM and L 1-RBM outperforms RBM and SRBM in terms of object recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Bengio, Y., Courville, A., Vincent, P.: Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(8), 1798–1828 (2013)

    Article  Google Scholar 

  2. Lee, H., Grosse, R., Ranganath, R., Ng, A.Y.: Unsupervised learning of hierarchical representations with convolutional deep belief networks. Commun. ACM 54(10), 95–103 (2011)

    Article  Google Scholar 

  3. Mohamed, A.-r., Dahl, G.E., Hinton, G.: Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing 20(1), 14–22 (2012)

    Article  Google Scholar 

  4. Goertzel, B., Pennachin, C., Geisweiller, N.: Perceptual and motor hierarchies, in Engineering General Intelligence, Part 2, pp. 143–161. Springer (2014)

  5. Ian, L., Honglak, L., Ashutosh, S.: Deep learning for detecting robotic grasps. arXiv:1301.3592v6 (2014)

  6. Noda, K., Arie, H., Suga, Y., Ogata, T.: Multimodal integration learning of robot behavior using deep neural networks. Robot. Auton. Syst. 62, 721–736 (2014)

    Article  Google Scholar 

  7. Längkvist, M., Karlsson, L., Loutfi, A.: A review of unsupervised feature learning and deep learning for time-series modeling. Pattern Recogn. Lett. 42, 11–24 (2014)

    Article  Google Scholar 

  8. Ortega, A.C., Blanco, R.R., Diaz, Y.Á.: Educational data mining: User categorization in virtual learning environments. In: Soft Computing for Business Intelligence, pp. 225–237. Springer (2014)

  9. Liu, C., Chu, W.W., Sabb, F., Parker, D.S., Bilder, R.: Path knowledge discovery: Multilevel text mining as a methodology for phenomics. In: Data Mining and Knowledge Discovery for Big Data, pp. 153–192. Springer (2014)

  10. Sarikaya, R., Hinton, G., Deoras, A.: Application of deep belief networks for natural language understanding. ACM Transcation on Audio Speech and Language Processing 22, 778–784 (2014)

    Google Scholar 

  11. Zhou, S., Chen, Q., Wang, X.: Fuzzy deep belief networks for semi-supervised sentiment classification. Neurocomputing 131, 312–322 (2014)

    Article  Google Scholar 

  12. Tang, Y., Li, Y.: Contour coding based rotating adaptive model for human detection and tracking in thermal catadioptric omnidirectional vision. Appl. Opt. 51(27), 6641–6652 (2012)

    Article  Google Scholar 

  13. Eslami, S., Heess, N., Williams, C., Winn, J.: The shape boltzmann machine: A strong model of object shape. Int. J. Comput. Vis. 107(2), 155–176 (2014)

    Article  MATH  MathSciNet  Google Scholar 

  14. Huang, G.B., Lee, H., Learned-Miller, E.: Learning hierarchical representations for face verification with convolutional deep belief networks. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2518–2525. IEEE (2012)

  15. Lee, H., Ekanadham, C., Ng, A.: Sparse deep belief net model for visual area v2. In: Advances in neural information processing systems, pp. 873–880 (2007)

  16. Ng, A.: Sparse autoencoder, CS294A Lecture notes, p. 72 (2011)

  17. Olshausen, B.A., et al.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583), 607–609 (1996)

    Article  Google Scholar 

  18. Lee, D.D., Seung, H.S.: Learning the parts of objects by non-negative matrix factorization. Nature 401(6755), 788–791 (1999)

    Article  Google Scholar 

  19. Hoyer, P.O.: Non-negative sparse coding, in Neural Networks for Signal Processing. In: Proceedings of the 2002 Workshop on 12th IEEE, pp. 557–565. IEEE (2002)

  20. Pascual-Montano, A., Carazo, J.M., Kochi, K., Lehmann, D., Pascual-Marqui, R.D.: Nonsmooth nonnegative matrix factorization (nsnmf). IEEE Transactions on Pattern Analysis and Machine Intelligence 28(3), 403–415 (2006)

    Article  Google Scholar 

  21. Ngiam, J., Koh, P.W., Chen, Z., Bhaskar, S.A., Ng, A.Y.: Sparse filtering. In NIPS, vol. 11, pp. 1125–1133 (2011)

  22. Xu, Z., Chang, X., Xu, F., Zhang, H.: l 1/2 regularization: A thresholding representation theory and a fast solver. IEEE Transactions on neural networks and learning systems 23(7), 1013–1027 (2012)

    Article  Google Scholar 

  23. Cui, Z., Cao, Z., Yang, J., Feng, J.: Sar target recognition using nonnegative matrix factorization with l 1/2 constraint. In: Proceedings of the 2014 IEEE Radar Conference. Accepted (2014)

  24. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  25. Salakhutdinov, R.: Learning deep generative models. PhD thesis, University of Toronto (2009)

  26. Kim, C.-J., Nelson, C.R.: State-space models with regime switching: classical and gibbs-sampling approaches with applications, vol. 1. MIT Press Books (1999)

  27. Hinton, G.E.: Training products of experts by minimizing contrastive divergence. Neural Comput. 14(8), 1771–1800 (2002)

    Article  MATH  MathSciNet  Google Scholar 

  28. Ge, S.S., Li, Z., Yang, H.: Data driven adaptive predictive control for holonomic constrained under-actuated biped robots. IEEE Trans. Control Syst. Technol. 20(3), 787–795 (2012)

    Article  Google Scholar 

  29. Sohn, K., Zhou, G., Lee, C., Lee, H.: Learning and selecting features jointly with point-wise gated {B} oltzmann machines. In: Proceedings of The 30th International Conference on Machine Learning, pp. 217–225 (2013)

  30. Chan, T.-H., Jia, K., Gao, S., Lu, J., Zeng, Z., Ma, Y.: Pcanet: A simple deep learning baseline for image classification? arXiv:1404.3606 (2014)

  31. http://www.iro.umontreal.ca/lisa/twiki/bin/view.cgi/Public/MnistVariations

  32. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)

    Article  MATH  MathSciNet  Google Scholar 

  33. Hoyer, P.O.: Non-negative matrix factorization with sparseness constraints. J. Mach. Learn. Res. 5, 1457–1469 (2004)

    MATH  MathSciNet  Google Scholar 

  34. Ge, S.S., Zhang, Z., He, H. Weighted graph model based sentence clustering and ranking for document summarization. In: 2011 4th International Conference on Interaction Sciences (ICIS), pp. 90–95. IEEE (2011)

  35. Claudiu Ciresan, D., Meier, U., Gambardella, L.M., Schmidhuber, J.: Deep big simple neural nets excel on handwritten digit recognition. arXiv:1003.0358 (2010)

Download references

Author information

Authors and Affiliations

  1. School of Electronic Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China

    Zongyong Cui, Zongjie Cao & Jianyu Yang

  2. Department of Biomedical Engineering, National University of Singapore, Singapore, 117575, Singapore

    Zongyong Cui & Hongliang Ren

  3. Social Robotics Lab, Interactive Digital Media Institute, National University of Singapore, Singapore, 119613, Singapore

    Zongyong Cui & Shuzhi Sam Ge

Authors
  1. Zongyong Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Shuzhi Sam Ge
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Zongjie Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jianyu Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hongliang Ren
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zongyong Cui.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cui, Z., Ge, S.S., Cao, Z. et al. Analysis of Different Sparsity Methods in Constrained RBM for Sparse Representation in Cognitive Robotic Perception. J Intell Robot Syst 80 (Suppl 1), 121–132 (2015). https://doi.org/10.1007/s10846-015-0213-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-015-0213-3

Keywords