Skip to main content
Log in

Sparse Auto-encoder with Smoothed \(l_1\) Regularization

  • Published:
Neural Processing Letters Aims and scope Submit manuscript

Abstract

Improving the performance on data representation of an auto-encoder could help to obtain a satisfying deep network. One of the strategies to enhance the performance is to incorporate sparsity into an auto-encoder. Fortunately, sparsity for the auto-encoder has been achieved by adding a Kullback–Leibler (KL) divergence term to the risk functional. In compressive sensing and machine learning, it is well known that the \(l_1\) regularization is a widely used technique which can induce sparsity. Thus, this paper introduces a smoothed \(l_1\) regularization instead of the mostly used KL divergence to enforce sparsity for auto-encoders. Experimental results show that the smoothed \(l_1\) regularization works better than the KL divergence.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127

    Article  MATH  Google Scholar 

  2. Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507

    Article  MathSciNet  MATH  Google Scholar 

  3. Hinton GE, Osindero S, Teh Y (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  4. Fischer A, Igel C (2012) An introduction to restricted Boltzmann machines. In: Progress in pattern recognition, image analysis, computer vision, and applications. pp 14–36

  5. Hinton GE, Zemel RS (1993) Autoencoders, minimum description length and Helmholtz free energy. Adv Neural Inf Process Syst 6:3–10

    Google Scholar 

  6. Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. In: Conference on neural information processing systems. pp 153–160

  7. Lennie P (2003) The cost of cortical computation. Curr Biol 13:493–497

    Article  Google Scholar 

  8. Simoncelli EP (2005) Statistical modeling of photographic images, 2nd edn. Academic Press, Cambridge

    Google Scholar 

  9. Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583):607–609

    Article  Google Scholar 

  10. Olshausen BA, Field DJ (1997) Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis Res 37(33):3311–3325

    Article  Google Scholar 

  11. Lee H, Ekanadham C, Ng AY (2007) Sparse deep belief net model for visual area V2. In: Conference on neural information processing systems. pp 873–880

  12. Luo H, Shen R, Niu C, Ullrich C (2011) Sparse group restricted Boltzmann machines. In: AAAI conference on artificial intelligence. pp 429–434

  13. Ng AY (2011) Sparse Autoencoder. CS294A Lecture, Stanford University. http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf

  14. Le QV, Ngiam J, Coates A, Lahiri A, Prochnow B, Ng AY (2011) On optimization methods for deep learning. In: International conference on machine learning. pp 265–272

  15. Deng J, Zhang ZX, Marchi E, Schuller B (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition. In: Humaine association conference on affective computing and intelligent interaction. pp 511–516

  16. Lee H, Battle A, Raina R, Ng AY (2006) Efficient sparse coding algorithms. In: Conference on neural information processing systems. pp 801–808

  17. Candes E, Tao T (2005) Decoding by linear programming. IEEE Trans Inf Theory 15(12):4203–4215

    Article  MathSciNet  MATH  Google Scholar 

  18. Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306

    Article  MathSciNet  MATH  Google Scholar 

  19. Ng AY (2004) Feature selection, \(L_1\) vs. \(L_2\) regularization, and rotational invariance. In: International conference on machine learning

  20. Moreau JJ (1965) Proximite et Dualite dans un espace Hilbertien. Bull de la Soc Math Matique de France 93:273–299

    Article  MathSciNet  MATH  Google Scholar 

  21. Nesterov Y (2005) Smooth minimization of non-smooth functions. Math Progr 103(1):127–152

    Article  MathSciNet  MATH  Google Scholar 

  22. Bech A, Teboulle M (2012) Smoothing and first order methods: a unified framework. SIAM J Optim 22(2):557–580

    Article  MathSciNet  MATH  Google Scholar 

  23. Ng AY, Ngiam J, Foo CY, Mai Y, Susen C (2012) Ufldl tutorial. http://ufldl.stanford.edu/wiki/resources/sparseae_exercise.zip

  24. LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324

    Article  Google Scholar 

  25. Nene SA, Nayar SK, Murase H (1996) Columbia object image library (COIL-100). Technical Report, CUCS-006-96, Department of Computer Science, Columbia University

  26. Hinton GE (2010) A practical guide to training restricted Boltzmann machines. Neural Netw Tricks Trade 7700:599–619

    Article  Google Scholar 

  27. Olshausen BA, Field DJ (2004) Sparse coding of sensory inputs. Curr Opin Neurobiol 14(4):481–487

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61373093, 61672364 and 61672365, by the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20140008, by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant No.13KJA520001, and by the Soochow Scholar Project.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Li Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, L., Lu, Y., Wang, B. et al. Sparse Auto-encoder with Smoothed \(l_1\) Regularization. Neural Process Lett 47, 829–839 (2018). https://doi.org/10.1007/s11063-017-9668-5

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11063-017-9668-5

Keywords

Navigation