Abstract
Improving the performance on data representation of an auto-encoder could help to obtain a satisfying deep network. One of the strategies to enhance the performance is to incorporate sparsity into an auto-encoder. Fortunately, sparsity for the auto-encoder has been achieved by adding a Kullback–Leibler (KL) divergence term to the risk functional. In compressive sensing and machine learning, it is well known that the \(l_1\) regularization is a widely used technique which can induce sparsity. Thus, this paper introduces a smoothed \(l_1\) regularization instead of the mostly used KL divergence to enforce sparsity for auto-encoders. Experimental results show that the smoothed \(l_1\) regularization works better than the KL divergence.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Bengio Y (2009) Learning deep architectures for AI. Found Trends Mach Learn 2(1):1–127
Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507
Hinton GE, Osindero S, Teh Y (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554
Fischer A, Igel C (2012) An introduction to restricted Boltzmann machines. In: Progress in pattern recognition, image analysis, computer vision, and applications. pp 14–36
Hinton GE, Zemel RS (1993) Autoencoders, minimum description length and Helmholtz free energy. Adv Neural Inf Process Syst 6:3–10
Bengio Y, Lamblin P, Popovici D, Larochelle H (2006) Greedy layer-wise training of deep networks. In: Conference on neural information processing systems. pp 153–160
Lennie P (2003) The cost of cortical computation. Curr Biol 13:493–497
Simoncelli EP (2005) Statistical modeling of photographic images, 2nd edn. Academic Press, Cambridge
Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583):607–609
Olshausen BA, Field DJ (1997) Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis Res 37(33):3311–3325
Lee H, Ekanadham C, Ng AY (2007) Sparse deep belief net model for visual area V2. In: Conference on neural information processing systems. pp 873–880
Luo H, Shen R, Niu C, Ullrich C (2011) Sparse group restricted Boltzmann machines. In: AAAI conference on artificial intelligence. pp 429–434
Ng AY (2011) Sparse Autoencoder. CS294A Lecture, Stanford University. http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf
Le QV, Ngiam J, Coates A, Lahiri A, Prochnow B, Ng AY (2011) On optimization methods for deep learning. In: International conference on machine learning. pp 265–272
Deng J, Zhang ZX, Marchi E, Schuller B (2013) Sparse autoencoder-based feature transfer learning for speech emotion recognition. In: Humaine association conference on affective computing and intelligent interaction. pp 511–516
Lee H, Battle A, Raina R, Ng AY (2006) Efficient sparse coding algorithms. In: Conference on neural information processing systems. pp 801–808
Candes E, Tao T (2005) Decoding by linear programming. IEEE Trans Inf Theory 15(12):4203–4215
Donoho DL (2006) Compressed sensing. IEEE Trans Inf Theory 52(4):1289–1306
Ng AY (2004) Feature selection, \(L_1\) vs. \(L_2\) regularization, and rotational invariance. In: International conference on machine learning
Moreau JJ (1965) Proximite et Dualite dans un espace Hilbertien. Bull de la Soc Math Matique de France 93:273–299
Nesterov Y (2005) Smooth minimization of non-smooth functions. Math Progr 103(1):127–152
Bech A, Teboulle M (2012) Smoothing and first order methods: a unified framework. SIAM J Optim 22(2):557–580
Ng AY, Ngiam J, Foo CY, Mai Y, Susen C (2012) Ufldl tutorial. http://ufldl.stanford.edu/wiki/resources/sparseae_exercise.zip
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
Nene SA, Nayar SK, Murase H (1996) Columbia object image library (COIL-100). Technical Report, CUCS-006-96, Department of Computer Science, Columbia University
Hinton GE (2010) A practical guide to training restricted Boltzmann machines. Neural Netw Tricks Trade 7700:599–619
Olshausen BA, Field DJ (2004) Sparse coding of sensory inputs. Curr Opin Neurobiol 14(4):481–487
Acknowledgements
This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61373093, 61672364 and 61672365, by the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20140008, by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant No.13KJA520001, and by the Soochow Scholar Project.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Zhang, L., Lu, Y., Wang, B. et al. Sparse Auto-encoder with Smoothed \(l_1\) Regularization. Neural Process Lett 47, 829–839 (2018). https://doi.org/10.1007/s11063-017-9668-5
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-017-9668-5