Abstract
To obtain a satisfying deep network, it is important to improve the performance on data representation of an auto-encoder. One of the strategies to enhance the performance is to incorporate sparsity into an auto-encoder. Fortunately, sparsity for the auto-encoder has been achieved by adding a Kullback-Leibler (KL) divergence term to the risk functional. In compressive sensing and machine learning, it is well known that the \(l_1\) regularization is a widely used technique which can induce sparsity. Thus, this paper introduces a smoothed \(l_1\) regularization instead of the mostly used KL divergence to enforce sparsity for auto-encoders. Experimental results show that the smoothed \(l_1\) regularization works better than the KL divergence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn. 2(1), 1–127 (2009)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Hinton, G.E., Osindero, S., Teh, Y.: A fast learning algorithm for deep belief nets. Neural Comput. 18(7), 1527–1554 (2006)
Fischer, A., Igel, C.: An Introduction to restricted Boltzmann machines. In: Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, pp. 14–36 (2012)
Hinton, G.E., Zemel, R.S.: Autoencoders, minimum description length and Helmholtz free energy. Adv. Neural Inf. Process. Syst. 6, 3–10 (1993)
Bengio, Y., Lamblin, P., Popovici, D., Larochelle, H.: Greedy layer-wise training of deep networks. In: Conference on Neural Information Processing Systems, pp. 153–160 (2006)
Lennie, P.: The cost of cortical computation. Current Biol. 13, 493–497 (2003)
Simoncelli, E.P.: Statistical Modeling of Photographic Images, 2nd edn. Academic Press, San Diego (2005)
Olshausen, B.A., Field, D.J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583), 607–609 (1996)
Olshausen, B.A., Field, D.J.: Sparse coding with an overcomplete basis set: a strategy employed by V1? Vis. Res. 37(33), 3311–3325 (1997)
Lee, H., Ekanadham, C., Ng, A.Y.: Sparse deep belief net model for visual area V2. In: Conference on Neural Information Processing Systems, pp. 873–880 (2007)
Luo, H., Shen, R., Niu, C., Ullrich, C.: Sparse group restricted Boltzmann machines. In: AAAI Conference on Artificial Intelligence, pp. 429–434 (2011)
Ng, A.Y.: Sparse autoencoder. CS294A Lecture, Stanford University (2011). http://web.stanford.edu/class/cs294a/sparseAutoencoder_2011new.pdf
Le, Q.V., Ngiam, J., Coates, A., Lahiri, A., Prochnow, B., Ng, A.Y.: On optimization methods for deep learning. In: International Conference on Machine Learning, pp. 265–272 (2011)
Deng, J., Zhang, Z.X., Marchi, E., Schuller, B.: Sparse autoencoder-based feature transfer learning for speech emotion recognition. In: Humaine Association Conference on Affective Computing and Intelligent Interaction, pp. 511–516 (2013)
Lee, H., Battle, A., Raina, R., Ng, A.Y.: Efficient sparse coding algorithms. In: Conference on Neural Information Processing Systems, pp. 801–808 (2006)
Candes, E., Tao, T.: Decoding by linear programming. IEEE Trans. Inf. Theory 15(12), 4203–4215 (2005)
Donoho, D.L.: Compressed sensing. IEEE Trans. Inf. Theory 52(4), 1289–1306 (2006)
Ng, A.Y.: Feature selection, \(L_1\) vs. \(L_2\) regularization, and rotational invariance. In: International Conference on Machine Learning (2004)
Moreau, J.J.: Proximite et Dualite dans un espace Hilbertien. Bulletin de la Society Math matique de France 93, 273–299 (1965)
Nesterov, Y.: Smooth minimization of non-smooth functions. Math. Program. 103(1), 127–152 (2005)
Bech, A., Teboulle, M.: Smoothing and first order methods: a unified framework. SIAM J. Optimization 22(2), 557–580 (2012)
Ng, A.Y., Ngiam, J., Foo, C.Y., Mai, Y., Susen, C.: Ufldl Tutorial (2012). http://ufldl.stanford.edu/wiki/resources/sparseae_exercise.zip
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Nene, S.A., Nayar, S.K., Murase, H.: Columbia Object Image Library (COIL-100). Technical Report, CUCS-006-96, Department of Computer Science, Columbia University (1996)
Hinton, G.E.: A practical guide to training restricted Boltzmann machines. Neural Netw. Tricks Trade 7700, 599–619 (2010)
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China under Grant Nos. 61373093, and 61402310, by the Natural Science Foundation of Jiangsu Province of China under Grant No. BK20140008, by the Natural Science Foundation of the Jiangsu Higher Education Institutions of China under Grant No.13KJA520001, and by the Soochow Scholar Project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing AG
About this paper
Cite this paper
Zhang, L., Lu, Y., Zhang, Z., Wang, B., Li, F. (2016). Sparse Auto-encoder with Smoothed \(l_1\) Regularization. In: Hirose, A., Ozawa, S., Doya, K., Ikeda, K., Lee, M., Liu, D. (eds) Neural Information Processing. ICONIP 2016. Lecture Notes in Computer Science(), vol 9949. Springer, Cham. https://doi.org/10.1007/978-3-319-46675-0_61
Download citation
DOI: https://doi.org/10.1007/978-3-319-46675-0_61
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-46674-3
Online ISBN: 978-3-319-46675-0
eBook Packages: Computer ScienceComputer Science (R0)