Deep Learning Approach of Sparse Autoencoders with Lp/L2 Regularization
Authors: 
Ziheng Wu, Cong Li, Baigen PanAuthors Info & Claims
Ziheng Wu, Cong Li, Baigen PanAuthors Info & ClaimsArticle No.: 33, Pages 1 - 5
Abstract
In this paper, we put forward a novel deep learning approach with Lp / L2 regularization based on sparse autoencoders network, trained by a nonnegativity constraint algorithm. Since L2 norm regularization penalizes the negative weights with smaller magnitudes much weaker than those with bigger magnitudes, lots of the weights could take small negative values. In order to address this issue, non-Lipschitz nonconvex LP norm (0<p<1) regularization which could force most of the negative weights to become non-negative is introduced, and the combination of LP/L2 norm regularization is applied for nonnegativity constraint. The proposed approach is analyzed for accuracy on the MNIST dataset for image classification, the experimental results have indicated that using both LP and L2 regularizations could induce non-negativity of weights and have promising performance.
References
[1]
Y Bengio (2009). Learning deep architectures for AI[J]. Found. Trends Mach. Learn., 2(1), 1--127.
[2]
G E Hinton and R R Salakhutdinov (2006). Reducing the dimensionality of data with neural networks[J]. Science, 313, (5786), 504--507.
[3]
S Bengio, L Deng, H Larochelle, H Lee and R Salakhutdinov (2013). Guest editors' introduction: Special section on learning deep architectures[J]. IEEE Trans. Pattern Anal. Mach. Intell, 35(8), 1795--1797.
[4]
B Hutchinson, L Deng and D Yu (2013). Tensor deep stacking networks[J]. IEEE Trans. Pattern Anal. Mach. Intell., 35(8), 1944--1957.
[5]
G E Hinton, S Osindero and Y W Teh (2006). A fast learning algorithm for deep belief nets[J]. Neural Comput., 18(7), 1527--1554.
[6]
P Vincent, H Larochelle, Y Bengio and P A Manzagol (2008). Extracting and Composing Robust Features with Denoising Autoencoders[C]. International Conference on Machine Learning, 1096--1103.
[7]
Y Liu and Y Zhang (2018). Low-dose CT restoration via stacked sparse denoising autoencoders[J]. Neurocomputing, 284, 80--89.
[8]
L Meng, S Ding and Y Xue (2017). Research on denoising sparse autoencoder[J]. International Journal of Machine Learning & Cybernetics, 8, 1719--1729.
[9]
J Chorowski and J M Zurada (2015). Learning understandable neural networks with nonnegative weight constraints[J]. IEEE transactions on neural networks and learning systems, 26, 62--69.
[10]
Ali F Yangyu (2017). Automatic Modulation Classification Using Deep Learning Based on Sparse Autoencoders With Nonnegativity Constraints[J]. IEEE Signal Processing Letters, 24, 1626--1630.
[11]
C Q Hong, J Yu, J Wan, D Tao and M Wang (2015). Multimodal Deep Autoencoder for Human Pose Recovery[J]. IEEE Transactions on Image Processing, 24, 5659--5670.
[12]
H Liu, T Taniguchi, Y Tanaka, K Takenaka and T Bando (2017). Visualization of driving behavior based on hidden feature extraction by using deep learning[J]. IEEE Transactions on Intelligent Transportation Systems, 1--13.
[13]
Xu R B, Che Y, Wang X M, et al. (2020). Stacked autoencoder-based community detection method via an ensemble clustering framework[J]. Information sciences, 526, 151--165.
[14]
Chen P Y and Huang J J (2019). A Hybrid Autoencoder Network for Unsupervised Image Clustering[J]. Algorithms, 12(6), 122.
[15]
B O Ayinde, E Hosseini-Asl and J M Zurada (2016). Visualizing and Understanding Nonnegativity Constrained Sparse Autoencoder in Deep Learning[C]. International Conference on Artificial Intelligence and Soft Computing, 3--14.
[16]
Hosseini-Asl E, Zurada J M and Nasraoui O (2016). Deep Learning of Part-based Representation of Data Using Sparse Autoencoders with Nonnegativity Constraints[J]. IEEE Transactions on Neural Networks & Learning Systems, 27(12), 2486--2498.
[17]
Azar R, Jingjia X and Linwei W (2013). Lp-Norm Regularization in Volumetric Imaging of Cardiac Current Sources[J]. Computational and Mathematical Methods in Medicine, 2013, 1--10.
[18]
Chen X, Ng M and Zhang C (2012). Non-Lipschitz lp-regularization and box constrained model for image restoration[J]. IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, 21(12), 4709--4721.
[19]
M E Davies and R Gribonval (2009). Restricted isometry constants where lp sparse recovery can fail for 0 < p ≤ 1 [J]. IEEE Transactions on Information Theory, 55(5), 2203--2214.
[20]
Kobayashi T (2013). Lp-regularized optimization by using orthant-wise approach for inducing sparsity[C]. IEEE International Conference on Acoustics, IEEE.
[21]
Byrd R H, Lu P and Nocedal J (1995). A Limited Memory Algorithm for Bound Constrained Optimization[J]. SIAM Journal on Scientific Computing, 16(5), 1190--1208.
[22]
Y LeCun, L Bottou, Y Bengio and P Haffner (1998). Gradient-based learning applied to document recognition[J]. Proc. IEEE, 86(11), 2278--2324.
Index Terms
- Deep Learning Approach of Sparse Autoencoders with Lp/L2 Regularization
Recommendations
An antinoise sparse representation method for robust face recognition via joint l1 and l2 regularization
L1 or L2 regularization based representation is not antinoise enough.An antinoise sparse representation via joint L1 and L2 is proposed.The rationale of the objective function for fusion is analyzed.Recognition of noisy samples is evaluated as true ...
On group-wise ℓp regularization
Following advances in compressed sensing and high-dimensional statistics, many pattern recognition methods have been developed with 1 regularization, which promotes sparse solutions. In this work, we instead advocate the use of p ( 2 p 1 ) ...
A sparse autoencoder compressed sensing method for acquiring the pressure array information of clothing
In this paper, we integrate some ideas of sparse autoencoder of deep learning into compressed sensing (CS) theory, and set up a sparse autoencoder compressed sensing (SAECS) model, which can improve the compressed sampling process of CS with compression ...
Comments
Information & Contributors
Information
Published In
October 2020
1038 pages
ISBN:9781450377720
DOI:10.1145/3424978
- Conference Chair:
- Ali Emrouznejad,
- Program Chair:
- Jui-Sheng Rayson Chou
Copyright © 2020 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 20 October 2020
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- science and technology projects of Xuancheng
- science foundation for young scientists of Anhui university of technology
- natural science foundation Anhui Province
- open project of Anhui province key laboratory of special and heavy load robot
Conference
CSAE 2020
CSAE 2020: The 4th International Conference on Computer Science and Application Engineering
October 20 - 22, 2020
Sanya, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 54Total Downloads
- Downloads (Last 12 months)2
- Downloads (Last 6 weeks)0
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in