Abstract
Recently, deep learning models exhibit promising performance in various applications. However, most of them converge slowly due to gradient vanishing. To address this problem, we propose a fast convergent fully connected deep learning network in this study. Through constraining the input values of nodes on the fully connected layers, the proposed method is able to well mitigate the gradient vanishing problems in training phase, and thus greatly reduces the training iterations required to reach convergence. Nevertheless, the drop of generalization performance is negligible. Experimental results validate the effectiveness of the proposed method.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv:1409.1556
Szegedy C, Liu W, Jia Y, Sermanet P, Reed S, Anguelov D, Erhan D, Vanhoucke V, Rabinovich A (2014) Going Deeper with Convolutions, arXiv preprint arXiv:1409.4842
He K, Zhang X, Ren S, Sun J (2016) In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778
Yang W, Zhang H, Yang J, Wu J, Yin X, Chen Y, Shu H, Luo L, Coatrieux G, Gui Z (2017) Improving low-dose ct image using residual convolutional network. IEEE Access 5:24698
Hu J, Shen L, Sun G (2017) Squeeze-and-excitation networks, arXiv preprint arXiv:1709.01507
Chan TH, Jia K, Gao S, Lu J, Zeng Z, Ma Y (2014) Pcanet: a simple deep learning baseline for image classification? IEEE Trans Image Process 24(12):5017
Zeng R, Wu J, Shao Z, Chen Y, Chen B, Senhadji L, Shu H (2016) Color image classification via quaternion principal component analysis network. Neurocomputing 216:416
Ding C, Li Y, Xia Y, Wei W, Zhang L, Zhang Y (2017) Convolutional neural networks based hyperspectral image classification method with adaptive kernels. Remote Sens 9(6):618
Zhang L, Wei W, Zhang Y, Shen C, Hengel AVD, Shi Q (2018) Cluster sparsity field: an internal hyperspectral imagery prior for reconstruction. Int J Comput Vis. https://doi.org/10.1007/s11263-018-1080-8
Wei W, Zhang L, Tian C, Plaza A, Zhang Y (2017) Structured sparse coding-based hyperspectral imagery denoising with intracluster filtering. IEEE Trans on Geosci Remote Sens 55(12):6860
Wang C, Zhang L, Wei W, Zhang Y (2018) When low rank representation based hyperspectral imagery classification meets segmented stacked denoising auto-encoder based spatial-spectral feature. Remote Sens 10(2):284
Krger J, Westermann R (2003) In: ACM SIGGRAPH, pp 908–916
Byong-Heon K, Burm-Suk S (2005) Design and implementation of jpeg image display board using FFGA. J Digit Contents Soc 6(3):169–174
Le QV, Ngiam J, Coates A, Lahiri A, Prochnow B, Ng AY (2011) In: International conference on machine learning, ICML 2011, Bellevue, Washington, Usa, June 28–July, pp. 265–272
Orr GB, Müller KR (1998) Neural networks: tricks of the trade. Can J Anaesth 41(7):658
Salimans T, Kingma DP (2016) In: Advances in neural information processing systems, pp. 901–909
Ba JL, Kiros JR, Hinton GE (2016) Layer normalization, arXiv preprint arXiv:1607.06450
Qing-kun S, Min HAO (2006) Sturctural optimization of BP neural network based on correlation pruning algorithm. Control Theor Appl 25:4–6
Huang G, Liu Z, Weinberger KQ, van der Maaten L (2017) In: Proceedings of the IEEE conference on computer vision and pattern recognition, vol 1, p 3
Ye SJY, Ning G (2016) A research of optimization algorithm in convolution neural network, Qi. Qi Har Univ (Natural science) 32(2):27
Glorot X, Bordes A, Bengio Y (2011) Deep sparse rectifier neural networks. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp 315–323
Acknowledgements
This work was supported in part by the Key Project of the National Natural Science Foundation of China under Grant 61231016, in part by the National Natural Science Foundations of China under Grants 61471297, 61771397, 61671385 and 61301192, in part by the National Key Research and Development Program of China, and in part by the China 863 Program under Grant 2015AA016402.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ding, C., Li, Y., Zhang, L. et al. Fast-Convergent Fully Connected Deep Learning Model Using Constrained Nodes Input. Neural Process Lett 49, 995–1005 (2019). https://doi.org/10.1007/s11063-018-9872-y
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-018-9872-y