Abstract
In this work, a training method was proposed for Deep Neural Networks (DNNs) based on a two-stage structure. Local DNN models are trained in all local machines and uploaded to the center with partial training data. These local models are integrated as a new DNN model (combination DNN). With another DNN model (optimization DNN) connected, the combination DNN forms a global DNN model in the center. This results in greater accuracy than local DNN models with smaller amounts of data uploaded. In this case, the bandwidth of the uploaded data is saved, and the accuracy is maintained as well. Experiments are conducted on MNIST dataset, CIFAR-10 dataset and LFW dataset. The results show that with less training data uploaded, the global model produces greater accuracy than local models. Specifically, this method focuses on condition of big data.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The basic MATLAB code of the proposed model can be freely downloaded from this website: https://pan.baidu.com/s/1kVg5UXD, please click the button with download symbols to get the Two-stage DNN.rar file.
References
Bengio Y, Lamblin P, Popovici D, Larochelle H et al (2007) Greedy layer-wise training of deep networks. Adv Neural Inf Process Syst 19:153
Bourlard H, Kamp Y (1988) Auto-association by multilayer perceptrons and singular value decomposition. Biol Cybern 59(4–5):291–294
Breiman L (1996) Bagging predictors. Mach Learn 24(2):123–140
Gehring J, Miao Y, Metze F, Waibel A (2013) Extracting deep bottleneck features using stacked auto-encoders. In: Acoustics, speech and signal processing (ICASSP), 2013 IEEE international conference on. IEEE, pp 3377–3381
Hinton GE, Salakhutdinov RR (2006) Reducing the dimensionality of data with neural networks. Science 313(5786):504–507
Hinton GE, Salakhutdinov RR (2009) Replicated softmax: an undirected topic model. In: Advances in neural information processing systems, pp 1607–1614
Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In: Technical Report 07-49, University of Massachusetts, Amherst
Krizhevsky A, Hinton G (2009) Learning multiple layers of features from tiny images
Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
LeCun Y, Boser B, Denker JS, Henderson D, Howard RE, Hubbard W, Jackel LD (1989) Backpropagation applied to handwritten zip code recognition. Neural comput 1(4):541–551
LeCun Y, Bottou L, Bengio Y, Haffner P (1998) Gradient-based learning applied to document recognition. Proc IEEE 86(11):2278–2324
Rumelhart DE, McClelland JL, Group PR et al. (1988) Parallel distributed processing, vol 1. IEEE
Sun Y, Chen Y, Wang X, Tang X (2014) Deep learning face representation by joint identification–verification. In: Advances in neural information processing systems, pp 1988–1996
Author information
Authors and Affiliations
Corresponding author
Additional information
This research is supported by the strategic priority research program—“Real-time Processing System of Massive Network Traffic Based on Sea-cloud Collaboration” of the Chinese Academy of Science (Grant No. XDA060112030).
Rights and permissions
About this article
Cite this article
Ding, C., Hu, Z., Karmoshi, S. et al. A Novel Two-stage Learning Pipeline for Deep Neural Networks. Neural Process Lett 46, 159–169 (2017). https://doi.org/10.1007/s11063-017-9578-6
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-017-9578-6