Abstract
Strategically injected noise can speed up convergence during Neural Network training using backpropagation algorithm. Noise injection during Neural Network training have been proven empirically to improve convergence and generalizability. In this work, a new methodology proven to be efficient for speeding up learning convergence using weight noise in Single Layer Feed-forward Network (SLFN) architecture is presented. We present efficient and effective methods in which local minimum entrapment can be avoided. Our proposed controlled introduction of noise is based on 4 proven analytical and experimental methods. We show that criteria-based mini-batch noise injection to the weights during training often outperforms the noiseless weights as well as fixed noise introduction as seen in literature both in network generalization and convergence speed. The effectiveness of this methodology has been empirically shown as well as it achieving on an average 15%–25% improvement in convergence speed when compared to fixed and noiseless networks. The proposed method is evaluated on the MNIST dataset and other datasets from UCI repository. The comparative analysis confirms that the proposed method achieves superior performance regarding convergence speed.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15, 1929–1958 (2014)
Russell, R.: Pruning algorithms-a survey. IEEE Trans. Neural Netw. 4, 740–747 (1993)
Kartik, A., Osoba, O., Kosko, B.: Noise-enhanced convolutional neural networks. Neural Net. 78, 15–23 (2016)
Reed, R., Marks, R.J., Oh, S.: Similarities of error regularization, sigmoid gain scaling, target smoothing, and training with jitter. IEEE Trans. Neural Netw. 6(3), 529–538 (1995)
Murray, A.F., Edwards, P.J.: Enhanced MLP performance and fault tolerance resulting from synaptic weight noise during training. IEEE Trans. Neural Netw. 5(5), 792–802 (1994)
Murray, A.F., Edwards, P.J.: Synaptic weight noise during multilayer perceptron training: fault tolerance and training improvements. IEEE Trans. Neural Netw. 4(4), 722–725 (1993)
Jim, K.C., Giles, C.L., Horne, B.G.: Synaptic noise in dynamically-driven recurrent neural networks: convergence and generalization (1998)
Holmstrom, L., Koistinen, P.: Using additive noise in back-propagation training. IEEE Trans. Neural Netw. 3(1), 24–38 (1992)
Motaz, S., Kurita, T.: Effect of additive noise for multi-layered perceptron with autoencoders. IEICE Trans. Inf. Syst. 100(7), 1494–1504 (2017)
Di, X., Yu, P.: Multiplicative noise channel in generative adversarial networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1165–1172 (2017)
Chandra, P., Singh, Y.: Regularization and feedforward artificial neural network training with noise. In: IEEE International Joint Conference on Neural Networks (IJCNN), vol. 3, pp. 2366–2371 (2003)
An, G.: The effects of adding noise during backpropagation training on a generalization performance. Neural Comput. 8(3), 643–674 (1996)
Jim, K.C., Giles, C.L., Horne, B.G.: An analysis of noise in recurrent neural networks: convergence and generalization. IEEE Trans. Neural Netw. 7(6), 1424–1438 (1996)
Max, W., Teh, Y.W.: Bayesian learning via stochastic gradient Langevin dynamics. In: Proceedings of the 28th International Conference on Machine Learning (ICML), pp. 681–688 (2011)
Wang, C., Principe, J.C.: Training neural networks with additive noise in the desired signal. IEEE Trans. Neural Networks. 10(6), 1511–1517 (1999)
University of California Machine Learning Repository. https://archive.ics.uci.edu/ml/
MNIST handwritten digit database. http://yann.lecun.com/exdb/mnist/
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Wuraola, A., Patel, N. (2018). Stochasticity-Assisted Training in Artificial Neural Network. In: Cheng, L., Leung, A., Ozawa, S. (eds) Neural Information Processing. ICONIP 2018. Lecture Notes in Computer Science(), vol 11302. Springer, Cham. https://doi.org/10.1007/978-3-030-04179-3_52
Download citation
DOI: https://doi.org/10.1007/978-3-030-04179-3_52
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-04178-6
Online ISBN: 978-3-030-04179-3
eBook Packages: Computer ScienceComputer Science (R0)