Abstract
Convolutional neural networks have delivered exceptional performance in various areas of computer vision. There has been growing research to develop deeper architectures with the availability of large datasets. Training such deep networks on large datasets is a tedious process as it involves optimizing a loss function by updating the parameters of the network. Weight initialization is a vital step before training neural networks as the correct choice of network weights ensures that the optimization converges to global minima in the least time. The weight initialization strategies in the literature can be categorized as (1) Initialization without pre-training, and (2) Initialization with pre-training. This paper presents a comparative analysis of the convergence performance of some widely used weight initialization techniques in these categories. This analysis is based on the diversity insights measured in terms of mean standard deviation captured from the feature maps. The experimentation has been carried out by training the AlexNet and VGG16 network on CIFAR-10 and CIFAR-100 datasets. The experimentation results demonstrate that the He initialization technique, which shows the best convergence performance among the others considered for the study, leads the training process such that the diversity of feature maps increases with epochs for both AlexNet and VGG16 network.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Fukushima, K.: Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 193–202 (1980)
Glorot, X., Bengio, Y.: Understanding the difficulty of training deep feedforward neural networks. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 249–256 (2010)
Gu, J., Wang, Z., Kuen, J., Ma, L., Shahroudy, A., Shuai, B., Liu, T., Wang, X., Wang, G., Cai, J., et al.: Recent advances in convolutional neural networks. Pattern Recogn. 77, 354–377 (2018)
He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Hubel, D.H., Wiesel, T.N.: Receptive fields of single neurones in the cat’s striate cortex. J. Physiol. 148(3), 574–591 (1959)
Koturwar, S., Merchant, S.: Weight initialization of deep neural networks (dnns) using data statistics. arXiv preprint arXiv:1710.10570 (2017)
Krizhevsky, A., Nair, V., Hinton, G.: Cifar-10 and cifar-100 datasets, vol. 6 (2009). https://www.cs.toronto.edu/kriz/cifar.html
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
LeCun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. In: Advances in Neural Information Processing Systems, pp. 396–404 (1990)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
LeCun, Y.A., Bottou, L., Orr, G.B., Müller, K.R.: Efficient backprop. In: Neural Networks: Tricks of the Trade, pp. 9–48. Springer, Berlin (2012)
Li, H., Ma, C., Xu, W., Liu, X.: Feature statistics guided efficient filter pruning. arXiv preprint arXiv:2005.12193 (2020)
Li, J., Cheng, J.h., Shi, J.y., Huang, F.: Brief introduction of back propagation (bp) neural network algorithm and its improvement. In: Advances in Computer Science and Information Engineering, pp. 553–558. Springer, Berlin (2012)
Masci, J., Meier, U., Cireşan, D., Schmidhuber, J.: Stacked convolutional auto-encoders for hierarchical feature extraction. In: International Conference on Artificial Neural Networks, pp. 52–59. Springer, Berlin (2011)
Mishkin, D., Matas, J.: All you need is a good init. arXiv preprint arXiv:1511.06422 (2015)
Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning internal representations by error propagation. California Univ San Diego La Jolla Inst for Cognitive Science, Technical report (1985)
Saxe, A.M., McClelland, J.L., Ganguli, S.: Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120 (2013)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–9 (2015)
Zeiler, M.D., Fergus, R.: Visualizing and understanding convolutional networks. In: European Conference on Computer Vision, pp. 818–833. Springer, Cham (2014)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Narkhede, M., Bartakke, P.P., Sutaone, M.S. (2021). Delving into Feature Maps: An Explanatory Analysis to Evaluate Weight Initialization. In: Abraham, A., et al. Proceedings of the 12th International Conference on Soft Computing and Pattern Recognition (SoCPaR 2020). SoCPaR 2020. Advances in Intelligent Systems and Computing, vol 1383. Springer, Cham. https://doi.org/10.1007/978-3-030-73689-7_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-73689-7_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-73688-0
Online ISBN: 978-3-030-73689-7
eBook Packages: Intelligent Technologies and RoboticsIntelligent Technologies and Robotics (R0)