Abstract
The performance of all types of neural networks is affected by initialization of weights since most of these networks follow some form of the derivative gradient descent algorithm for weight optimization that tends to get trapped in local minima. A new scheme is presented in our work for initializing the weights of Convolutional Neural Networks (CNNs) by part-learning using only 5% of the training data, in a minimalistic approach. The problem at hand is the classification of handwritten numeral images. The parts that are learned, by two-way CNNs, comprise of the top-half and bottom-half of the numeral image respectively, with the second half of the image kept masked. The two set of weights initialized in this manner are respectively fine-tuned on the remaining 95% training images. The probabilistic softmax scores of the two CNNs are fused in the last stage to decide the test label. Our work validates the theory inspired from human cognition that learning in stages with increasing size and complexity of the training data improves the performance over time, rather than training on the complete dataset at one go. Experiments on the benchmark MNIST handwritten English numeral dataset yield high accuracies as compared to the state of the art.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
Bansal, A., Ranjan, R., Castillo, C.D., Chellappa, R.: Deep features for recognizing disguised faces in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 10–16 (2018)
Wang, J., Chen, Y., Hao, S., Peng, X., Lisha, H.: Deep learning for sensor-based activity recognition: a survey. Pattern Recogn. Lett. 119, 3–11 (2019)
Shamsolmoali, P., Kumar Jain, D., Zareapoor, M., Yang, J., Afshar Alam, M.: High-dimensional multimedia classification using deep CNN and extended residual units. Multimedia Tools Appl. 78(17), 23867–23882 (2018). https://doi.org/10.1007/s11042-018-6146-7
Susan, S., Dwivedi, M.: Dynamic growth of hidden-layer neurons using the non-extensive entropy. In: 2014 Fourth International Conference on Communication Systems and Network Technologies, pp. 491–495. IEEE (2014)
Susan, S., Ranjan, R., Taluja, U., Rai, S., Agarwal, P.: Neural net optimization by weight-entropy monitoring. In: Verma, N., Ghosh, A. (eds.) Computational Intelligence: Theories, Applications and Future Directions -, vol. II, pp. 201–213. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-1135-2_16
Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. IJCAI 89, 762–767 (1989)
Aljarah, I., Faris, H., Mirjalili, S.: Optimizing connection weights in neural networks using the whale optimization algorithm. Soft. Comput. 22(1), 1–15 (2016). https://doi.org/10.1007/s00500-016-2442-1
Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Dewa, C.K.: Suitable CNN weight initialization and activation function for javanese vowels classification. Procedia Comput. Sci. 144, 124–132 (2018)
Saini, M., Susan, S.: Comparison of deep learning, data augmentation and bag of-visual-words for classification of imbalanced image datasets. In: Santosh, K.C., Hegadi, Ravindra S. (eds.) RTIP2R 2018. CCIS, vol. 1035, pp. 561–571. Springer, Singapore (2019). https://doi.org/10.1007/978-981-13-9181-1_49
Saini, M., Susan, S.: Deep transfer with minority data augmentation for imbalanced breast cancer dataset. Appl. Soft Comput. 97, 106759 (2020)
Carneiro, G., Nascimento, J., Bradley, Andrew P.: Unregistered multiview mammogram analysis with pre-trained deep learning models. In: Navab, N., Hornegger, J., Wells, William M., Frangi, Alejandro F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 652–660. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_78
Bulat, A., Tzimiropoulos, G., Kossaifi, J., Pantic, M.: Improved training of binary networks for human pose estimation and image recognition. arXiv preprint arXiv:1904.05868 (2019)
Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press, Oxford (1995)
Tang, P., Wang, X., Huang, Z., Bai, X., Liu, W.: Deep patch learning for weakly supervised object classification and discovery. Pattern Recogn. 71, 446–459 (2017)
Lin, W., Zhang, Y., Jiwen, L., Zhou, B., Wang, J., Zhou, Yu.: Summarizing surveillance videos with local-patch-learning-based abnormality detection, blob sequence optimization, and type-based synopsis. Neurocomputing 155, 84–98 (2015)
Susan, S., Sethi, D., Arora, K.: CW-CAE: pulmonary nodule detection from imbalanced dataset using class-weighted convolutional autoencoder. In: Gupta, D., Khanna, A., Bhattacharyya, S., Hassanien, A.E., Anand, S., Jaiswal, A. (eds.) International Conference on Innovative Computing and Communications. AISC, vol. 1166, pp. 825–833. Springer, Singapore (2021). https://doi.org/10.1007/978-981-15-5148-2_71
Susan, S., Singh, V.: On the discriminative power of different feature subsets for handwritten numeral recognition using the box-partitioning method. In: 2011 Annual IEEE India Conference, pp. 1–5. IEEE (2011)
Huang, F.-A., Su, C.-Y., Chu, T.-T.: Kinect-based mid-air handwritten digit recognition using multiple segments and scaled coding. In: 2013 International Symposium on Intelligent Signal Processing and Communication Systems, pp. 694–697. IEEE (2013)
Sun, J., Ponce, J.: Learning discriminative part detectors for image classification and cosegmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3400–3407 (2013)
Zagoruyko, S., Komodakis, N.: Learning to compare image patches via convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4353–4361 (2015)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Liu, X., Wang, X., Matwin, S.: Interpretable deep convolutional neural networks via meta-learning. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–9. IEEE (2018)
Yang, S., Luo, P., Loy, C.C., Shum, K.W., Tang, X.: Deep representation learning with target coding. In: Twenty-Ninth AAAI Conference on Artificial Intelligence (2015)
Visin, F., Kastner, K., Cho, K., Matteucci, M., Courville, A., Bengio, Y.: ReNet: a recurrent neural network based alternative to convolutional networks. arXiv preprint arXiv:1505.00393 (2015)
Srivastava, R.K., Greff, K., Schmidhuber, J.: Training very deep networks. In: Advances in Neural Information Processing Systems, pp. 2377–2385 (2015)
Susan, S., Malhotra, J.: Learning interpretable hidden state structures for handwritten numeral recognition. In: 2020 4th International Conference on Computational Intelligence and Networks (CINE), pp. 1–6. IEEE (2020)
Susan, S., Malhotra, J.: Recognising devanagari script by deep structure learning of image quadrants. DESIDOC J. Libr. Inf. Technol. 40(5), 268–271 (2020)
https://github.com/JMalhotra7/CNN-Pre-Initialization-by-Minimalistic-Part-Learning-for-Handwritten-Numeral-Recognition. Accessed 20 Dec 2019
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Susan, S., Malhotra, J. (2020). CNN Pre-initialization by Minimalistic Part-Learning for Handwritten Numeral Recognition. In: B. R., P., Thenkanidiyoor, V., Prasath, R., Vanga, O. (eds) Mining Intelligence and Knowledge Exploration. MIKE 2019. Lecture Notes in Computer Science(), vol 11987. Springer, Cham. https://doi.org/10.1007/978-3-030-66187-8_30
Download citation
DOI: https://doi.org/10.1007/978-3-030-66187-8_30
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66186-1
Online ISBN: 978-3-030-66187-8
eBook Packages: Computer ScienceComputer Science (R0)