Abstract
The discrimination of similar patterns is important because they are the major sources of the classification error. This paper proposes a novel method to improve the discrimination ability of convolutional neural networks (CNNs) by hybrid learning. The proposed method embeds a collection of discriminators as well as a recognizer in a shared CNN. By visualizing contrastive class saliency, we show that learning with embedded discriminators leads the shared CNN to detect and catch the differences among similar classes. Also proposed is a hybrid learning algorithm that learns recognition and discrimination together. The proposed method learns recognition focusing on the differences among similar classes, and thereby improves the discrimination ability of the CNN. Unlike conventional discrimination methods, the proposed method does not require predefined sets of similar classes or additional step to integrate its result with that of the recognizer. In experiments on two handwritten Hangul databases SERI95a and PE92, the proposed method reduced classification error from 2.56 to 2.33, and from 4.04 to 3.66 % respectively. These improvement lead to relative error reduction rates of 8.97 % on SERI95a, and 9.42 % on PE92. Our best results update the state-of-the-art performance which were 4.04 % on SERI95a and 7.08 % on PE92.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
SERI95a is also known as KU-1.
References
Kim, I.-J., Kim, J.: Pairwise discrimination based on a stroke importance measure. Pattern Recognit. 35(10), 2259–2266 (2002)
Leung, K.C., Leung, C.H.: Recognition of handwritten Chinese characters by critical region analysis. Pattern Recognit. 43(3), 949–961 (2010)
Xu, B., Huang, K., Liu, C.L.: Similar characters recognition by critical region selection based on average symmetric uncertainty. In: Proceedings of 12th ICFHR, Kolkata, India, pp. 527–532 (2010)
Ryu, S.-J., Kim, I.-J.: Discrimination of similar characters using nonlinear normalization based on regional importance measure. Int. J. Doc. Anal. Recognit. (IJDAR) 17(1), 79–89 (2014)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Kim, I.-J., Xie, X.: Handwritten Hangul recognition using deep convolutional neural networks. Int. J. Doc. Anal. Recognit. 18(1), 1–13 (2014)
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition, arXiv:1409.1556 (2014)
Yin, F., Wang, Q.F., Zhang, X.X.Y., Liu, C.L.: ICDAR 2013 Chinese handwriting recognition competition. http://www.nlpr.ia.ac.cn/events/CHRcompetition2013/competition/ICDAR%202013%20CHR%20competition.pdf
Kim, D.-I., Kim, S.-Y., Lee, S.-W.: Design and construction of a large-set off-line handwritten hangul character image database KU-1. In: Proceedings of \(9{{\rm th}}\) National Conference on Korean Language Information Processing, pp. 152–159 (1997) (in Korean)
Kim, D.H., Bang, S.Y.: An overview of hangul handwritten image database PE92. In: Proceedings of 4\({{\rm th}}\) National Conference on Korean Language Information Processing, pp. 152–159 (1992) (in Korean)
Kim, I.-J., Kim, J.: Pairwise discrimination based on a stroke importance measure. Pattern Recognit. 35(10), 2259–2266 (2002)
Goodfellow, I. et al.: Maxout networks, arXiv:1302.4389 (2013)
Lin, M., Chen, Q., Yan, S.: Network in network. In: Proceedings of ICLR (2014)
Szegedy, C. et al.: Going deeper with convolutions, arXiv:1409.4842 (2014)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift, arXiv:1502.03167 (2015)
Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network, arXiv:1503.02531 (2015)
Girshick, R. et al.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE (2014)
Socher, R. et al.: Convolutional-recursive deep learning for 3d object classification. Advances in Neural Information Processing Systems (2012)
Zeiler, M., Fergus, R.: Visualizing and understanding convolutional networks. Compu Vis-ECCV 2014, 818–833 (2014)
Simonyan, K., Vedaldi, A., Zisserman, A.: Deep inside convolutional networks: visualising image classification models and saliency maps, arXiv:1312.6034 (2013)
Sato, A., Yamada, K.: Generalized learning vector quantization. Adv. Neural Inf. Process. Syst. 15(8), 423–429 (1996)
Caruana, R.: Multitask learning. Mach. Learn. 28(1), 41–75 (1997)
Bengio, Y. et al.: Curriculum learning. In: Proceedings of 26th Annual International Conference on Machine Learning. ACM, pp. 41–48 (2009)
Simard, P., Steinkraus, D., Platt, J.: Best practices for convolutional neural networks applied to visual document analysis, null. IEEE, p. 958 (2003)
Srivastava, N., et al.: Dropout: a simple way to prevent neural networks from overfitting. J Mach. Learn. Res. 15(1), 1929–1958 (2014)
Acknowledgments
This work was supported by the DGIST R&D Program of the Ministry of Science, ICT and Future Planning (15-IT-03).
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Kim, IJ., Choi, C. & Lee, SH. Improving discrimination ability of convolutional neural networks by hybrid learning. IJDAR 19, 1–9 (2016). https://doi.org/10.1007/s10032-015-0256-9
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10032-015-0256-9