Abstract
Handwriting and hand-sketched graphics carry rich information to reveal the insights of the physical and emotional state of the writer. Before analyzing the personal traits of handwriting and hand-sketched graphics, detecting them from the image is the most immediate subproblem in handwritten document analysis and understanding. In this paper, we introduce two Convolutional Neural Networks (CNN) based methods to extract multimodal information (handwriting and hand-sketched graphics) from questionnaire documents. A Connectionist Text Proposal Network (CTPN) based method is proposed to detect handwriting. The first stage employs the VGG-16 model to generate the convolutional feature maps of the original images. Then the second stage adopts a BLSTM based detector to predict the scores of candidate zones. An instance segmentation method using the Mask Region Convolutional Neural Network (Mask-RCNN) is also proposed to solve hand-sketched graphics detection problem. The Mask-RCNN based approach has two parts: the backbone and the head. The backbone is to extract the features over the original image, and the head is to perform bounding boxes regression and mask prediction. At first, a simple Region Proposal Network (RPN) is adopted to generate the proposals of hand-sketched graphics efficiently. Then, the Region of Interest (RoI) features of the above proposals are fed into the Fast-RCNN branch and the mask branch to obtain the bounding boxes and the graphics segmentation results. The best handwriting detection performance of 200 test questionnaire images is that the precision rate is 99.5%, the recall rate is 99.2% and the F-measure score is 99.4%. The best detection performance of hand-sketched graphics is that the precision rate is 99.0%, the recall rate is 99.5% and the F-measure score is 99.3%. Experiments demonstrate that the proposed methods achieve promising results in both handwriting and hand-sketched graphics detection tasks.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Gatys, L., Ecker, A., Bethge, M.: Image style transfer using convolutional neural networks. In: Proceeding of IEEE Conference on Computer Vision Pattern Recognition, pp. 2414–2423 (2016)
Zhi, T., Huang, W., He, T., He, P., Qiao, Y.: Detecting text in natural image with connectionist text proposal network. arXiv e-prints. arXiv:1609.03605 (2016)
Simonyan, K. and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint. arXiv:1409.1556 (2014)
He, K., Gkioxari, G., Dollar, P., Girshick, R.B.: Mask R-CNN. In: Proceeding of IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
Girshick, R.B.: Fast R-CNN. In: Proceeding of IEEE International Conference on Computer Vision, pp. 1440–1448 (2015)
Lin, T., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceeding of IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
Ye, Q., Doermann, D.: Text detection and recognition in imagery: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 37(7), 1480–1500 (2015)
Neumann, L., Matas, J.: A method for text localization and recognition in realworld images. In: Proceeding of Asian Conference on Computer Vision, pp. 770–783 (2010)
Neumann, L., Matas, J.: Real-time scene text localization and recognition. In: Proceeding of IEEE Conference on Computer Vision Pattern Recognition, pp. 3538–3545 (2012)
Epshtein, B., Ofek, E., Wexler, Y.: Detecting text in natural scenes with stroke width transform. In: Proceeding of IEEE Conference on Computer and Vision Pattern Recognition, pp. 2963–2970 (2010)
Yao, C., Bai, X., Liu, W.: A unified framework for multioriented text detection and recognition. IEEE Trans. Image Process. 23(11), 4737–4749 (2014)
Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: ASTER: an attentional scene text recognizer with flexible rectification. IEEE Trans. Pattern Anal. Mach. Intell. 41(9), 2035–2048 (2019)
Liu, W., et al.: Single shot multibox detector. In: Proceeding of European Conference on Computer Vision, pp. 21—37 (2016)
Zhong, Z., Sun, L., Huo, Q.: Improved localization accuracy by LocNet for faster R-CNN based text detection in natural scene images. In: Proceeding of IEEE Conference Document and Analysis, in press
Shi, B., Bai, X., Belongie, S.: Detecting oriented text in natural images by linking segments. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 2550–2558 (2017)
Liao, M., Lyu, P., He, M., Yao, C., Wu, W., Bai, X.: Mask TextSpotter: an end-to-end trainable neural network for spotting text with arbitrary shapes. In: IEEE Transactions on Pattern Analysis and Machine Intelligence, in press
Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)
Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: Proceeding of IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)
Lin, T., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: Proceeding IEEE Conference on Computer Vision and Pattern Recognition, pp. 2117–2125 (2017)
Dai, J., He, K., Li, Y., Ren, S., Sun, J.: Instance-sensitive fully convolutional networks. In: Proceeding European Conference on Computer Vision, pp. 534–549 (2016)
He, P., Huang, W., Qiao, Y., Loy, C., Tang, X.: Reading scene text in deep convolutional sequences. In: Proceeding of AAAI Conference on Artificial Intelligence, pp. 3501–3508 (2016)
Acknowledgment
This work is sponsored by Shanghai Sailing Program (Grant No. 19YF1418400).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Cheng, SY., Xiong, YJ., Zhang, JQ., Cao, YC. (2020). Handwriting and Hand-Sketched Graphics Detection Using Convolutional Neural Networks. In: Lu, Y., Vincent, N., Yuen, P.C., Zheng, WS., Cheriet, F., Suen, C.Y. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2020. Lecture Notes in Computer Science(), vol 12068. Springer, Cham. https://doi.org/10.1007/978-3-030-59830-3_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-59830-3_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-59829-7
Online ISBN: 978-3-030-59830-3
eBook Packages: Computer ScienceComputer Science (R0)