Skip to main content
SpringerLink
Log in

Handwritten Text Recognition with Convolutional Prototype Network and Most Aligned Frame Based CTC Training

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12821))

Included in the following conference series:

Abstract

End-to-end Frameworks with Connectionist Temporal Classification (CTC) have achieved great success in text recognition. Despite high accuracies with deep learning, CTC-based text recognition methods also suffer from poor alignment (character boundary positioning) in many applications. To address this issue, we propose an end-to-end text recognition method based on robust prototype learning. In the new CTC framework, we formulate the blank as the rejection of character classes and use the one-vs-all prototype classifier as the output layer of the convolutional neural network. For network learning, based on forced alignment between frames and character labels, the most aligned frame is up-weighted in CTC training strategy to reduce estimation errors in decoding. Experiments of handwritten text recognition on four benchmark datasets of different languages show that the proposed method consistently improves the accuracy and alignment of CTC-based text recognition baseline.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bartz, C., Bethge, J., Yang, H., Meinel, C.: Kiss: keeping it simple for scene text recognition. arXiv preprint arXiv:1911.08400 (2019)

  2. Bluche, T.: Joint line segmentation and transcription for end-to-end handwritten paragraph recognition. arXiv preprint arXiv:1604.08352 (2016)

  3. Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing attention: towards accurate text recognition in natural images. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 5076–5084 (2017)

    Google Scholar 

  4. Cheng, Z., Xu, Y., Bai, F., Niu, Y., Pu, S., Zhou, S.: Aon: towards arbitrarily-oriented text recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5571–5579 (2018)

    Google Scholar 

  5. Diem, M., et al.: ICFHR 2014 competition on handwritten digit string recognition in challenging datasets (HDSRC 2014). In: Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition, pp. 779–784. IEEE (2014)

    Google Scholar 

  6. Espana-Boquera, S., Castro-Bleda, M.J., Gorbe-Moya, J., Zamora-Martinez, F.: Improving offline handwritten text recognition with hybrid HMM/ANN models. IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 767–779 (2010)

    Article  Google Scholar 

  7. Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd International Conference on Machine Learning, pp. 369–376 (2006)

    Google Scholar 

  8. Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31(5), 855–868 (2008)

    Article  Google Scholar 

  9. Huang, Y.S., et al.: A simulated annealing approach to construct optimized prototypes for nearest-neighbor classification. In: Proceedings of 13th International Conference on Pattern Recognition, vol. 4, pp. 483–487. IEEE (1996)

    Google Scholar 

  10. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  11. Kohonen, T.: The self-organizing map. Proc. IEEE 78(9), 1464–1480 (1990)

    Article  Google Scholar 

  12. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  13. Lee, S.W., Song, H.H.: Optimal design of reference models for large-set handwritten character recognition. Pattern Recogn. 27(9), 1267–1274 (1994)

    Article  Google Scholar 

  14. Liu, C.L.: Classifier combination based on confidence transformation. Pattern Recogn. 38(1), 11–28 (2005)

    Article  Google Scholar 

  15. Liu, C.L., Eim, I.J., Kim, J.H.: High accuracy handwritten Chinese character recognition by improved feature matching method. In: Proceedings of the Fourth International Conference on Document Analysis and Recognition, vol. 2, pp. 1033–1037. IEEE (1997)

    Google Scholar 

  16. Liu, C.L., Nakagawa, M.: Evaluation of prototype learning algorithms for nearest-neighbor classifier in application to handwritten character recognition. Pattern Recogn. 34(3), 601–615 (2001)

    Article  Google Scholar 

  17. Liu, C.L., Sako, H., Fujisawa, H.: Effects of classifier structures and training regimes on integrated segmentation and recognition of handwritten numeral strings. IEEE Trans. Pattern Anal. Mach. Intell. 26(11), 1395–1407 (2004)

    Article  Google Scholar 

  18. Liu, C.L., Yin, F., Wang, D.H., Wang, Q.F.: Casia online and offline Chinese handwriting databases. In: Proceedings of the International Conference on Document Analysis and Recognition, pp. 37–41. IEEE (2011)

    Google Scholar 

  19. Liu, H., Jin, S., Zhang, C.: Connectionist temporal classification with maximum entropy regularization. In: Proceedings of the Advances in Neural Information Processing Systems, pp. 831–841 (2018)

    Google Scholar 

  20. Ly, N.T., Nguyen, K.C., Nguyen, C.T., Nakagawa, M.: Recognition of anomalously deformed kana sequences in Japanese historical documents. IEICE Trans. Inf. Syst. 102(8), 1554–1564 (2019)

    Article  Google Scholar 

  21. Marti, U.V., Bunke, H.: The IAM-database: an English sentence database for offline handwriting recognition. Int. J. Doc. Anal. Recogn. 5(1), 39–46 (2002). https://doi.org/10.1007/s100320200071

    Article  MATH  Google Scholar 

  22. Sato, A., Yamada, K.: Generalized learning vector quantization. In: Advances in Neural Information Processing Systems, pp. 423–429 (1996)

    Google Scholar 

  23. Sato, A., Yamada, K.: A formulation of learning vector quantization using a new misclassification measure. In: Proceedings of the Fourteenth International Conference on Pattern Recognition, vol. 1, pp. 322–325. IEEE (1998)

    Google Scholar 

  24. Shi, B., Bai, X., Yao, C.: An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(11), 2298–2304 (2016)

    Article  Google Scholar 

  25. Shi, B., Wang, X., Lyu, P., Yao, C., Bai, X.: Robust scene text recognition with automatic rectification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4168–4176 (2016)

    Google Scholar 

  26. Shi, B., Yang, M., Wang, X., Lyu, P., Yao, C., Bai, X.: Aster: an attentional scene text recognizer with flexible rectification. IEEE Trans. Pattern Anal. Mach. Intell. 41(9), 2035–2048 (2018)

    Article  Google Scholar 

  27. Su, Bolan, Lu, Shijian: Accurate scene text recognition based on recurrent neural network. In: Cremers, Daniel, Reid, Ian, Saito, Hideo, Yang, Ming-Hsuan. (eds.) ACCV 2014. LNCS, vol. 9003, pp. 35–48. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16865-4_3

    Chapter  Google Scholar 

  28. Sueiras, J., Ruiz, V., Sanchez, A., Velez, J.F.: Offline continuous handwriting recognition using sequence to sequence neural networks. Neurocomputing 289, 119–128 (2018)

    Article  Google Scholar 

  29. Wang, Q., Lu, Y.: A sequence labeling convolutional network and its application to handwritten string recognition. In: Proceedings of the International Joint Conference on Artificial Intelligence, pp. 2950–2956 (2017)

    Google Scholar 

  30. Wang, Q.F., Yin, F., Liu, C.L.: Handwritten Chinese text recognition by integrating multiple contexts. IEEE Trans. Pattern Anal. Mach. Intell. 34(8), 1469–1481 (2011)

    Article  Google Scholar 

  31. Wang, S., Chen, L., Xu, L., Fan, W., Sun, J., Naoi, S.: Deep knowledge training and heterogeneous CNN for handwritten Chinese text recognition. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 84–89. IEEE (2016)

    Google Scholar 

  32. Wang, T., et al.: Decoupled attention network for text recognition. Proc. AAAI Conf. Artif. Intell. 34, 12216–12224 (2020)

    Google Scholar 

  33. Wang, Z.X., Wang, Q.F., Yin, F., Liu, C.L.: Weakly supervised learning for over-segmentation based handwritten Chinese text recognition. In: Proceedings of the 17th International Conference on Frontiers in Handwriting Recognition, pp. 157–162. IEEE (2020)

    Google Scholar 

  34. Wang, Z.R., Du, J., Wang, W.C., Zhai, J.F., Hu, J.S.: A comprehensive study of hybrid neural network hidden Markov model for offline handwritten Chinese text recognition. Proc. Int. J. Doc. Anal. Recogn. 21(4), 241–251 (2018)

    Article  Google Scholar 

  35. Wu, Y.C., Yin, F., Chen, Z., Liu, C.L.: Handwritten Chinese text recognition using separable multi-dimensional recurrent neural network. In: Proceedings of the 14th IAPR International Conference on Document Analysis and Recognition, vol. 1, pp. 79–84. IEEE (2017)

    Google Scholar 

  36. Wu, Y.C., Yin, F., Liu, C.L.: Improving handwritten Chinese text recognition using neural network language models and convolutional neural network shape models. Pattern Recogn. 65, 251–264 (2017)

    Article  Google Scholar 

  37. Yang, H.M., Zhang, X.Y., Yin, F., Liu, C.L.: Robust classification with convolutional prototype learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3474–3482 (2018)

    Google Scholar 

  38. Yang, H.M., Zhang, X.Y., Yin, F., Yang, Q., Liu, C.L.: Convolutional prototype network for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, early access (2020)

    Google Scholar 

  39. Yin, F., Wang, Q.F., Zhang, X.Y., Liu, C.L.: ICDAR 2013 Chinese handwriting recognition competition. In: Proceedings of the 12th International Conference on Document Analysis and Recognition, pp. 1464–1470. IEEE (2013)

    Google Scholar 

  40. Yin, F., Wu, Y.C., Zhang, X.Y., Liu, C.L.: Scene text recognition with sliding convolutional character models. arXiv preprint arXiv:1709.01727 (2017)

  41. Zhan, H., Lyu, S., Lu, Y.: Handwritten digit string recognition using convolutional neural network. In: Proceedings of the 24th International Conference on Pattern Recognition, pp. 3729–3734. IEEE (2018)

    Google Scholar 

  42. Zhan, H., Wang, Q., Lu, Y.: Handwritten digit string recognition by combination of residual network and RNN-CTC. In: Proceedings of the International Conference on Neural Information Processing, pp. 583–591. Springer (2017). https://doi.org/10.1007/978-3-319-70136-3_62

  43. Zhang, Y., Nie, S., Liu, W., Xu, X., Zhang, D., Shen, H.T.: Sequence-to-sequence domain adaptation network for robust text image recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2740–2749 (2019)

    Google Scholar 

Download references

Acknowledgements

This work has been supported by the National Key Research and Development Program Grant 2020AAA0109702, the National Natural Science Foundation of China (NSFC) grants 61936003, 61721004.

Author information

Authors and Affiliations

  1. National Laboratory of Pattern Recognition (NLPR), Institution of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Likun Gao, Heng Zhang & Cheng-Lin Liu

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China

    Likun Gao & Cheng-Lin Liu

  3. CAS Center for Excellence in Brain Science and Intelligence Technology, Beijing, 100190, China

    Cheng-Lin Liu

Authors
  1. Likun Gao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heng Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Cheng-Lin Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Cheng-Lin Liu .

Editor information

Editors and Affiliations

  1. Universitat Autònoma de Barcelona, Barcelona, Spain

    Josep Lladós

  2. Lehigh University, Bethlehem, PA, USA

    Daniel Lopresti

  3. Kyushu University, Fukuoka-shi, Japan

    Seiichi Uchida

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Gao, L., Zhang, H., Liu, CL. (2021). Handwritten Text Recognition with Convolutional Prototype Network and Most Aligned Frame Based CTC Training. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12821. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86549-8_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86548-1

  • Online ISBN: 978-3-030-86549-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation