Abstract
We present a temporal classification constraint as an auxiliary learning method for improving the recognition of Handwritten Mathematical Expression (HME). Connectionist temporal classification (CTC) is used to learn the temporal alignment of the input feature sequence and corresponding symbol label sequence. The CTC alignment is trained with the encoder-decoder alignment through a combination of CTC loss and encoder-decoder loss to improve the feature learning of the encoder in the encoder-decoder model. We show the effectiveness of the approach in improving symbol classification and expression recognition on the CROHME datasets.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Anthony, L., Yang, J., Koedinger, K.R.: Toward next-generation, intelligent tutors: adding natural handwriting input. IEEE Multimed. 15, 64–68 (2008)
Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-Markup generation with coarse-to-fine attention. In: Proceedings of the 34th International Conference on Machine Learning, pp. 980–989 (2017)
Zhang, J., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recognit. 71, 196–206 (2017)
Zhang, J., Du, J., Dai, L.: Track, Attend, and Parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimed. 21, 221–233 (2019)
Le, A.D., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions by using improved structural analysis. International Journal on Document Analysis and Recognition (IJDAR) 19(4), 305–319 (2016). https://doi.org/10.1007/s10032-016-0272-4
Álvaro, F., Sánchez, J.A., Benedí, J.M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recognit. Lett. 35, 58–67 (2014)
Mahdavi, M., Zanibbi, R., Mouchère, H.: ICDAR 2019 CROHME + TFD : competition on recognition of handwritten mathematical expressions and typeset formula detection. In: Proceedings of the 15th International Conference on Document Analysis and Recognition, pp. 1533–1538 (2019)
Wang, D.H., et al.: ICFHR 2020 competition on offline recognition and spotting of handwritten mathematical expressions-OffRaSHME. In: Proceedings of the 17th International Conference on Frontiers in Handwriting Recognition, pp. 211–215 (2020)
Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: Proceedings of the 24th International Conference on Pattern Recognition, pp. 2245–2250 (2018)
Wu, J.W., Yin, F., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Handwritten mathematical expression recognition via paired adversarial learning. Int. J. Comput. Vis. 128, 2386–2401 (2020)
Truong, T.N., Nguyen, C.T., Phan, K.M., Nakagawa, M.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: Proceedings of the 17th International Conference on Frontiers in Handwriting Recognition, pp. 181–186 (2020)
Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H., Schmidhuber, J.: A novel connectionist system for unconstrained handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 31, 855–868 (2009)
Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: Proceedings of the 38th IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649 (2013)
Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of the 12th Conference on Empirical Methods in Natural Language Processing, pp. 1412–1421 (2015)
Li, Z., Jin, L., Lai, S., Zhu, Y.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: Proceedings of the 17th International Conference on Frontiers in Handwriting Recognition, pp. 175–180 (2020)
Tu, Z., Lu, Z., Yang, L., Liu, X., Li, H.: Modeling coverage for neural machine translation. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 76–85 (2016)
Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: Proceedings of the 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796 (2014)
Mouchère, H., et al.: ICFHR 2016 CROHME : competition on recognition of online handwritten mathematical expressions. In: Proceedings of the 15th International Conference on Frontiers in Handwriting Recognition, pp. 607–612 (2016)
Acknowledgement
This work is being partially supported by the Grant-in-Aid for Scientific Research (A) 19H01117 and that for Early-Career Scientists 21K17761.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Nguyen, C.T., Nguyen, H.T., Morizumi, K., Nakagawa, M. (2021). Temporal Classification Constraint for Improving Handwritten Mathematical Expression Recognition. In: Barney Smith, E.H., Pal, U. (eds) Document Analysis and Recognition – ICDAR 2021 Workshops. ICDAR 2021. Lecture Notes in Computer Science(), vol 12917. Springer, Cham. https://doi.org/10.1007/978-3-030-86159-9_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-86159-9_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86158-2
Online ISBN: 978-3-030-86159-9
eBook Packages: Computer ScienceComputer Science (R0)
