Abstract
Recently, attention based encoder-decoder methods have been widely used in online handwritten mathematical expression recognition, which achieve significant improvements compared to traditional methods. The encoder-decoder methods usually employ string decoders to generate the recognition result, which are not well matched for tree-structured languages like math expression. A novel sequential relation decoder (SRD) was introduced to recognize the online mathematical expression as a math tree, which can be decomposed into a subtree sequence and each subtree consists of a relation node and two symbol nodes (related symbol node and primary symbol node). However, the alignments between these two symbol nodes were implemented by spatial attention probabilities, leading to incorrect recognition if spatial attention is not accurate. In this paper, we propose a memory relation decoder (MRD), equipped with a memory based attention model to determine the correspondence between two symbol nodes. Specifically, at each decoding step, this memory based attention finds the corresponding primary symbol node in the memory and treats it as the related symbol node, which actually achieves the alignments between two symbol nodes in an explicit manner. Besides, we propose to introduce global visual information while calculating attention probabilities to help alleviate the ambiguous problems in online handwritten mathematical expression recognition. Evaluated on a benchmark published by CROHME competition, the proposed approach can substantially outperform previous encoder-decoder methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Álvaro, F., Sánchez, J.A., Benedí, J.M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35, 58–67 (2014)
Alvaro, F., Sánchez, J.A., Benedí, J.M.: An integrated grammar-based approach for mathematical expression recognition. Pattern Recogn. 51, 135–147 (2016)
Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc., Symposium, pp. 436–459 (1967)
Awal, A.M., Mouchère, H., Viard-Gaudin, C.: A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recogn. Lett. 35, 68–77 (2014)
Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., Bengio, Y.: End-to-end attention-based large vocabulary speech recognition. In: International Conference on Acoustics, Speech and Signal Processing, pp. 4945–4949 (2016)
Belaid, A., Haton, J.P.: A syntactic approach for handwritten mathematical formula recognition. IEEE Trans. Pattern Anal. Mach. Intell. 1, 105–111 (1984)
Chan, K.F., Yeung, D.Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3(1), 3–15 (2000)
Chan, W., Jaitly, N., Le, Q., Vinyals, O.: Listen, attend and spell: a neural network for large vocabulary conversational speech recognition. In: International Conference on Acoustics, Speech and Signal Processing, pp. 4960–4964 (2016)
Cho, K.: Natural language understanding with distributed representation. arXiv preprint arXiv:1511.07916 (2015)
Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention. In: International Conference on Machine Learning, pp. 980–989 (2017)
He, T., et al.: Layer-wise coordination between encoder and decoder for neural machine translation. In: Advances in Neural Information Processing Systems, pp. 7944–7954 (2018)
Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)
Huang, P.Y., Liu, F., Shiang, S.R., Oh, J., Dyer, C.: Attention-based multimodal neural machine translation. In: Conference on Machine Translation, vol. 2, pp. 639–645 (2016)
Mahdavi, M., Zanibbi, R., Mouchere, H., Viard-Gaudin, C., Garain, U.: ICDAR 2019 CROHME+ TFD: competition on recognition of handwritten mathematical expressions and typeset formula detection. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1533–1538. IEEE (2019)
Miller, E.G., Viola, P.A.: Ambiguity and constraint in mathematical expression recognition. In: AAAI, pp. 784–791 (1998)
Mouchere, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: International Conference on Frontiers in Handwriting Recognition, pp. 791–796 (2014)
Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 607–612. IEEE (2016)
Vaswani, A., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)
Wang, J., Du, J., Zhang, J.: Stroke constrained attention network for online handwritten mathematical expression recognition. arXiv preprint arXiv:2002.08670 (2020)
Wang, J., Du, J., Zhang, J., Wang, Z.R.: Multi-modal attention network for handwritten mathematical expression recognition. In: International Conference on Document Analysis and Recognition, pp. 1181–1186 (2019)
Wu, J.-W., Yin, F., Zhang, Y.-M., Zhang, X.-Y., Liu, C.-L.: Image-to-markup generation via paired adversarial learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11051, pp. 18–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_2
Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)
Zhang, J., Du, J., Dai, L.: Track, Attend and Parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimedia 21(1), 221–233 (2019)
Zhang, J., Du, J., Yang, Y., Song, Y.Z., Dai, L.: SRD: a tree structure based decoder for online handwritten mathematical expression recognition. IEEE Trans. Multimedia (2020)
Zhang, J., Du, J., Yang, Y., Song, Y.Z., Wei, S., Dai, L.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning, pp. 11076–11085. PMLR (2020)
Zhang, J., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)
Acknowledgement
This work was supported in part by the MOE-Microsoft Key Laboratory of USTC, and Youtu Lab of Tencent.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, J., Wang, Q., Du, J., Zhang, J., Wang, B., Ren, B. (2021). MRD: A Memory Relation Decoder for Online Handwritten Mathematical Expression Recognition. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12823. Springer, Cham. https://doi.org/10.1007/978-3-030-86334-0_3
Download citation
DOI: https://doi.org/10.1007/978-3-030-86334-0_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-86333-3
Online ISBN: 978-3-030-86334-0
eBook Packages: Computer ScienceComputer Science (R0)
