Skip to main content
SpringerLink
Log in

Offline Handwritten Mathematical Expression Recognition via Graph Reasoning Network

  • Conference paper
  • First Online:
Pattern Recognition (ACPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13188))

Included in the following conference series:

Abstract

Handwritten mathematical expression recognition (HMER) remains a challenge due to the complex 2-D structure layout and variable writing styles. Recently, large progress has been made by using deep encoder-decoder networks, which treat HMER as an Image-to-Sequence task and parse the math expression into a sequence (i.e. LaTeX). However, (1) mathematical expression is a 2-D structure pattern and sequence representation can not explicitly explore the structural relationship between symbols. (2) Image-to-Sequence as recurrent models can not infer in parallel during test stage. In this paper, we formulate mathematical expression recognition as an Image-to-Graph task and propose a Graph Reasoning Network (GRN) for offline HMER task. Compared with sequence representation, graph representation is more interpretable and more consistent with human visual cognition. Our method builds graph on math symbols detected from image, aggregates node and edge features via a Graph Neural Network (GNN) and parses the graph to give Symbol Layout Tree (SLT) format recognition result via node and edge classification. Experiments on public datasets show that our model achieve competitive results against other methods and can interpret the located symbols and inter-relationship explicitly.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Alvaro, F., Sánchez, J.A., Benedí, J.M.: Offline features for classifying handwritten math symbols with recurrent neural networks. In: 2014 22nd International Conference on Pattern Recognition, pp. 2944–2949. IEEE (2014)

    Google Scholar 

  2. Alvaro, F., Sánchez, J.A., Benedí, J.M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35, 58–67 (2014)

    Article  Google Scholar 

  3. Álvaro, F., Sánchez, J.A., Benedí, J.M.: An integrated grammar-based approach for mathematical expression recognition. Pattern Recogn. 51, 135–147 (2016)

    Article  Google Scholar 

  4. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014)

  5. Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., Bengio, Y.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4945–4949. IEEE (2016)

    Google Scholar 

  6. Chan, C.: Stroke extraction for offline handwritten mathematical expression recognition. IEEE Access 8, 61565–61575 (2020)

    Article  Google Scholar 

  7. Chan, K.F., Yeung, D.Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3(1), 3–15 (2000). https://doi.org/10.1007/PL00013549

    Article  Google Scholar 

  8. Dai, B., Zhang, Y., Lin, D.: Detecting visual relationships with deep relational networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3076–3086 (2017)

    Google Scholar 

  9. Deng, Y., Kanervisto, A., Rush, A.M.: What you get is what you see: a visual markup decompiler (2016)

    Google Scholar 

  10. Deng, Y., Kanervisto, A., Ling, J., Rush, A.M.: Image-to-markup generation with coarse-to-fine attention. In: International Conference on Machine Learning, pp. 980–989. PMLR (2017)

    Google Scholar 

  11. He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2961–2969 (2017)

    Google Scholar 

  12. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  13. Hu, L., Zanibbi, R.: Line-of-sight stroke graphs and Parzen shape context features for handwritten math formula representation and symbol segmentation. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 180–186. IEEE (2016)

    Google Scholar 

  14. Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4700–4708 (2017)

    Google Scholar 

  15. Jocher, G., et al.: ultralytics/yolov5: v5.0 - YOLOv5-P6 1280 models, AWS, Supervise.ly and YouTube integrations, April 2021. https://doi.org/10.5281/zenodo.4679653

  16. Julca-Aguilar, F., Mouchère, H., Viard-Gaudin, C., Hirata, N.S.: A general framework for the recognition of online handwritten graphics. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 143–160 (2020)

    Article  Google Scholar 

  17. Le, A.D., Nakagawa, M.: Training an end-to-end system for handwritten mathematical expression recognition by generated patterns. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 1056–1061. IEEE (2017)

    Google Scholar 

  18. Li, L., Tang, S., Deng, L., Zhang, Y., Tian, Q.: Image caption with global-local attention. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  19. Li, X.-H., Yin, F., Liu, C.-L.: Page segmentation using convolutional neural network and graphical model. In: Bai, X., Karatzas, D., Lopresti, D. (eds.) DAS 2020. LNCS, vol. 12116, pp. 231–245. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-57058-3_17

    Chapter  Google Scholar 

  20. Li, Z., Jin, L., Lai, S., Zhu, Y.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 175–180. IEEE (2020)

    Google Scholar 

  21. MacLean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recogn. (IJDAR) 16(2), 139–163 (2013). https://doi.org/10.1007/s10032-012-0184-x

    Article  Google Scholar 

  22. Mahdavi, M., Zanibbi, R.: Visual parsing with query-driven global graph attention (QD-GGA): preliminary results for handwritten math formula recognition. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 570–571 (2020)

    Google Scholar 

  23. Mouchère, H., Viard-Gaudin, C., Zanibbi, R., Garain, U.: ICFHR 2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 607–612. IEEE (2016)

    Google Scholar 

  24. Truong, T.N., Nguyen, C.T., Phan, K.M., Nakagawa, M.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 181–186. IEEE (2020)

    Google Scholar 

  25. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Lio, P., Bengio, Y.: Graph attention networks. arXiv preprint arXiv:1710.10903 (2017)

  26. Wang, D.H., et al.: ICFHR 2020 competition on offline recognition and spotting of handwritten mathematical expressions-OFFRASHME. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 211–215 (2020). https://doi.org/10.1109/ICFHR2020.2020.00047

  27. Wu, J.-W., Yin, F., Zhang, Y.-M., Zhang, X.-Y., Liu, C.-L.: Image-to-markup generation via paired adversarial learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11051, pp. 18–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_2

    Chapter  Google Scholar 

  28. Wu, J.-W., Yin, F., Zhang, Y.-M., Zhang, X.-Y., Liu, C.-L.: Handwritten mathematical expression recognition via paired adversarial learning. Int. J. Comput. Vis. 128(10), 2386–2401 (2020). https://doi.org/10.1007/s11263-020-01291-5

    Article  MathSciNet  MATH  Google Scholar 

  29. Wu, J.W., Yin, F., Zhang, Y.M., Zhang, X.Y., Liu, C.L.: Graph-to-graph: towards accurate and interpretable online handwritten mathematical expression recognition. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 2925–2933 (2021)

    Google Scholar 

  30. Yamamoto, R., Sako, S., Nishimoto, T., Sagayama, S.: On-line recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: Tenth international workshop on frontiers in handwriting recognition. Suvisoft (2006)

    Google Scholar 

  31. Yang, J., Lu, J., Lee, S., Batra, D., Parikh, D.: Graph R-CNN for scene graph generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part I, pp. 690–706. Springer International Publishing, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_41

    Chapter  Google Scholar 

  32. Zanibbi, R., Blostein, D.: Recognition and retrieval of mathematical expressions. Int. J. Doc. Anal. Recogn. (IJDAR) 15(4), 331–357 (2012). https://doi.org/10.1007/s10032-011-0174-4

    Article  Google Scholar 

  33. Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 902–907. IEEE (2017)

    Google Scholar 

  34. Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2245–2250. IEEE (2018)

    Google Scholar 

  35. Zhang, J., Du, J., Yang, Y., Song, Y.Z., Wei, S., Dai, L.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning, pp. 11076–11085. PMLR (2020)

    Google Scholar 

  36. Zhang, J., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)

    Article  Google Scholar 

Download references

Acknowledgments

This work has been supported by the National Key Research and Development Program Grant 2020AAA0109702, and the National Natural Science Foundation of China (NSFC) grants 61733007.

Author information

Authors and Affiliations

  1. Beijing Jiaotong University, Beijing, China

    Jia-Man Tang & Lin-Lin Huang

  2. National Laboratory of Pattern Recognition, Institute of Automation of Chinese Academy of Sciences, Beijing, China

    Jin-Wen Wu & Fei Yin

  3. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

    Jin-Wen Wu

Authors
  1. Jia-Man Tang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jin-Wen Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fei Yin
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Lin-Lin Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jia-Man Tang .

Editor information

Editors and Affiliations

  1. Korea University, Seoul, Korea (Republic of)

    Christian Wallraven

  2. Nanjing University, Nanjing, China

    Qingshan Liu

  3. Osaka University, Osaka, Japan

    Hajime Nagahara

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tang, JM., Wu, JW., Yin, F., Huang, LL. (2022). Offline Handwritten Mathematical Expression Recognition via Graph Reasoning Network. In: Wallraven, C., Liu, Q., Nagahara, H. (eds) Pattern Recognition. ACPR 2021. Lecture Notes in Computer Science, vol 13188. Springer, Cham. https://doi.org/10.1007/978-3-031-02375-0_2

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-02375-0_2

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-02374-3

  • Online ISBN: 978-3-031-02375-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation