Skip to main content
SpringerLink
Log in

Semantic-Aware Non-local Network for Handwritten Mathematical Expression Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Computer Vision (PRCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13536))

Included in the following conference series:

  • 1654 Accesses

Abstract

Handwritten mathematical expression recognition (HMER) is a challenging task due to its complex two-dimensional structure of mathematical expressions and the high similarity between handwritten texts. Most existing encoder-decoder approaches for HMER mainly depend on local visual features but are seldom studied in explicit global semantic information. Besides, existing works for HMER primarily focus on local information. However, this obtained information is difficult to transmit between distant locations. In this paper, we propose a semantic-aware non-local network to tackle the above problems for HMER. Specifically, we propose to adopt the non-local network to capture long-term dependencies while integrating local and non-local features. Moreover, we customized the FastText language model to our backbone to learn the semantic-aware information. The experimental results illustrate that our design consistently outperforms the state-of-the-art methods on the Competition on Recognition of Online Handwritten Mathematical Expressions (CROHME) 2014 and 2016 datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. He, F., Tan, J., Bi, N.: Handwritten mathematical expression recognition: a survey. In: Lu, Y., Vincent, N., Yuen, P.C., Zheng, W.-S., Cheriet, F., Suen, C.Y. (eds.) ICPRAI 2020. LNCS, vol. 12068, pp. 55–66. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59830-3_5

    Chapter  Google Scholar 

  2. Mouchere, H., et al.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: International Conference on Frontiers in Handwriting Recognition. IEEE (2014)

    Google Scholar 

  3. Mouchère, H., et al.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2016)

    Google Scholar 

  4. Wang, D.-H., et al.: ICFHR 2020 competition on offline recognition and spotting of handwritten mathematical expressions-OffRaSHME. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)

    Google Scholar 

  5. Cheng, Z., Bai, F., Xu, Y., Zheng, G., Pu, S., Zhou, S.: Focusing attention: towards accurate text recognition in natural images. In: ICCV 2017, pp. 5086–5094 (2017)

    Google Scholar 

  6. Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q.V., Salakhutdinov, R.: Transformer-XL: attentive language models beyond a fixed-length context. In: ACL (1), pp. 2978–2988 (2019)

    Google Scholar 

  7. Cheng, J., Dong, L., Lapata, M.: Long short-term memory-networks for machine reading. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016)

    Google Scholar 

  8. Gordo, A., et al.: LEWIS: latent embeddings for word images and their semantics. In: IEEE International Conference on Computer Vision. IEEE (2015)

    Google Scholar 

  9. Wilkinson, T., Brun, A.: Semantic and verbatim word spotting using deep neural networks. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2016)

    Google Scholar 

  10. Wang, Q.-F., Yin, F., Liu, C.-L.: Handwritten Chinese text recognition by integrating multiple contexts. IEEE Trans. Pattern Anal. Mach. Intell. 34(8), 1469–1481 (2011)

    Article  Google Scholar 

  11. Wang, X., et al.: Non-local neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

    Google Scholar 

  12. Bojanowski, P., et al.: Enriching word vectors with subword information. Trans. Assoc. Comput. Linguist. 5, 135–146 (2017)

    Article  Google Scholar 

  13. Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436–459. ACM (1967)

    Google Scholar 

  14. Chan, K.F., Yeung, D.Y.: Error detection, error correction and performance evaluation in on-line mathematical expression recognition. Pattern Recogn. 34(8), 1671–1684 (2001)

    Article  Google Scholar 

  15. Lavirotte, S., Pottier, L.: Mathematical formula recognition using graph grammar. Proc. SPIE Int. Soc. Opt. Eng. 3305, 44–52 (2016)

    Google Scholar 

  16. Yamamoto, R., et al.: On-line recognition of handwritten mathematical expressions based on stroke-based stochastic context-free grammar. In: Proceedings of International Workshop on Frontiers in Handwriting Recognition, pp. 249–254, October 2006

    Google Scholar 

  17. Maclean, S., Labahn, G.: A new approach for recognizing handwritten mathematics using relational grammars and fuzzy sets. Int. J. Doc. Anal. Recogn. 16(2), 139–163 (2013)

    Article  Google Scholar 

  18. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. Computer Science (2014)

    Google Scholar 

  19. Bahdanau, D., et al.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2016)

    Google Scholar 

  20. Zhang, J., et al.: Radical analysis network for zero-shot learning in printed Chinese character recognition. In: 2018 IEEE International Conference on Multimedia and Expo (ICME). IEEE (2018)

    Google Scholar 

  21. Xu, K., et al.: Show, attend and tell: neural image caption generation with visual attention. Computer Science, pp. 2048–2057 (2015)

    Google Scholar 

  22. Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE (2018)

    Google Scholar 

  23. Zhang, J., et al.: Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)

    Article  Google Scholar 

  24. Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR) (2018)

    Google Scholar 

  25. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. Computer Science (2014)

    Google Scholar 

  26. Huang, G., et al.: Densely connected convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

    Google Scholar 

  27. Zhang, J., et al.: A tree-structured decoder for image-to-markup generation. In: International Conference on Machine Learning. PMLR (2020)

    Google Scholar 

  28. Wu, J.-W., Yin, F., Zhang, YM., Zhang, X.-Y., Liu, C.-L.: Image-to-markup generation via paired adversarial learning. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS, vol. 11051, pp. 18–34. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_2

  29. Wu, J.-W., et al.: Handwritten mathematical expression recognition via paired adversarial learning. Int. J. Comput. Vis. 128(10), 2386–2401 (2020)

    Article  MathSciNet  Google Scholar 

  30. Li, Z., et al.: Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)

    Google Scholar 

  31. Zhao, W., Gao, L., Yan, Z., Peng, S., Du, L., Zhang, Z.: Handwritten mathematical expression recognition with bidirectionally trained transformer. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) ICDAR 2021. LNCS, vol. 12822, pp. 570–584. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86331-9_37

  32. Truong, T.-N., et al.: Improvement of end-to-end offline handwritten mathematical expression recognition by weakly supervised learning. In: 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR). IEEE (2020)

    Google Scholar 

  33. Truong, T.-N., Nguyen, C.T., Nakagawa, M.: Syntactic data generation for handwritten mathematical expression recognition. Pattern Recogn. Lett. 153, 83–91 (2022)

    Article  Google Scholar 

  34. Bian, X., et al.: Handwritten mathematical expression recognition via attention aggregation based bi-directional mutual learning. arXiv e-prints (2021)

    Google Scholar 

  35. Yuan, Y., et al.: Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint arXiv:2203.01601 (2022)

  36. Liu, Y.-L., et al.: A robust and fast non-local means algorithm for image denoising. J. Comput. Sci. Technol. 23(2), 270–279 (2008)

    Article  MathSciNet  Google Scholar 

Download references

Acknowledgement

This work is supported by Industry-University Cooperation Project of Fujian Science and Technology Department (No. 2021H6035), and the Science and Technology Planning Project of Fujian Province (No. 2021J011191, 2020H0023, 2020Y9064), and the Joint Funds of 5th Round of Health and Education Research Program of Fujian Province (No. 2019-WJ-41).

Author information

Authors and Affiliations

  1. School of Computer Science and Information Engineering, Xiaman University of Technology, Xiamen, 361024, China

    Xiang-Hao Liu, Da-Han Wang, Xia Du & Shunzhi Zhu

  2. Fujian Key Laboratory of Pattern Recognition and Image Understanding, Xiamen, 361024, China

    Xiang-Hao Liu, Da-Han Wang, Xia Du & Shunzhi Zhu

Authors
  1. Xiang-Hao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Da-Han Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xia Du
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shunzhi Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Da-Han Wang .

Editor information

Editors and Affiliations

  1. Southern University of Science and Technology, Shenzhen, China

    Shiqi Yu

  2. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Zhaoxiang Zhang

  3. Hong Kong Baptist University, Hong Kong, China

    Pong C. Yuen

  4. Northwestern Polytechnical University, Xi'an, China

    Junwei Han

  5. Institute of Automation, Chinese Academy of Sciences, Beijing, China

    Tieniu Tan

  6. Hong Kong Baptist University, Hong Kong, China

    Yike Guo

  7. Sun Yat-sen University, Guangzhou, China

    Jianhuang Lai

  8. Southern University of Science and Technology, Shenzhen, China

    Jianguo Zhang

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, XH., Wang, DH., Du, X., Zhu, S. (2022). Semantic-Aware Non-local Network for Handwritten Mathematical Expression Recognition. In: Yu, S., et al. Pattern Recognition and Computer Vision. PRCV 2022. Lecture Notes in Computer Science, vol 13536. Springer, Cham. https://doi.org/10.1007/978-3-031-18913-5_28

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18913-5_28

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18912-8

  • Online ISBN: 978-3-031-18913-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation