Skip to main content
SpringerLink
Log in

Application of Deep Learning in Handwritten Mathematical Expressions Recognition

  • Conference paper
  • First Online:
Pattern Recognition and Artificial Intelligence (ICPRAI 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12068))

Abstract

Handwritten mathematical expression (HME) is a research highlight in pattern recognition area since it has many practical application scenarios such as scientific documents digitalization and online education. However, it can be very challenging due to its complicated two-dimensional structure and great variety among different individuals. Inspired by the excellent performance of deep learning in various recognition problems, some researchers in this field began to use deep learning method to solve the problem of handwriting recognition in order to reach the prediction accuracy that traditional resolutions cannot achieve. In this survey paper, we aim to review the solutions of deep learning (encoder-decoder, GRU, attention mechanism and so on) to various problems of handwritten mathematical expression recognition. In particular, some important methods and issues will be described in depth. Moreover, we will try to discuss the future application of deep learning to HME recognition. All these together will help us have a clear picture about how deep learning methods are applied to in this research area.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 109.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 139.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Chan, K.-F., Yeung, D.-Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3(1), 3–15 (2000). https://doi.org/10.1007/PL00013549

  2. Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc. Symposium, pp. 436–459. ACM (1967)

    Google Scholar 

  3. Mouchère, H., Viardgaudin, C., Garain, U., et al.: CROHME2011: Competition on Recognition of Online Handwritten Mathematical Expressions (2011)

    Google Scholar 

  4. Plamondon, R., Srihari, S.N.: On-line and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. (T-PAMI) 22(1), 63–84 (2002)

    Article  Google Scholar 

  5. Blostein, D., Grbavec, A.: Recognition of mathematical notation. In: Handbook of Character Recognition and Document Image Analysis, pp. 557–582 (1997)

    Google Scholar 

  6. Zhang, J., Du, J., Zhang, S., et al.: Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)

    Article  Google Scholar 

  7. Zhang, J., Du, J., Dai, L.: Track, Attend, and Parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimedia 21(1), 221–233 (2018)

    Article  Google Scholar 

  8. Zhang, J., Du, J., Dai, L.: Track, Attend, and Parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimedia 21(1), 221–233 (2018)

    Article  Google Scholar 

  9. Hu, L., Zanibbi, R.: Segmenting handwritten math symbols using AdaBoost and multi-scale shape context features. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1180–1184. IEEE (2013)

    Google Scholar 

  10. Suykens, J.A.K., Vandewalle, J.: Least Squares Support Vector Machine Classifiers. Kluwer Academic Publishers (1999)

    Google Scholar 

  11. Gardner, M.W.: Artificial neural networks (the multiplayer perceptron)-a review of applications in the atmospheric sciences. Atmospheric Environ. 32 (1998)

    Google Scholar 

  12. Álvaro, F., Sánchez, J.-A., Benedí, J.-M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35(1), 58–67 (2014)

    Article  Google Scholar 

  13. Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Machine Intell. 24(11), 1455–1467 (2002)

    Google Scholar 

  14. Garain, U., Chaudhuri, B.B.: Recognition of online handwritten mathematical expressions. IEEE Trans. Syst. Man Cybern. Part B: Cybern. 34(6), 2366–2376 (2004)

    Google Scholar 

  15. Lee, H.J., Wang, J.S.: Design of a mathematical expression recognition system. Pattern Recogn. Lett. 18(3), 1084–1087 (1997)

    Google Scholar 

  16. Chan, K.F., Yeung, D.Y.: PenCalc: a novel application of on-line mathematical expression recognition technology. In: Proceedings of Sixth International Conference on Document Analysis and Recognition, pp. 774–778. IEEE (2001)

    Google Scholar 

  17. Awal, A.-M., Mouchère, H., Viard-Gaudin, C.: A global learning approach for an online handwritten mathematical expression recognition system. Pattern Recogn. Lett. 35, 68–77 (2014)

    Google Scholar 

  18. Alvaro, F., Sánchez, J.A., Benedí, J.M.: An integrated grammar-based approach for mathematical expression recognition. Pattern Recogn. 51, 135–147 (2016)

    Article  Google Scholar 

  19. Zhang, J., Du, J., Zhang, S., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)

    Article  Google Scholar 

  20. Zhang, J., Du, J., Dai, L.: A GRU-based encoder-decoder approach with attention for online handwritten mathematical expression recognition. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), vol. 1, pp. 902–907. IEEE (2017)

    Google Scholar 

  21. Zhang, J., Du, J., Dai, L.: Track, Attend, and Parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimedia 21(1), 221–233 (2018)

    Article  Google Scholar 

  22. Zhang, J., Du, J., Dai, L.: Multi-scale attention with dense encoder for handwritten mathematical expression recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 2245–2250. IEEE (2018)

    Google Scholar 

  23. Zhang, T., Mouchère, H., Viard-Gaudin, C.: A tree-BLSTM-based recognition system for online handwritten mathematical expressions. Neural Comput. Appl. 32, 4689–4708 (2020)

    Google Scholar 

  24. Huang, Z., Xu, W., Yu, K.: Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991 (2015)

  25. Zhang, A., Lipton, Z.C., Li, M., et al.: Dive into Deep Learning. Unpublished draft, vol. 3, p. 319 (2019). Accessed

    Google Scholar 

  26. Graves, A., Mohamed, A., Hinton, G.: Speech recognition with deep recurrent neural networks. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6645–6649. IEEE (2013)

    Google Scholar 

  27. Hong, Z., You, N., Tan, J., et al.: Residual BiRNN based Seq2Seq model with transition probability matrix for online handwritten mathematical expression recognition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 635–640. IEEE (2019)

    Google Scholar 

  28. Vaswani, A., Shazeer, N., Parmar, N., et al.: Attention is all you need. In: Advances in Neural Information Processing Systems, pp. 5998–6008 (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Mathematics, Sun Yat-Sen University, Guangzhou, China

    Hong Lin & Jun Tan

Authors
  1. Hong Lin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Lin .

Editor information

Editors and Affiliations

  1. East China Normal University, Shanghai, China

    Yue Lu

  2. Paris Descartes University, Paris, France

    Nicole Vincent

  3. Hong Kong Baptist University, Kowloon, Hong Kong

    Pong Chi Yuen

  4. Sun Yat-sen University, Guangzhou, China

    Wei-Shi Zheng

  5. Polytechnique Montréal, Montreal, QC, Canada

    Farida Cheriet

  6. Concordia University, Montreal, QC, Canada

    Ching Y. Suen

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Lin, H., Tan, J. (2020). Application of Deep Learning in Handwritten Mathematical Expressions Recognition. In: Lu, Y., Vincent, N., Yuen, P.C., Zheng, WS., Cheriet, F., Suen, C.Y. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2020. Lecture Notes in Computer Science(), vol 12068. Springer, Cham. https://doi.org/10.1007/978-3-030-59830-3_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59830-3_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59829-7

  • Online ISBN: 978-3-030-59830-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation