Skip to main content
Springer Nature Link
Log in

Handwritten Mathematical Expression Recognition: A Survey

  • Conference paper
  • First Online:
Pattern Recognition and Artificial Intelligence (ICPRAI 2020)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12068))

  • 1791 Accesses

Abstract

While the handwritten character recognition has reached a point of maturity, the recognition of handwritten mathematics is still a challenging problem. The problem usually consists of three major parts: strokes segmentation, single symbol recognition and structural analysis. In this paper, we present a review on handwritten mathematical expression recognition to show how the recognition technique is developed. In particular, we put emphasis on the differences between systems.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Anderson, R.H.: Syntax-directed recognition of hand-printed two-dimensional mathematics. In: Symposium on Interactive Systems for Experimental Applied Mathematics: Proceedings of the Association for Computing Machinery Inc., Symposium. ACM, pp. 436–459 (1967)

    Google Scholar 

  2. Belaid, A., Haton, J.-P.: A syntactic approach for handwritten mathematical formula recognition. IEEE Trans. Pattern Anal. Mach. Intell. 1, 105–111 (1984)

    Article  Google Scholar 

  3. Simistira, F., Katsouros, V., Carayannis, G.: Recognition of online handwritten mathematical formulas using probabilistic SVMs and stochastic context free grammars. Pattern Recogn. Lett. 53, 85–92 (2015)

    Article  Google Scholar 

  4. Chan, K.-F., Yeung, D.-Y.: Mathematical expression recognition: a survey. Int. J. Doc. Anal. Recogn. 3(1), 3–15 (2000)

    Article  MathSciNet  Google Scholar 

  5. Lamport, L.: LATEX: A Document Preparation System: User’s Guide and Reference Manual. Addison-Wesley, Boston (1994)

    MATH  Google Scholar 

  6. Le, A.D., Van Phan, T., Nakagawa, M.: A system for recognizing online handwritten mathematical expressions and improvement of structure analysis. In: 11th International Workshop on Document Analysis Systems (DAS 2014). IEEE (2014)

    Google Scholar 

  7. Mouchère, H., Zanibbi, R., Garain, U., et al.: Advancing the state of the art for handwritten math recognition: the CROHME competitions, 2011–2014. Int. J. Doc. Anal. Recogn. (IJDAR) 19(2), 173–189 (2016)

    Article  Google Scholar 

  8. Zanibbi, R., Blostein, D., Cordy, J.R.: Recognizing mathematical expressions using tree transformation. IEEE Trans. Pattern Anal. Mach. Intell. 24(11), 1455–1467 (2002)

    Article  Google Scholar 

  9. Álvaro, F., Sánchez, J.-A., Bened’ı, J.-M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35, 58–67 (2014)

    Article  Google Scholar 

  10. Chou, P.A.: Recognition of equations using a two-dimensional stochastic context-free grammar. Proc. SPIE 1199, 852–863 (1989)

    Article  Google Scholar 

  11. Nakayama, Y.: Mathematical formula editor for CAI. ACM SIGCHI Bull. 20(SI), 387–392 (1989)

    Article  Google Scholar 

  12. Fateman, R.J., Tokuyasu, T., Berman, B.P., et al.: Optical character recognition and parsing of typeset mathematics1. J. Vis. Commun. Image Representation 7(1), 2–15 (1996)

    Article  Google Scholar 

  13. Belaid, A., Haton, J.P.: A syntactic approach for handwritten mathematical formula recognition. IEEE Trans. Pattern Anal. Mach. Intell. 1, 105–111 (1984)

    Article  Google Scholar 

  14. Chan, K.F., Yeung, D.Y.: Recognizing on-line handwritten alphanumeric characters through flexible structural matching. Pattern Recogn. 32(7), 1099–1114 (1999)

    Article  Google Scholar 

  15. Chen, L.: A system for on-line recognition of handwritten mathematical expressions. Comput. Process. Chin. Orient. Lang. 6(1), 19–39 (1992)

    MathSciNet  Google Scholar 

  16. Lee, H.J., Lee, M.C.: Understanding mathematical expressions in a printed document. In: Proceedings of 2nd International Conference on Document Analysis and Recognition (ICDAR 1993), pp. 502–505. IEEE (1993)

    Google Scholar 

  17. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)

    Article  Google Scholar 

  18. Koschinski, M., Winkler, H.J., Lang, M.: Segmentation and recognition of symbols within handwritten mathematical expressions. In: 1995 International Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. 2439–2442. IEEE (1995)

    Google Scholar 

  19. MacLean, S., Labahn, G.: A Bayesian model for recognizing handwritten mathematical expressions. Pattern Recogn. 48(8), 2433–2445 (2015)

    Article  Google Scholar 

  20. Chappelier, J.C., Rajman, M.: A generalized CYK algorithm for parsing stochastic CFG. TAPD 98(133–137), 5 (1998)

    Google Scholar 

  21. Rhee, T.H., Kim, J.H.: Efficient search strategy in structural analysis for handwritten mathematical expression recognition. Pattern Recogn. 42(12), 3192–3201 (2009)

    Article  Google Scholar 

  22. Álvaro, F., Sánchez, J.A., Benedí, J.M.: Recognition of on-line handwritten mathematical expressions using 2D stochastic context-free grammars and hidden Markov models. Pattern Recogn. Lett. 35, 58–67 (2014)

    Article  Google Scholar 

  23. Sutskever, I., Vinyals, O., Le, Q.V.: Sequence to sequence learning with neural networks. In: Advances in Neural Information Processing Systems, pp. 3104–3112 (2014)

    Google Scholar 

  24. Luong, M.-T., Sutskever, I., Le, Q.V., Vinyals, O., Zaremba, W.: Addressing the rare word problem in neural machine translation, arXiv preprint arXiv: 1410.8206 (2014)

  25. Bahdanau, D., Chorowski, J., Serdyuk, D., Brakel, P., Bengio, Y.: End-to-end attention-based large vocabulary speech recognition. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4945–4949. IEEE (2016)

    Google Scholar 

  26. Chan, W., Jaitly, N., Le, Q.V., Vinyals, O.: Listen, attend and spell, arXiv preprint arXiv: 1508.01211 (2015)

  27. Zhang, J., Du, J., Zhang, S., et al.: Watch, attend and parse: an end-to-end neural network based approach to handwritten mathematical expression recognition. Pattern Recogn. 71, 196–206 (2017)

    Article  Google Scholar 

  28. Zhang, J., Du, J., Dai, L.: Track, attend, and parse (TAP): an end-to-end framework for online handwritten mathematical expression recognition. IEEE Trans. Multimed. 21(1), 221–233 (2018)

    Article  Google Scholar 

  29. Zhang, T., Mouchère, H., Viard-Gaudin, C.: A tree-BLSTM-based recognition system for online handwritten mathematical expressions. Neural Comput. Appl. 1–20 (2018)

    Google Scholar 

  30. Beal, M.J., Ghahramani, Z., Rasmussen, C.E.: The infinite hidden Markov model. In: Advances in neural information processing systems, pp. 577–584 (2002)

    Google Scholar 

  31. Saidane, Z., Garcia, C.: Automatic scene text recognition using a convolutional neural network. In: Workshop on Camera-Based Document Analysis and Recognition, vol. 1 (2007)

    Google Scholar 

  32. Kim, G., Govindaraju, V., Srihari, S.N.: An architecture for handwritten text recognition systems. Int. J. Doc. Anal. Recogn. 2(1), 37–44 (1999)

    Article  Google Scholar 

  33. Espana-Boquera, S., Castro-Bleda, M.J., Gorbe-Moya, J., et al.: Improving offline handwritten text recognition with hybrid HMM/ANN models. IEEE Trans. Pattern Anal. Mach. Intell. 33(4), 767–779 (2010)

    Article  Google Scholar 

  34. Bottou, L.: Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010. Physica-Verlag HD, pp. 177–186 (2010)

    Google Scholar 

  35. Bottou, L.: Stochastic gradient descent tricks. In: Montavon, G., Orr, G.B., Müller, K.-R. (eds.) Neural Networks: Tricks of the Trade. LNCS, vol. 7700, pp. 421–436. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_25

    Chapter  Google Scholar 

  36. Tillmann, C., Ney, H.: Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Comput. Linguist. 29(1), 97–133 (2003)

    Article  Google Scholar 

  37. Mouchere, H., Viard-Gaudin, C., Zanibbi, R., et al.: ICFHR 2014 competition on recognition of on-line handwritten mathematical expressions (CROHME 2014). In: 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 791–796. IEEE (2014)

    Google Scholar 

  38. Mouchère, H., Viard-Gaudin, C., Zanibbi, R., et al.: ICFHR2016 CROHME: competition on recognition of online handwritten mathematical expressions. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 607–612. IEEE (2016)

    Google Scholar 

  39. Cho, K., Van Merriënboer, B., Gulcehre, C., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)

  40. Mouchere, H., Viard-Gaudin, C., Zanibbi, R., et al.: ICDAR 2013 CROHME: third international competition on recognition of online handwritten mathematical expressions. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 1428–1432. IEEE (2013)

    Google Scholar 

  41. Dam, H.K., Pham, T., Ng, S.W., et al.: A deep tree-based model for software defect prediction. arXiv preprint arXiv:1802.00921 (2018)

  42. Sankaran, N., Jawahar, C.V.: Recognition of printed Devanagari text using BLSTM Neural Network. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), pp. 322–325. IEEE (2012)

    Google Scholar 

  43. Hannun, A.: Sequence modeling with CTC. Distill 2(11), e8 (2017)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported by the Guangdong Provincial Government of China through the “Computational Science Innovative Research Team” program and the Guangdong Province Key Laboratory of Computational Science at the Sun Yat-sen University ,and the National Science Foundation of China (grant no. 11471012).

Author information

Authors and Affiliations

  1. School of Mathematics, Sun Yat-sen University, Guangzhou, China

    Fukeng He, Jun Tan & Ning Bi

Authors
  1. Fukeng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jun Tan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ning Bi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fukeng He .

Editor information

Editors and Affiliations

  1. East China Normal University, Shanghai, China

    Yue Lu

  2. Paris Descartes University, Paris, France

    Nicole Vincent

  3. Hong Kong Baptist University, Kowloon, Hong Kong

    Pong Chi Yuen

  4. Sun Yat-sen University, Guangzhou, China

    Wei-Shi Zheng

  5. Polytechnique Montréal, Montreal, QC, Canada

    Farida Cheriet

  6. Concordia University, Montreal, QC, Canada

    Ching Y. Suen

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

He, F., Tan, J., Bi, N. (2020). Handwritten Mathematical Expression Recognition: A Survey. In: Lu, Y., Vincent, N., Yuen, P.C., Zheng, WS., Cheriet, F., Suen, C.Y. (eds) Pattern Recognition and Artificial Intelligence. ICPRAI 2020. Lecture Notes in Computer Science(), vol 12068. Springer, Cham. https://doi.org/10.1007/978-3-030-59830-3_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-59830-3_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-59829-7

  • Online ISBN: 978-3-030-59830-3

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics