Skip to main content
Log in

Contextual Stroke Classification in Online Handwritten Documents with Edge Graph Attention Networks

SN Computer Science Aims and scope Submit manuscript

Abstract

The task of grouping strokes into different categories is an essential processing step in the automatic analysis of online handwritten documents. The technical challenge originates from the variation of the handwriting style, content heterogeneity and lack of prior layout knowledge. In this work, we propose the edge graph attention network (EGAT) to address the stroke classification problem. In this framework, the stroke classification problem is formulated as a node classification problem in a relational graph, which is constructed based on the temporal and spatial relationship of strokes. Then distributed node and edge features for classification are learned by stacking of multiple edge graph attention layers, in which various attention mechanisms are exploited to aggregate information between neighborhood nodes. In the task of text/nontext classification, the proposed model achieves accuracies 98.65% and 98.90% on the IAMOnDo and Kondate datasets, respectively. In the task of multi-class classification, the achieved accuracies are 95.81%, 97.36% and 99.05% on the IAMOnDo, FC and FA datasets, respectively. In addition, we conduct ablation experiments to quantitatively and qualitatively evaluate the key modules of our model.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Awal AM, Feng G, Mouchere H, Viard-Gaudin C. First experiments on a new online handwritten flowchart database. In: Document Recognition and Retrieval, vol. 7874, p. 78740A. International Society for Optics and Photonics 2011.

  2. Bishop CM, Svensen M, Hinton GE. Distinguishing text from graphics in on-line handwritten ink. In: International Conference on Frontiers in Handwriting Recognition, 2004;142–147.

  3. Bresler M, Prusa D, Hlavác V. Detection of arrows in on-line sketched diagrams using relative stroke positioning. Winter Conf Appl Comput Vis. 2015;10:610–617.

    Google Scholar 

  4. Bresler M, Prusa D, Hlavác V. Online recognition of sketched arrow-connected diagrams. Int J Doc Anal Recogn. 2016;19(3):253–267.

    Article  Google Scholar 

  5. Bresler M, Van Phan T, Prusa D, Nakagawa M, Hlavác V. Recognition system for on-line sketched diagrams. In: International Conference on Frontiers in Handwriting Recognition, 2014;563–568 .

  6. Carton C, Lemaitre A, Coüasnon B. Fusion of statistical and structural information for flowchart recognition. In: International Conference on Document Analysis and Recognition, 2013;1210–1214.

  7. Defferrard M, Bresson X, Vandergheynst P. Convolutional neural networks on graphs with fast localized spectral filtering. In: Advances in Neural Information Processing Systems, 2016;3844–3852.

  8. Delaye A, Liu CL. Contextual text/non-text stroke classification in online handwritten notes with conditional random fields. Pattern Recogn. 2014;47(3):959–968.

    Article  Google Scholar 

  9. Delaye A, Liu CL. Multi-class segmentation of free-form online documents with tree conditional random fields. Int J Doc Anal Recogn. 2014;17(4):313–329.

    Article  Google Scholar 

  10. Glorot X, Bengio Y. Understanding the difficulty of training deep feedforward neural networks. In: International Conference on Artificial Intelligence and Statistics, 2010;249–256.

  11. Gong L, Cheng Q. Exploiting edge features for graph neural networks. In: Conference on Computer Vision and Pattern Recognition, 2019;9211–9219.

  12. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9(8):1735–1780.

    Article  Google Scholar 

  13. Indermühle E. Analysis of digital ink in electronic documents. Ph.D. thesis, University of Bern 2012.

  14. Indermühle E, Frinken V, Bunke H. Mode detection in online handwritten documents using blstm neural networks. In: International Conference on Frontiers in Handwriting Recognition, 2012;302–307.

  15. Indermühle E, Liwicki M, Bunke H. Iamondo-database: an online handwritten document database with non-uniform contents. In: International Workshop on Document Analysis Systems, 2010;97–104.

  16. Jain AK, Namboodiri AM, Subrahmonia J. Structure in on-line documents. In: International Conference on Document Analysis and Recognition, 2001;844–848.

  17. Kingma D, Ba J. Adam: A method for stochastic optimization. In: International Conference on Learning Representation 2015.

  18. Kipf T, Welling M. Semi-supervised classification with graph convolutional networks. In: International Conference on Learning Representations 2017.

  19. Koller D, Friedman N. Probabilistic graphical models: principles and techniques. New York: MIT press; 2009.

    MATH  Google Scholar 

  20. Lafferty J, McCallum A, Pereira FC. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In: International Conference on Machine Learning, 2001;282–289.

  21. Lemaitre A, Mouchère H, Camillerapp J, Coüasnon B. Interest of syntactic knowledge for on-line flowchart recognition. In: International Workshop on Graphics Recognition, pp. 89–98. Springer, 2011.

  22. Mochida K, Nakagawa M. Separating figures, mathematical formulas and japanese text from free handwriting in mixed online documents. Int J Pattern Recognit Artif Intell. 2004;18(07):1173–1187.

    Article  Google Scholar 

  23. Paszke A, Gross S, Massa F, Lerer A, Bradbury J, Chanan G, Killeen T, Lin Z, Gimelshein N, Antiga L, et al. Pytorch: An imperative style, high-performance deep learning library. In: Advances in Neural Information Processing Systems, 2019, pp. 8024–8035.

  24. Peterson EJ, Stahovich TF, Doi E, Alvarado C. Grouping strokes into shapes in hand-drawn diagrams. In: AAAI Conference on Artificial Intelligence, 2010; 974–979.

  25. Phan TV, Nakagawa M. Combination of global and local contexts for text/non-text classification in heterogeneous online handwritten documents. Pattern Recogn. 2016;51:112–124.

    Article  Google Scholar 

  26. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser Ł, Polosukhin I. Attention is all you need. In: Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.

  27. Veličković P, Cucurull G, Casanova A, Romero A, Lio P, Bengio Y. Graph attention networks. In: International Conference on Learning Representation 2018.

  28. Wang C, Mouchère H, Lemaitre A, Viard-Gaudin C. Online flowchart understanding by combining max-margin markov random field with grammatical analysis. Int J Doc Anal Recogn. 2017;20(2):123–136.

    Article  Google Scholar 

  29. Wang C, Mouchere H, Viard-Gaudin C, Jin L. Combined segmentation and recognition of online handwritten diagrams with high order markov random field. In: International Conference on Frontiers in Handwriting Recognition, 2016, pp. 252–257.

  30. Wang M, Yu L, Zheng D, Gan Q, Gai Y, Ye Z, Li M, Zhou J, Huang Q, Ma C et al. Deep graph library: Towards efficient and scalable deep learning on graphs. In: International Conference on Learning Representation 2019.

  31. Weber M, Liwicki M, Schelske YT, Schoelzel C, Strauß F, Dengel A. Mcs for online mode detection: Evaluation on pen-enabled multi-touch interfaces. In: International Conference on Document Analysis and Recognition, 2011, pp. 957–961.

  32. Wu J, Wang C, Zhang L, Rui Y. Offline sketch parsing via shapeness estimation. In: International Joint Conference on Artificial Intelligence, 2015, pp. 1200–1206.

  33. Ye JY, Zhang YM, Liu CL. Joint training of conditional random fields and neural networks for stroke classification in online handwritten documents. In: International Conference on Pattern Recognition, 2016, pp. 3264–3269.

  34. Ye JY, Zhang YM, Yang Q, Liu CL. Contextual stroke classification in online handwritten documents with graph attention networks. In: International Conference on Document Analysis and Recognition, 2019, pp. 993–998.

  35. Zhou XD, Liu CL. Text/non-text ink stroke classification in japanese handwriting based on markov random fields. Int Conf Document Anal Recogn. 2007;1:377–381.

    Google Scholar 

Download references

Acknowledgements

This work has been supported in part by the National Key Research and Development Program Grant 2018YFB1005000 and the National Natural Science Foundation of China (NSFC) Grants 61773376 and 61721004.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Cheng-Lin Liu.

Ethics declarations

Conflict of interest

This work has no conflict of interest with any personal or funding parties.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This article is part of the topical collection "Document Analysis and Recognition" guest edited by Michael Blumenstein, Seiichi Uchida and Cheng-Lin Liu.

Appendix

Appendix

Dataset Statistics

This section is supplementary to section 4.1 and presents the statistics of each dataset.

Table 10 Statistics of IAMOnDo, Kondate, FC and FA datasets: number of documents, strokes and strokes per category

Hyperparameters

This section is supplementary to Section 3.4 and presents the chosen hyperparameters for all experiments. For all edge attention layers, the hyperparameters of each layer (\((C',D',K)\) are kept the same. We tune the hyperparameters on the validation set by random search.

Table 11 Hyperparameters for all experiments

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ye, JY., Zhang, YM., Yang, Q. et al. Contextual Stroke Classification in Online Handwritten Documents with Edge Graph Attention Networks. SN COMPUT. SCI. 1, 163 (2020). https://doi.org/10.1007/s42979-020-00177-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s42979-020-00177-0

Navigation