Skip to main content
SpringerLink
Log in

DiagramNet: Hand-Drawn Diagram Recognition Using Visual Arrow-Relation Detection

  • Conference paper
  • First Online:
Document Analysis and Recognition – ICDAR 2021 (ICDAR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12821))

Included in the following conference series:

Abstract

Offline hand-drawn diagram recognition is concerned with digitizing diagrams sketched on paper or whiteboard to enable further editing. Our proposed DiagramNet model addresses this recognition problem. We combine shape detection and visual arrow-relation detection to recognize arrows between shape pairs. A shape degree predictor predicts the number of in- and outgoing arrows in each direction. An optimization procedure uses the generated predictions to find the set of globally coherent arrows. Previous offline methods focus on clean images from online datasets with nicely layouted diagrams. We show that our approach is effective in the domain of camera-captured diagrams with chaotic layouts and various recognition challenges such as crossing arrows. To that end, we introduce a new dataset of hand-drawn business process diagrams that originate from textual process modeling tasks. Our evaluation shows that DiagramNet considerably outperforms prior state-of-the-art in this challenging domain.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Awal, A.M., Feng, G., Mouchère, H., Viard-Gaudin, C.: First experiments on a new online handwritten flowchart database. In: Document Recognition and Retrieval XVIII, p. 78740A (January 2011)

    Google Scholar 

  2. Bresler, M., Phan, T.V., Průša, D., Nakagawa, M., Hlaváč, V.: Recognition system for on-line sketched diagrams. In: ICFHR, pp. 563–568 (September 2014)

    Google Scholar 

  3. Bresler, M., Průša, D., Hlaváč, V.: Recognizing off-line flowcharts by reconstructing strokes and using on-line recognition techniques. In: ICFHR, pp. 48–53 (October 2016)

    Google Scholar 

  4. Bresler, M., Průša, D., Hlaváč, V.: Online recognition of sketched arrow-connected diagrams. Int. J. Doc. Anal. Recogn. (IJDAR) 19(3), 253–267 (2016). https://doi.org/10.1007/s10032-016-0269-z

    Article  Google Scholar 

  5. Carton, C., Lemaitre, A., Coüasnon, B.: Fusion of statistical and structural information for flowchart recognition. In: ICDAR, pp. 1210–1214 (August 2013)

    Google Scholar 

  6. Cherubini, M., Venolia, G., DeLine, R., Ko, A.J.: Let’s go to the whiteboard: how and why software developers use drawings. In: CHI, pp. 557–566 (2007)

    Google Scholar 

  7. Davis, B., Morse, B., Cohen, S., Price, B., Tensmeyer, C.: Deep visual template-free form parsing. In: ICDAR, pp. 134–141 (September 2019)

    Google Scholar 

  8. Doermann, D., Liang, J., Li, H.: Progress in camera-based document image analysis. In: ICDAR, pp. 606–616, vol. 1 (August 2003)

    Google Scholar 

  9. Dumas, M., Rosa, M.L., Mendling, J., Reijers, H.: Fundamentals of Business Process Management, 2nd edn. Springer, Heidelberg (2018). https://doi.org/10.1007/978-3-662-56509-4

  10. Gervais, P., Deselaers, T., Aksan, E., Hilliges, O.: The DIDI dataset: digital ink diagram data. arXiv:2002.09303 [cs] (February 2020)

  11. He, K., Gkioxari, G., Dollar, P., Girshick, R.: Mask R-CNN. In: ICCV, pp. 2961–2969 (2017)

    Google Scholar 

  12. Howard, A.G., et al.: MobileNets: efficient convolutional neural networks for mobile vision applications. arXiv:1704.04861 [cs] (April 2017)

  13. Julca-Aguilar, F.D., Hirata, N.S.T.: Symbol detection in online handwritten graphics using faster R-CNN. In: DAS, pp. 151–156 (April 2018)

    Google Scholar 

  14. Julca-Aguilar, F., Mouchère, H., Viard-Gaudin, C., Hirata, N.S.T.: A general framework for the recognition of online handwritten graphics. Int. J. Doc. Anal. Recogn. (IJDAR) 23(2), 143–160 (2020). https://doi.org/10.1007/s10032-019-00349-6

    Article  Google Scholar 

  15. Lemaitre, A., Mouchère, H., Camillerapp, J., Coüasnon, B.: Interest of syntactic knowledge for on-line flowchart recognition. In: Graphics Recognition. New Trends and Challenges, pp. 89–98 (2013)

    Google Scholar 

  16. Lin, T.Y., Dollar, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection. In: CVPR, pp. 2117–2125 (2017)

    Google Scholar 

  17. Pinggera, J., et al.: Styles in business process modeling: an exploration and a model. Softw. Syst. Model. 14(3), 1055–1080 (2015)

    Article  Google Scholar 

  18. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. In: NeurIPS, pp. 91–99 (2015)

    Google Scholar 

  19. Schäfer, B., Keuper, M., Stuckenschmidt, H.: Arrow R-CNN for handwritten diagram recognition. Int. J. Doc. Anal. Recogn. (IJDAR) 24(1), 3–17 (2021). https://doi.org/10.1007/s10032-020-00361-1

    Article  Google Scholar 

  20. Schäfer, B., Stuckenschmidt, H.: Arrow R-CNN for flowchart recognition. In: ICDARW, p. 7 (September 2019)

    Google Scholar 

  21. Wang, C., Mouchère, H., Viard-Gaudin, C., Jin, L.: Combined segmentation and recognition of online handwritten diagrams with high order Markov random field. In: ICFHR, pp. 252–257 (October 2016)

    Google Scholar 

  22. Wang, C., Mouchère, H., Lemaitre, A., Viard-Gaudin, C.: Online flowchart understanding by combining max-margin Markov random field with grammatical analysis. Int. J. Doc. Anal. Recogn. (IJDAR) 20(2), 123–136 (2017). https://doi.org/10.1007/s10032-017-0284-8

    Article  Google Scholar 

  23. Wu, J., Wang, C., Zhang, L., Rui, Y.: Offline sketch parsing via shapeness estimation. In: IJCAI (June 2015)

    Google Scholar 

  24. Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2 (2019). https://github.com/facebookresearch/detectron2

  25. Xu, D., Zhu, Y., Choy, C.B., Fei-Fei, L.: Scene graph generation by iterative message passing. In: CVPR, pp. 5410–5419 (2017)

    Google Scholar 

  26. Yang, J., Lu, J., Lee, S., Batra, D., Parikh, D.: Graph R-CNN for scene graph generation. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11205, pp. 690–706. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01246-5_41

    Chapter  Google Scholar 

  27. Yun, X.-L., Zhang, Y.-M., Ye, J.-Y., Liu, C.-L.: Online handwritten diagram recognition with graph attention networks. In: Zhao, Y., Barnes, N., Chen, B., Westermann, R., Kong, X., Lin, C. (eds.) ICIG 2019. LNCS, vol. 11901, pp. 232–244. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-34120-6_19

    Chapter  Google Scholar 

  28. Zellers, R., Yatskar, M., Thomson, S., Choi, Y.: Neural Motifs: scene graph parsing with global context. In: CVPR, pp. 5831–5840 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Intelligent Robotic Process Automation, SAP SE, Walldorf, Germany

    Bernhard Schäfer

  2. Data and Web Science Group, University of Mannheim, Mannheim, Germany

    Bernhard Schäfer & Heiner Stuckenschmidt

Authors
  1. Bernhard Schäfer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Heiner Stuckenschmidt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bernhard Schäfer .

Editor information

Editors and Affiliations

  1. Universitat Autònoma de Barcelona, Barcelona, Spain

    Josep Lladós

  2. Lehigh University, Bethlehem, PA, USA

    Daniel Lopresti

  3. Kyushu University, Fukuoka-shi, Japan

    Seiichi Uchida

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Schäfer, B., Stuckenschmidt, H. (2021). DiagramNet: Hand-Drawn Diagram Recognition Using Visual Arrow-Relation Detection. In: Lladós, J., Lopresti, D., Uchida, S. (eds) Document Analysis and Recognition – ICDAR 2021. ICDAR 2021. Lecture Notes in Computer Science(), vol 12821. Springer, Cham. https://doi.org/10.1007/978-3-030-86549-8_39

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-86549-8_39

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-86548-1

  • Online ISBN: 978-3-030-86549-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation