Skip to main content
SpringerLink
Log in

Towards Making Flowchart Images Machine Interpretable

  • Conference paper
  • First Online:
Document Analysis and Recognition - ICDAR 2023 (ICDAR 2023)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14191))

Included in the following conference series:

  • 741 Accesses

Abstract

Computer programming textbooks and software documentations often contain flowcharts to illustrate the flow of an algorithm or procedure. Modern OCR engines often tag these flowcharts as graphics and ignore them in further processing. In this paper, we work towards making flowchart images machine-interpretable by converting them to executable Python codes. To this end, inspired by the recent success in natural language to code generation literature, we present a novel transformer-based framework, namely FloCo-T5. Our model is well-suited for this task, as it can effectively learn semantics, structure, and patterns of programming languages, which it leverages to generate syntactically correct code. We also used a task-specific pre-training objective to pre-train FloCo-T5 using a large number of logic-preserving augmented code samples. Further, to perform a rigorous study of this problem, we introduce the FloCo dataset that contains 11,884 flowchart images and their corresponding Python codes. Our experiments show promising results, and FloCo-T5 clearly outperforms related competitive baselines on code generation metrics. We make our dataset and implementation publicly available (https://vl2g.github.io/projects/floco).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 119.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 159.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://vl2g.github.io/projects/floco/.

  2. 2.

    https://pypi.org/project/pyflowchart/.

  3. 3.

    https://diagrams.mingrammer.com/.

References

  1. Easy OCR (2022). https://pypi.org/project/easyocr/1.6.2/

  2. Ahmad, W., Chakraborty, S., Ray, B., Chang, K.W.: Unified pre-training for program understanding and generation. In: Proceedings of NAACL-HLT (2021)

    Google Scholar 

  3. Akermi, I., Heinecke, J., Herledan, F.: Transformer based natural language generation for question-answering. In: Proceedings of the 13th International Conference on Natural Language Generation (2020)

    Google Scholar 

  4. Austin, J., et al.: Program synthesis with large language models. CoRR abs/2108.07732 (2021)

    Google Scholar 

  5. Awal, A.M., Feng, G., Mouchère, H., Viard-Gaudin, C.: Handwritten flowchart dataset (OHFCD). Document Recognition and Retrieval XVIII, January 2011, San Fransisco, United States, pp. 7874–78740A (2011). https://doi.org/10.1117/12.876624

  6. Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection. In: Proceedings of CVPR (2019)

    Google Scholar 

  7. Bradski, G.: The OpenCV library. Dr. Dobb’s J. Softw. Tools 120, 122–125 (2000)

    Google Scholar 

  8. Brown, T., et al.: Language models are few-shot learners. In: Proceedings of NeurIPS (2020)

    Google Scholar 

  9. Carton, C., Lemaitre, A., Coüasnon, B.: Fusion of statistical and structural information for flowchart recognition. In: Proceedings of ICDAR (2013)

    Google Scholar 

  10. Chaudhuri, A.: Flowchart and Algorithm Basics: The Art of Programming. Mercury Learning and Information (2020)

    Google Scholar 

  11. Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: ELECTRA: pre-training text encoders as discriminators rather than generators. In: Proceedings of ICLR (2020)

    Google Scholar 

  12. Cook, D.: Flowgorithm (2022). https://www.flowgorithm.org/

  13. Fang, J., Feng, Z., Cai, B.: DrawnNet: offline hand-drawn diagram recognition based on keypoint prediction of aggregating geometric characteristics. Entropy 24(3), 425 (2022)

    Article  Google Scholar 

  14. Feng, Z., et al.: CodeBERT: a pre-trained model for programming and natural languages. In: Findings of the ACL: EMNLP (2020)

    Google Scholar 

  15. Guo, D., et al.: GraphCodeBERT: pre-training code representations with data flow. In: Proceedings of ICLR (2020)

    Google Scholar 

  16. Herrera-Camara, J.I., Hammond, T.: Flow2Code: from hand-drawn flowcharts to code execution. In: Proceedings of the Symposium on Sketch-Based Interfaces and Modeling (2017)

    Google Scholar 

  17. Kenton, J.D.M.W.C., Toutanova, L.K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT (2019)

    Google Scholar 

  18. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) Proceedings of ICLR (2015)

    Google Scholar 

  19. Lewis, M., et al.: BART: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of ACL (2020)

    Google Scholar 

  20. Li, M., et al.: TrOCR: transformer-based optical character recognition with pre-trained models. arXiv preprint arXiv:2109.10282 (2021)

  21. Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019)

    Google Scholar 

  22. Lu, S., et al.: CodeXGLUE: a machine learning benchmark dataset for code understanding and generation. arXiv (2021)

    Google Scholar 

  23. Montellano, C.D.B., Garcia, C.O.F.C., Leija, R.O.C.: Recognition of handwritten flowcharts using convolutional neural networks. Int. J. Comput. Appl. (2022)

    Google Scholar 

  24. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of ACL (2002)

    Google Scholar 

  25. Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., et al.: Improving language understanding by generative pre-training. OpenAI (2018)

    Google Scholar 

  26. Raffel, C., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21, 1–67 (2020)

    MathSciNet  MATH  Google Scholar 

  27. Ren, S., et al.: CodeBLEU: a method for automatic evaluation of code synthesis. CoRR abs/2009.10297 (2020)

    Google Scholar 

  28. Schäfer, B., Stuckenschmidt, H.: Arrow R-CNN for flowchart recognition. In: Proceedings of ICDAR Workshop (2019)

    Google Scholar 

  29. Song, K., Tan, X., Qin, T., Lu, J., Liu, T.Y.: MASS: masked sequence to sequence pre-training for language generation. In: Proceedings of ICML (2019)

    Google Scholar 

  30. Supaartagorn, C.: Web application for automatic code generator using a structured flowchart. In: 2017 8th IEEE International Conference on Software Engineering and Service Science (ICSESS) (2017)

    Google Scholar 

  31. Tannert, S., Feighelstein, M., Bogojeska, J., Shtok, J., Staar, A.A.P., Karlinsky, A.S.J.K.L.: FlowchartQA: the first large-scale benchmark for reasoning over flowcharts. In: Document Intelligence Workshop @ KDD (2022)

    Google Scholar 

  32. Vaswani, A., et al.: Attention is all you need. In: Proceedings of NeurIPS (2017)

    Google Scholar 

  33. Wang, Y., Wang, W., Joty, S., Hoi, S.C.: CodeT5: identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In: Proceedings of EMNLP (2021)

    Google Scholar 

  34. Wolf, T., et al.: Transformers: state-of-the-art natural language processing. In: Proceedings of EMNLP: System Demonstrations (2020)

    Google Scholar 

  35. Wu, X.H., Qu, M.C., Liu, Z.Q., Li, J.Z., et al.: Research and application of code automatic generation algorithm based on structured flowchart. J. Softw. Eng. Appl. 4, 534–545 (2011)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partly supported by MeitY, Govt. of India (project number: S/MeitY/AM/20210114). Yogesh Kumar is supported by a UGC fellowship.

Author information

Authors and Affiliations

  1. Vision, Language and Learning Group (VL2G), IIT Jodhpur, Jodhpur, India

    Shreya Shukla, Prajwal Gatti, Yogesh Kumar, Vikash Yadav & Anand Mishra

Authors
  1. Shreya Shukla
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Prajwal Gatti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yogesh Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Vikash Yadav
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Anand Mishra
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Yogesh Kumar or Anand Mishra .

Editor information

Editors and Affiliations

  1. TU Dortmund University, Dortmund, Germany

    Gernot A. Fink

  2. Adobe, College Park, MN, USA

    Rajiv Jain

  3. Osaka Metropolitan University, Osaka, Japan

    Koichi Kise

  4. Rochester Institute of Technology, Rochester, NY, USA

    Richard Zanibbi

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Shukla, S., Gatti, P., Kumar, Y., Yadav, V., Mishra, A. (2023). Towards Making Flowchart Images Machine Interpretable. In: Fink, G.A., Jain, R., Kise, K., Zanibbi, R. (eds) Document Analysis and Recognition - ICDAR 2023. ICDAR 2023. Lecture Notes in Computer Science, vol 14191. Springer, Cham. https://doi.org/10.1007/978-3-031-41734-4_31

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-41734-4_31

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-41733-7

  • Online ISBN: 978-3-031-41734-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation