Skip to main content
SpringerLink
Log in

Matching of hand-drawn flowchart, pseudocode, and english description using transfer learning

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

An algorithm plays an important role when solving a problem. It is challenging to comprehend for computer novices or machines. Therefore, a textual explanation is provided to illustrate the algorithm. To understand an algorithm, a method needs to be devised to find or generate the corresponding text description and vice versa. This paper matches an algorithm in a variety of forms, such as pseudocode and hand-drawn flowchart, with the illustrative text written in English to facilitate a thorough understanding of the algorithm. The experiment includes a proposed set of rules for generating pseudocode from a hand-drawn flowchart and a proposed S-DistilBERT-based transfer learning method to determine the similarity match score between multiple forms of algorithm and text description. Basic block and line identification, as well as OCR-ization, are used to characterize the hand-drawn flowcharts. The experimental result show that we can generate the equivalent pseudocode in 85% cases, and our fine-tuned S-DistilBERT model can accommodate the matching text for the existing pseudocode with 75.59% and the generated pseudocode with 74.57% accuracy. We also find the appropriate description from an algorithm in the top five matches in 30 out of 50 cases. The rules are found to be adequate for non-recursive flowcharts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Algorithm 1
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Data Availability

The dataset used in this paper is available in the Google drive, https://drive.google.com/drive/folders/1mQlqAW4n6hD13apLAR-l93vsCBmyPsYn?usp=sharing.

References

  1. Alzubi J, Jain R, Kathuria A, Khandelwal A, Saxena A, Singh A (2020) Paraphrase identification using collaborative adversarial networks. J Intell Fuzzy Syst: Appl Eng Technol 39(12020):1021–1032. https://doi.org/10.3233/JIFS-191933

    Article  Google Scholar 

  2. Alzubi J, Jain R, Nagrath P, Satapathy S, Taneja S, Gupta P (2020) Deep image captioning using an ensemble of CNN and LSTM based deep neural networks, J Intell Fuzzy Syst, https://doi.org/10.3233/JIFS-189415https://doi.org/10.3233/JIFS-189415

  3. Bhowmick P, Bhattacharya B B (2007) Fast polygonal approximation of digital curves using relaxed straightness properties. IEEE Trans Pattern Anal Mach Intell 29(9):1590–1602

    Article  Google Scholar 

  4. Bourbakis GN, Rematska G, Mertoguno S (2021) Deep understanding of technical documents: part II. Auto Extract Pseudocode Int J Artif Intell Tools. World Scientific Publishing Company vol 30(3), https://doi.org/10.1142/S0218213021500160https://doi.org/10.1142/S0218213021500160

  5. Cer D, Yang Y, Kong S, Hua N, Limtiaco N, John R, Constant N, Guajardo-Cespedes M, Yuan S, Tar C (2018) Universal sentence encoder, arXiv:1803.11175

  6. Chakraborty S, Paul S, Ahsan SMM (2020) A novel approach to rapidly generate document from hand drawn flowcharts. IEEE Region 10 Symposium (TENSYMP), https://doi.org/10.1109/TENSYMP50017.2020.9231033https://doi.org/10.1109/TENSYMP50017.2020.9231033

  7. Chen X, Zitnick LC (2014) Learning a recurrent visual representation for image caption generation, arXiv:1411.5654

  8. Cho K, Merrienboer VB, Gulcehre C, Bahdanau D, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: EMNLP, pp 1724–1734

  9. Chung K L, Liao P H, Chang J M (2008) Novel efficient two-pass algorithm for closed polygonal approximation based on LISE and curvature constraint criteria. J Visual Comun Image Represent 19(4):219–230

    Article  Google Scholar 

  10. Climer S., Bhatia S. K. (2003) Local lines: a linear time line detector. Pattern Recogn Lett 24(14):2291–2300

    Article  MATH  Google Scholar 

  11. Devlin JM, Chang M, Lee K, Toutanova K (2018) BERT: pre-training of deep bidirectional transformers for language understanding, arXiv:1810.04805

  12. Dunham JG (1986) Optimum uniform piecewise linear approximation of planar curves. IEEE Trans Patt-Anal Mach Intell 8(1):67–75

    Article  Google Scholar 

  13. Liu H, Latecki LJ, Liu W (2008) A unified curvature definition for regular, polygonal, and digital planar curves. Int J Comput Vis 80(1):104–124

    Article  MATH  Google Scholar 

  14. Loshchilov I, Hutter F (2019) Decoupled weight decay regularization, international conference on learning representations(ICLR) arXiv:1711.05101v3

  15. Masood A (2008) Dominant point detection by reverse polygonization of digital curves. Image Vis Comput 26(5):702–715

    Article  Google Scholar 

  16. Melkman A, O’Rourke J (1988) On polygonal chain approximation, computational morphology. In: Toussaint GT (ed) North-Holland Amsterdam, Netherlands, p 87-95

  17. Miyao H, Maruyama R (2012) On-Line handwritten flowchart recognition, beautification and editing system. In: 2012 international conference on frontiers in handwriting recognition. IEEE, https://doi.org/10.1109/ICFHR.2012.250https://doi.org/10.1109/ICFHR.2012.250

  18. Mueller J, Thyagarajan A (2016) Siamese recur-rent architectures for learning sentence similarity. In: Proc. of the Thirtieth AAAI Conference on Artificial Intelligence. (AAAI-16)), pp 2786–2792

  19. Mutinda WF, Yada S, Wakamiya S, Aramaki E (2021) Semantic textual similarity in japanese clinical domain texts using BERT. Methods Inf Med 2021; pp e56-e64, https://doi.org/10.1055/s-0041-1731390

  20. Mvs A, Alzubi J, Gheisari M, Rahimi M, Mohan S, Abbasi A, Nabipour N (2021) Artificial neural networks training algorithm integrating invasive weed optimization with differential evolutionary model. J Ambient Intell Humanized Comput, https://doi.org/10.1007/s12652-020-02623-6

  21. Neumann R, Teisseron G (2002) Extraction of dominant points by estimation of the contour fluctuations. Pattern Recogn 35(7):1447–1462

    Article  MATH  Google Scholar 

  22. Peace-text-scanner (2021) https://peace-text-scanner.en.aptoide.com/app# Last accessed: 25 June 2021

  23. Pennington J, Socher R, Manning DC (2014) GloVe: global vectors for word representation. In: Empirical methods in natural language processing (EMNLP), pp 1532–1543

  24. Pontes L E, Huet S, Linhares CA, Torres-Moreno J (2018) Predicting the semantic textual similarity with siamese cnn and lstm. Association for Computational Linguistics. Actes de la Conference Taln, volume 1 - Articles longs articles courts deTALN

  25. Ray BK, Ray KS (1992) : an algorithm for detection of dominant points and polygonal approximation of digitized curves. Pattern Recogn Lett 13 (12):849–856

    Article  Google Scholar 

  26. Reimers N, Gurevych I (2019) Sentence-BERT: sentence embeddings using Siamese BERT-networks, Association for Computational Linguistics. In: Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pp 3982–3992

  27. Sanh V, Debut L, Chaumond J, Wolf T (2020) DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter, EMC\(\hat {2}\): 5th Edition Co-located with NeurIPS’19, arXiv:1910.01108

  28. Sauvola JJ, Pietik M (2000) äinen, adaptive document image binarization. Pattern Recogn 33(2):225–236

    Article  Google Scholar 

  29. Teh CH, Chin RT (1989) On the detection of dominant points on digital curves. IEEE Trans Pattern Anal Mach Intell 2(8):859–872

    Article  Google Scholar 

  30. Venugopalan S, Xu H, Donahue J, Rohrbach M, Mooney R, Saenko K (2014) Translating videos to natural language using deep recurrent neural networks, arXiv:1412.4729

  31. Wael HG, Aly AF (2013) A survey of text similarity approaches. International Journal of Computer Applications. pp. 0975–8887 68(13):13–18

    Google Scholar 

  32. Wall K, Danielsson P E (1984) A fast sequential method for polygonal approximation of digitized curves. Comput Vision Graphics Image Process 28(3):220–227

    Article  Google Scholar 

  33. Wang C, Mouchere H, Lemaitre A, ViardGaudin C (2017) Online flowchart understanding by combining max-margin Markov random field with grammatical analysis. Springer, Int J Doc Anal Recognit 20(2):123–136. https://doi.org/10.1007/s10032-017-0284-8

    Article  Google Scholar 

  34. Westermann H, Savelka J, Benyekhlef K (2021) Paragraph similarity scoring and fine-tuned BERT for legal information retrieval and entailment. In: Okazaki N, Yada K, Satoh K, Mineshima K (eds) New frontiers in artificial intelligence. JSAI-isAI 2020, lecture notes in computer science. Springer vol 12758, https://doi.org/10.1007/978-3-030-79942-7_18

  35. Yin PY (2003) Ant colony search algorithms for optimal polygonal approximation of plane curves. Pattern Recogn 36(8):1783–1797

    Article  MATH  Google Scholar 

  36. Yin PY (2004) A discrete particle swarm algorithm for optimal polygonal approximation of digital curves. J Visual Comun Image Represent 15 (2):241–260

    Article  MathSciNet  Google Scholar 

  37. Yuan Z, Pan H, Zhang L (2008) A novel pen-based flowchart recognition system for programming teaching. Springer, Berlin

    Book  Google Scholar 

  38. Yun X, Zhang Y, Yin F, Liu C (2021) Instance GNN: a learning framework for joint symbol segmentation and recognition in online handwritten diagrams. IEEE Trans Multimed, vol 24, https://doi.org/10.1109/TMM.2021.3087000https://doi.org/10.1109/TMM.2021.3087000

  39. Zwillinger D, Kokoska S (2000) CRC standard probability and statistics tables and formulae. Chapman & Hall, New York, p 372

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Computer Science & Engineering, Indian Institute of Information Technology Kalyani, Kalyani, 741235, West Bengal, India

    Sagarika Ghosh, Sanjoy Pratihar & Sanjay Chatterji

  2. Computer Science & Engineering, University of Engineering & Management Jaipur, Jaipur, 303807, Rajasthan, India

    Sagarika Ghosh

  3. Computer Science & Engineering, National Institute of Technology Durgapur, Durgapur, 713209, West Bengal, India

    Anupam Basu

Authors
  1. Sagarika Ghosh
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sanjoy Pratihar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sanjay Chatterji
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Anupam Basu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Sagarika Ghosh: Investigation, Analysis, Writing - Original draft preparation; Sanjay Chatterji: Conceptualization, Supervision, Review & Editing; Sanjoy Pratihar: Conceptualization, Supervision, Review & Editing; Anupam Basu: Conceptualization, Supervision.

Corresponding author

Correspondence to Sagarika Ghosh.

Ethics declarations

Conflict of Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper and this manuscript has not been submitted to, nor is under review at, another journal or other publishing venue.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghosh, S., Pratihar, S., Chatterji, S. et al. Matching of hand-drawn flowchart, pseudocode, and english description using transfer learning. Multimed Tools Appl 82, 27027–27055 (2023). https://doi.org/10.1007/s11042-023-14346-9

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-023-14346-9

Keywords

Search

Navigation