Skip to main content
SpringerLink
Log in

An object detection-based framework for automatically recognizing iStar hand drafts

  • Published:
Automated Software Engineering Aims and scope Submit manuscript

Abstract

Although various requirements modeling tools have been developed in the past decades to facilitate modeling practice, hand drafting is still preferred to a certain extent by some modelers. Thus, digitizing hand drafts has been essential in promoting modeling practice. In this paper, we focus on a requirement modeling language, iStar, which has been investigated for over two decades and has been widely accepted as a valuable method to model and analyze requirements, especially the social aspects. Specifically, we propose a comprehensive framework using customized object detection methods to systematically recognize and digitize iStar hand drafts, promoting the practical adoption of iStar. To evaluate the performance of our approach, we have systematically established a dataset that includes 630 hand drafts. To the best of our knowledge, this is the first iStar hand draft dataset and will contribute to further studies in this field. We have designed and conducted a series of experiments to evaluate our approach based on the data, showing that our approach can effectively digitize iStar hand drafts. The entity recognition rate reached 99.6%. The draft recognition rate reached 91.4%. In addition to the experiments, we have pragmatically applied our approach to actual modeling practice, illustrating the practical applicability of our approach.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26

Similar content being viewed by others

References

  • Ahmed, J., Huang, M.: Classification of role stereotypes for classes in UML class diagrams using machine learning. (2020)

  • Bellman, Richard: Dynamic programming. Science 153(3731), 34–37 (1966)

    Article  MATH  Google Scholar 

  • Breiman, Leo: Random forests. Mach. Learn. 45(1), 5–32 (2001)

    Article  MATH  Google Scholar 

  • Bresler, M., Prua, D., Hlavác, V.: Modeling flowchart structure recognition as a max-sum problem. In: 2013 12th international conference on document analysis and recognition. IEEE, (2013)

  • Bresler, M., Průša, D., Hlaváč, V.: Recognizing off-line flowcharts by reconstructing strokes and using on-line recognition techniques. In: 2016 15th international conference on frontiers in handwriting recognition (ICFHR). IEEE, (2016)

  • Bresler, M., Pruša, D., Hlavác, Václav.: Using agglomerative clustering of strokes to perform symbols over-segmentation within a diagram recognition system. CVWW 15, 67–74 (2015)

    Google Scholar 

  • Cares, C., et al. iStarML: an XML-based model interchange format for i. In: Proceedings of the 3rd international i* workshop. CEUR-WS. org, (2008)

  • Carton, C., Lemaitre, A., Coüasnon, B.: Fusion of statistical and structural information for flowchart recognition. In: 2013 12th international conference on document analysis and recognition. IEEE, (2013)

  • Chang, C.-C., Hsiao, J.-Y., Hsieh, C.-P.: An adaptive median filter for image denoising. In: 2008 second international symposium on intelligent information technology application. Vol. 2. IEEE, (2008)

  • Chen, Y., et al. T-star: a text-based istar modeling tool. In: 2019 IEEE 27th international requirements engineering conference (RE). IEEE, (2019)

  • Chen, Q., Grundy, J., Hosking, J.: SUMLOW: early design-stage sketching of UML diagrams on an E-whiteboard. Software Pract. Exp. 38(9), 961–994 (2008)

    Article  Google Scholar 

  • Coüasnon, Bertrand: Dmos, a generic document recognition method: application to table structure analysis in a general and in a specific way. Int. J. Doc. Anal. Recog. (IJDAR) 8(2), 111–122 (2006)

    Article  Google Scholar 

  • Dalpiaz, F., Franch, X., Horkoff, J.,: istar 2.0 language guide. arXiv:1605.07767, (2016)

  • Du, X., Li, T., Wang, D.: An automatic layout approach for iStar models. iStar. (2017)

  • Fang, J., Feng, Z., Cai, B.: DrawnNet: offline hand-drawn diagram recognition based on keypoint prediction of aggregating geometric characteristics. Entropy 24(3), 425 (2022)

    Article  Google Scholar 

  • Gervais, P., et al. The DIDI dataset: digital ink diagram data. arXiv:2002.09303 (2020)

  • Gil, J.Y., Kimmel, R.: Efficient dilation, erosion, opening, and closing algorithms. IEEE Trans. Pattern Anal. Mach. Intell. 24(12), 1606–1617 (2002)

    Article  MATH  Google Scholar 

  • Hammond, T., Davis, R: Tahuti: A geometrical sketch recognition system for uml class diagrams. ACM SIGGRAPH 2006 Courses. (2006). 25-es

  • Hearst, M.A., et al.: Support vector machines. IEEE Intell. Syst. Appl. 13(4), 18–28 (1998)

    Article  Google Scholar 

  • Horkoff, J., Maiden, N.A.M., A creative iStar modeling tool. iStar, Creative Leaf (2016)

  • Horkoff, J., et al.: Goal-oriented requirements engineering: an extended systematic mapping study. Requir. Eng. 24(2), 133–160 (2019)

    Article  Google Scholar 

  • Hwang, H., Haddad, Richard A.: Adaptive median filters: new algorithms and results. IEEE Trans. Image Process. 4(4), 499–502 (1995)

    Article  Google Scholar 

  • Leaf 2.0. http://www.cs.toronto.edu/~amgrubb/leaf-2.0/Tool.html

  • Lemaitre, A., et al. Interest of syntactic knowledge for on-line flowchart recognition. International Workshop on Graphics Recognition. Springer, Berlin, Heidelberg, (2011)

  • Li, T., Grubb, A.M., Horkoff, J.,: Understanding challenges and tradeoffs in iStar tool development. iStar. (2016)

  • Lin, T.-Y., et al. Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. (2017)

  • Lin, T.-Y., Patterson, G., Ronchi, M., R.: COCO Metrics. https://cocodataset.org/#detection-eval

  • Olsen, L., et al.: Sketch-based modeling: a survey. Comput. Graph. 33(1), 85–103 (2009)

    Article  Google Scholar 

  • Otsu, Nobuyuki: A threshold selection method from gray-level histograms. IEEE Trans. Syst. Man Cybern. 9(1), 62–66 (1979)

    Article  Google Scholar 

  • PiStar. https://www.cin.ufpe.br/~jhcp/pistar/tool/#

  • Plimmer, B., Freeman, I.: A toolkit approach to sketched diagram recognition. In: Proceedings of HCI 2007 The 21st British HCI group annual conference University of Lancaster, UK 21. (2007)

  • Rabiner, L., Juang, B.: An introduction to hidden Markov models. IEEE ASSP Mag. 3(1), 4–16 (1986)

    Article  Google Scholar 

  • Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv:1804.02767 (2018)

  • Ren, S., et al. Faster r-cnn: towards real-time object detection with region proposal networks. Adv. Neural Inf. Process. Syst. 28 (2015)

  • Rumbaugh, J.: The unified modeling language reference manual. Pearson Education India, (2005)

  • Sas, J., Markowska-Kaczmar, U.: Logical structure recognition of diagram images. In: 2015 federated conference on computer science and information systems (FedCSIS). IEEE, (2015)

  • Schäfer, B., Keuper, M., Stuckenschmidt, Heiner: Arrow R-CNN for handwritten diagram recognition. Int. J. Doc. Anal. Recog. (IJDAR) 24(1), 3–17 (2021)

    Article  Google Scholar 

  • Shi, D., et al. Research in domain-oriented sketch recognition method Nanjing University, (2016)

  • Trier, Oeivind Due, Taxt, Torfinn: Evaluation of binarization methods for document images. IEEE Trans. Pattern Anal. Mach. Intell. 17(3), 312–315 (1995)

    Article  Google Scholar 

  • Wang, Y., et al. A prototype for generating meaningful layout of istar models. In: International conference on conceptual modeling. Springer, Cham, (2018)

  • Wang, Y., et al.: Toward practical adoption of i* framework: an automatic two-level layout approach. Requir. Eng. 26(3), 301–323 (2021)

    Article  Google Scholar 

  • Wohlin, C., et al.: Experimentation in software engineering. Springer, Cham (2012)

    Book  Google Scholar 

  • Wu, J., et al. Offline sketch parsing via shapeness estimation. In: Twenty-Fourth international joint conference on artificial intelligence. (2015)

  • Wüest D., Glinz, M.: Flexible sketch-based requirements modeling. In: International working conference on requirements engineering: foundation for software quality. Springer, Berlin, Heidelberg, (2011)

  • Yu, Eric: Modeling strategic relationships for process reengineering. Soc. Model. Requir. Eng. 11(2011), 66–87 (2011)

    Google Scholar 

  • Yuan, Z., Pan, H., Zhang, L.: A novel pen-based flowchart recognition system for programming teaching. Workshop on Blended Learning. Springer, Berlin, Heidelberg (2008)

  • Zhu, Y., L, T.: iStar 630 dataset. https://zenodo.org/record/5336891, (2021)

Download references

Acknowledgements

This work is partially supported by the National Natural Science Foundation of China (Nos. 61902010, 62162051), the Project of Beijing Municipal Education Commission (No. KM202110005025), and Beijing Natural Science Foundation Project (No. Z200002).

Author information

Authors and Affiliations

  1. Beijing University of Technology, 100 Pingleyuan, Beijing, 100124, China

    Yuran Zhu & Tong Li

Authors
  1. Yuran Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tong Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

YZ implemented the technical solution, conducted all experiments, and wrote the main manuscript text. TL identified the research problem, proposed an initial technical solution, and reviewed the manuscript.

Corresponding author

Correspondence to Tong Li.

Ethics declarations

Conflict of interest

The authors declare that no conflict of interest exists in the submission of this manuscript, and all authors approve the manuscript for publication.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhu, Y., Li, T. An object detection-based framework for automatically recognizing iStar hand drafts. Autom Softw Eng 29, 57 (2022). https://doi.org/10.1007/s10515-022-00361-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10515-022-00361-x

Keywords

Search

Navigation