Skip to main content
SpringerLink
Log in

On the comprehension of workflows modeled with a precise style: results from a family of controlled experiments

  • Regular Paper
  • Published:
Software & Systems Modeling Aims and scope Submit manuscript

Abstract

In this paper, we present the results from a family of experiments conducted to assess whether the level of formality/precision in workflow modeling, based on UML activity diagrams, influences two aspects of construct comprehensibility: correctness of understanding and task completion time. In particular, we have considered two styles for workflow modeling with different levels of formality: a precise style (with specific rules and imposed constraints) and an ultra-light style (no rules, no imposed constraints). Experiments were conducted with 111 participants (Bachelor and Master students). In each experiment, participants accomplished comprehension tasks on two workflows, modeled either with the precise style or with a lighter variant. The main results from our data analysis can be summarized as follows: (i) all participants achieved a significantly better comprehension of workflows written in the precise style, (ii) the style had no significant impact on task completion time, (iii) more experienced participants benefited more, with respect to less experienced ones, from the precise style, as for their correctness of understanding, and (iv) all participants found the precise style useful in comprehending workflows.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Notes

  1. A particular manner or technique by which something is done, created, or performed. Merriam Webster’s Collegiate Dictionary, www.merriam-webster.com/.

  2. This grammatical error was present in the original activity diagram, downloaded from the Web.

  3. When a notation is evaluated by its proponents, researcher biases could be present. For example, questions in comprehension questionnaires might be trivially answered with the preferred notation and might be unanswerable with the other.

  4. This kind of replication introduces variations (e.g., different kinds of participants) in essential aspects of the experimental conditions [9].

  5. If participants are tested first under condition A, then under condition B, they could potentially exhibit a better performance under condition B, because of their prior practice under condition A.

  6. The value is computed using the equation: 0.62 \(+\) 0.62*x \(=\) 0.79, see Table 4.

References

  1. Abrahão, S.M.A., Gravino, C., Pelozo, E.I., Scanniello, G., Tortora, G.: Assessing the effectiveness of sequence diagrams in the comprehension of functional requirements: results from a family of five experiments. IEEE Trans. Softw. Eng. 39(3), 327–342 (2013)

    Article  Google Scholar 

  2. Agarwal, R., De, P., Sinha, A.P.: Comprehending object and process models: an empirical study. IEEE Trans. Softw. Eng. 25(4), 541–556 (1999)

    Article  Google Scholar 

  3. Ali, S., Yue, T., Briand, L.C.: Does aspect-oriented modeling help improve the readability of UML state machines? Softw. Syst. Model. pp. 1–33 (2012)

  4. Ambler, S.W.: The Elements of UML 2.0 Style. Cambridge University Press, Cambridge (2005)

    Book  Google Scholar 

  5. Aranda, J., Ernst, N., Horkoff, J., Easterbrook, S.: A framework for empirical evaluation of model comprehensibility. In: Proceedings of the International Workshop on Modeling in Software Engineering, MISE ’07, p. 7-, Washington, DC, USA, 2007. IEEE, Computer Society

  6. Astesiano, E., Reggio, G., Ricca, F.: Modeling business within a UML-based rigorous software development approach. In: Degano, P., DeNicola, R., Meseguer, J. (eds.) Concurrency, Graphs and Models, number 5065 in LNCS, pp. 261–277. Springer, Berlin (2008)

  7. Baker, R.: Modern permutation test software. In: Edgington, E. (ed.) Randomization Tests, Marcel Decker (1995)

  8. Basili, V.R., Caldiera, G., Rombach, D.H.: The Goal Question Metric Paradigm, Encyclopedia of Software Engineering. Wiley, London (1994)

    Google Scholar 

  9. Basili, V.R., Shull, F., Lanubile, F.: Building knowledge through families of experiments. IEEE Trans. Softw. Eng. 25(4), 456–473 (1999)

    Article  Google Scholar 

  10. Bauer, M.I., Johnson-Laird, P.N.: How diagrams can improve reasoning. Psychol. Sci. 4, 372–378 (1993)

    Article  Google Scholar 

  11. Birkmeier, D., Overhage, S.: Is BPMN really first choice in joint architecture development? an empirical study on the usability of BPMN and UML activity diagrams for business users. In: Research into Practice: Reality and Gaps, number 6093 in LNCS, pp. 119–134. Springer, Berlin (2010)

  12. Briand, L.C., Labiche, Y., Di Penta, M., Yan-Bondoc, H.D.: An experimental investigation of formality in UML-based development. IEEE Trans. Softw. Eng. 31(10), 833–849 (2005)

    Article  Google Scholar 

  13. Broy, M., Cengarle, M.V.: UML formal semantics: lessons learned. Softw. Syst. Model. 10(4), 441–446 (2011)

    Article  Google Scholar 

  14. Carver, J., Jaccheri, L., Morasca, S., Shull, F.: Issues in using students in empirical studies in software engineering education. In: 9th International Symposium on Software Metrics, p. 239, Washington, DC, USA, 2003. IEEE CS

  15. De Lucia, A., Francese, R., Tortora, G.: Deriving workflow enactment rules from UML activity diagrams: a case study. In: IEEE Symposium on Human Centric Computing Languages and. Environments, pp. 211–218 (2003)

  16. Di Cerbo, F., Dodero, G., Reggio, G., Ricca, F., Scanniello, G.: Precise vs. ultra-light activity diagrams—an experimental assessment in the context of business process modelling. In: International Conference on Product Focused Software Development and Process Improvement, number 6759 in LNCS, pp. 291–305. Springer (2011)

  17. Di Nitto, E., Lavazza, L., Schiavoni, M., Tracanella, E., Trombetta, M.: Deriving executable process descriptions from UML. In: 22rd International Conference on Software Engineering, pp. 155–165 (2002)

  18. Fowler, M.: UML Distilled: A Brief Guide to the Standard Object Modeling Language, 3rd edn. Addison-Wesley Longman Publishing Co. Inc., Boston (2003)

    Google Scholar 

  19. Gogolla, M., Richters, M.: On constraints and queries in UML. In: UML Workshop, pp. 109–121 (1997)

  20. Gross, A., Doerr, J.: EPC vs. UML activity diagram—two experiments examining their usefulness for requirements engineering. In: Proceedings of Requirements Engineering Conference, pp. 47–56, Washington, DC, USA, 2009. IEEE CS

  21. Havey, M.: Essential Business Process Modeling. O’Reilly Media Inc (2005)

  22. Hedges, L.V., Olkin, I.: Statistical Methods for Meta-Analysis. Academic Press, New York (1985)

    MATH  Google Scholar 

  23. Jurack, S., Lambers, L., Mehner, K., Taentzer, G., Wierse, G.: Object flow definition for refined activity diagrams. In: 12th International Conference on Fundamental Approaches to Software Engineering, pp. 49–63, Springer, Berlin (2009)

  24. Kampenes, V.B., Dybå, T., Hannay, J.E., Sjøberg, D.I.K.: Systematic review: a systematic review of effect size in software engineering experiments. Inf. Softw. Technol. 49, 1073–1086 (2007)

    Article  Google Scholar 

  25. Kim, J., Hahn, J., Hahn, H.: How do we understand a system with (so) many diagrams? cognitive integration processes in diagrammatic reasoning. Inf. Syst. Res. 11(3), 284–303 (2000)

    Article  Google Scholar 

  26. Kitchenham, B., Pfleeger, S., Pickard, L., Jones, P., Hoaglin, D., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)

    Article  Google Scholar 

  27. Kitchenham, B., Al-Khilidar, H., Babar, M., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., Zhu, L.: Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Eng. 13, 97–121 (2008)

    Article  Google Scholar 

  28. Marchetto, A., Ricca, F.: From objects to services: toward a stepwise migration approach for Java applications. Int. J. Softw. Tools Technol. Transf. 11, 427–440 (2009)

    Article  Google Scholar 

  29. Mendling, J., Reijers, H., van der Aalst, W.: Seven process modeling guidelines (7pmg). Inf. Softw. Technol. 52(2), 127–136 (2010)

    Article  Google Scholar 

  30. Mendona, M.G., Maldonado, J.C., de Oliveira, M.C.F., Carver, J., Fabbri, S.C.P.F.F., Shull, F., Travassos, G.H., Hohn, E.N., Basili, V.R.: A framework for software engineering experimental replications. In: International Conference on Engineering of Complex Computer Systems, pp. 203–212 IEEE, 2008

  31. Motulsky, H.: Intuitive Biostatistics: A Nonmathematical Guide to Statistical Thinking. Oxford University Press, Oxford (2010)

    Google Scholar 

  32. Nugroho, A., Flaton, B., Chaudron, M.R.V.: Empirical analysis of the relation between level of detail in UML models and defect density. In: Proceedings of International Conference on Model Driven Engineering Languages and Systems, pp. 600–614 (2008)

  33. Nugroho, A.: Level of detail in UML models and its impact on model comprehension: a controlled experiment. Inf. Softw. Technol. 51(12), 1670–1685 (2009)

    Article  Google Scholar 

  34. OMG. Business process model and notation (BPMN) Version 2.0. OMG Final Adopted Specification, Object Management Group (2006)

  35. Oppenheim, A.N.: Questionnaire Design, Interviewing and Attitude Measurement. Pinter, London (1992)

    Google Scholar 

  36. Organization for the Advancement of Structured Information Standards (OASIS). Web Services Business Process Execution Language—Version 2.0. OASIS Standard (2007)

  37. Peixoto, D., Batista, V., Atayde, A., Borges, E., Resende, R. ,Pádua, C. : A comparison of BPMN and UML 2.0 activity diagrams. In: VII Simposio Brasileiro de Qualidade de Software (2008)

  38. Ramsey, H.R., Atwood, M.E., Van Doren, J.R.: Flowcharts versus program design languages: an experimental comparison. Commun. ACM 26(6), 445–449 (1983)

    Article  Google Scholar 

  39. Reggio, G., Leotta, M.,Ricca, F. : Precise is better than light—a document analysis study about quality of business process models. In: Proceedings of EmpiRE 2011, pp. 61–68. IEEE Digital Library (2011)

  40. Reggio, G., Ricca, F., Astesiano, E., Leotta, M.: On business process modelling with the UML: a discipline and four styles. Technical Report DISI-TR-11-03, DISI—University of Genova, Italy, April 2011. Available at http://softeng.disi.unige.it/tech-rep/TECDOC.pdf

  41. Reggio, G., Ricca, F., Scanniello, G., Di Cerbo, F., Dodero, G.: A precise style for business process modelling: results from two controlled experiments. In: Model Driven Engineering Languages and Systems, 14th International Conference, MODELS 2011, Wellington, New Zealand, October 16–21, 2011. Proceedings, volume 6981 of LNCS, pp. 138–152. Springer (2011)

  42. Ricca, F., Di Penta, M., Torchiano, M., Tonella, P., Ceccato, M.: How developers’ experience and ability influence web application comprehension tasks supported by UML stereotypes: a series of four experiments. IEEE Trans. Softw. Eng. 36(1), 96–118 (2010)

    Article  Google Scholar 

  43. Rychly, M., Weiss, P.: Modeling of service oriented architecture: from business process to service realisation. In: Proceedings of International Working Conference on Evaluation of Novel Approaches to Software Engineering, pp. 140–146. Institute for Systems and Technologies of Information, Control and Communication (2008)

  44. Scaife, M., Rogers, Y.: External cognition: how do graphical representations work? Int. J. Hum.-Comput. Stud. 45(2), 185–213 (1996)

    Article  Google Scholar 

  45. Scanlan, D.A.: Structured flowcharts outperform pseudocode: an experimental comparison. IEEE Softw. 6(5), 28–36 (1989)

    Article  Google Scholar 

  46. Scanniello, G., Gravino, C., Genero, M., Cruz-Lemus, J.A., Tortora, G.: On the impact of UML analysis models on source code comprehensibility and modifiability. ACM Trans. Soft. Eng. Meth. (to appear)

  47. Scheer, A.: ARIS-Business Process Modeling. Springer, Berlin (2000)

    Book  Google Scholar 

  48. Shull, F., Mendonça, M., Basili, V., Carver, J., Maldonado, J.C., Fabbri, S., Travassos, G., Ferreira, M.: Knowledge-sharing issues in experimental software engineering. Empir. Softw. Eng. 9(1–2), 111–137 (2004)

    Article  Google Scholar 

  49. Shull, F.J., Carver, J.C., Vegas, S., Juristo, N.: The role of replications in empirical software engineering. Empir. Softw. Eng. 13(2), 211–218 (2008)

    Article  Google Scholar 

  50. Sjoberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanovic, A., Liborg, N., Rekdal, A.C.: A survey of controlled experiments in software engineering. IEEE Trans. Softw. Eng. 31(9), 733–753 (2005)

    Article  Google Scholar 

  51. Staron, M., Kuzniarz, L., Wohlin, C.: Empirical assessment of using stereotypes to improve comprehension of UML models: a set of experiments. J. Syst. Softw. 79(5), 727–742 (2006)

    Article  Google Scholar 

  52. Svahnberg, M., Aurum, A., Wohlin, C.: Using students as subjects—an empirical evaluation. In: Symposium on Empirical Software Engineering and Measurement, pp. 288–290, Kaiserslautern, Germany, 2008. IEEE Computer Society

  53. UML Revision Task Force. OMG Unified Modeling Language (OMG UML), Superstructure, V 2.4.1 (2011)

  54. Vegas, S., Juzgado, N.J., Moreno, A.M., Solari, M., Letelier, P.: Analysis of the influence of communication between researchers on experiment replication. In: International Symposium on Empirical Software Engineering, pp. 28–37, Rio de Janeiro, Brazil, 2006. IEEE Computer Society

  55. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M., Regnell, B., Kluwer, A .: Wesslén. Experimentation in Software Engineering—An Introduction. Kluwer (2000)

  56. Zimmerman, M.K., Lundqvist, K., Leveson, N.G.: Investigating the readability of state-based formal requirements specification languages. In: Proceedings of International Conference on Software Engineering, pp. 33–43 (2002)

Download references

Acknowledgments

We would like to thank all the participants in the experiments.

Author information

Authors and Affiliations

  1. DIBRIS, Università di Genova, Genoa, Italy

    Gianna Reggio & Filippo Ricca

  2. DiMIE, Università della Basilicata, Potenza, Italy

    Giuseppe Scanniello

  3. SAP Research, Sophia Antipolis, Valbonne, France

    Francesco Di Cerbo

  4. IDSE, Free University of Bozen-Bolzano, Bolzano, Italy

    Gabriella Dodero

Authors
  1. Gianna Reggio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Filippo Ricca
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Giuseppe Scanniello
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Francesco Di Cerbo
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gabriella Dodero
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Filippo Ricca.

Additional information

Communicated by Prof. Jon Whittle.

Appendix A (Comprehension questionnaire for DM)

Appendix A (Comprehension questionnaire for DM)

  1. 1.

    Indicate the participant/s of the workflow

    • Reviewer

    • Product Manager

    • Owner

    • Author

    • Approver

  2. 2.

    List the object/s (or datum/data) used in the workflow

    • Document

    • Client

    • Draft Document

    • Document items

    • Online Document

  3. 3.

    Indicate the responsible participant/s for the action/activity in charge of updating the document

    • Approver

    • Business manager

    • Reviewer

    • Online Document

    • Author

  4. 4.

    Indicate the relationship/s between a document and the other entities

    • Document is related to the change requests

    • Document is related to the online document

    • Approver approves the change requests of the document

    • Document is related to the document items

    • Reviewer works on the document

  5. 5.

    Indicate which of the following are states that a document can assume

    • Draft

    • To be updated

    • New

    • Revised

    • Deleted

  6. 6.

    Indicate the action(s)/activity(ies) accomplished by the Author

    • Create document

    • Create change requests

    • Revise document

    • Update document

    • Approve and archive the document

  7. 7.

    The action/activity in charge of applying changes takes as input

    • Only the change requests

    • Document with its associated change requests

    • Only the document without the change requests

    • Author, document and change requests

    • Online document

  8. 8.

    Mark the true proposition(s)

    • Approved document implies that the status of the document is online

    • The “else branch” of the decision node at the bottom is selected when changes are not needed (i.e., the document has not to be updated) and the document is not obsolete

    • The approver archives the document

    • After the action/activity “revise online” the status of the document is: obsolete or “to be updated” or online

    • The reviewer can revise the document two times at most

  9. 9

    . Which modification should be done in the workflow to store each online document version in a persistent mechanism (e.g., a database)?

    • A new action/activity should be added when the document is approved. That action, executed by the approver, should store in a persistent mechanism the document with a unique ‘id’

    • A new action/activity should be added when the document changes its status in “not approved.” That action, executed by the approver, should store in a persistent mechanism the document with a unique ‘id’

    • A new action/activity should be added when the document become obsolete. That action, executed by the approver, should store in a persistent mechanism the document with a unique ‘id’

    • A new action/activity should be added in the else branch of the decision node at the bottom. That action, executed by the reviewer, should store in a persistent mechanism the document with a unique ‘id’

    • A new action/activity should be added when the document is created. That action, executed by the author, should store in a persistent mechanism the document with a unique ‘id’

  10. 10.

    Which modification should be done in the workflow to take into account the fact that the reviewer can also ask the updating of the document to the author during the “revise draft” action/activity?

    • It is sufficient to substitute the revised document with “to be updated” document after the “revise draft” action/activity

    • A new decision node should be added after the “revise draft” action/activity with guards: change needed (or document to be updated) and else. The first will conduct to the approval and the second one to updating

    • It should be changed the decision node after the approval. The “not approve” guard should conduct to the “updates document” action/activity instead of activity final node

    • A new decision node should be added after the “revise draft” action/activity with guards: change needed (or document to be updated) and else. The first will conduct to the updating and the second one to approval

    • It should be changed the decision node after the approval. The “not approved” guard should conduct to the “revise draft” action/activity instead of activity final node

  11. 11.

    How should do you restructure (without changing the meaning) the workflow to make it clearer?

    • Swimlines should be added

    • The owner should be deleted. When the document is not approved it should be archived

    • The else branch of the decision node at the bottom should be deleted

    • The decision node at the bottom should be split in two decision nodes. The first coping with the obsolete/non-obsolete guard and the second one handling the change needed (to be updated) decision

    • All the object nodes should be removed to avoid cluttering

  12. 12.

    Suppose to implement a Web application that manages the proposed workflow. Imagine that document flows among the different Web browsers of the participants. Which attributes/fields are needed for the object Document?

    • Content: String

    • Obsolete: Boolean

    • NumberOfChanges: int

    • ChangeRequests: List

    • Status: DocumentStatus

Rights and permissions

Reprints and permissions

About this article

Cite this article

Reggio, G., Ricca, F., Scanniello, G. et al. On the comprehension of workflows modeled with a precise style: results from a family of controlled experiments. Softw Syst Model 14, 1481–1504 (2015). https://doi.org/10.1007/s10270-013-0386-9

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10270-013-0386-9

Keywords

Search

Navigation