skip to main content
10.1145/3593663.3593664acmotherconferencesArticle/Chapter ViewAbstractPublication PagesecseeConference Proceedingsconference-collections
research-article

Evaluating Graph-based Modeling Languages

Published: 19 June 2023 Publication History

Abstract

As humans, we tend to use models to describe reality. Modeling languages provide the formal frameworks for creating such models. Usually, the graphical design of individual model elements is based on subjective decisions; their suitability is determined at most by the prevalence of the modeling language. With other words: there is no objective way to compare different designs of model elements. The present paper addresses this issue: it introduces a systematic approach for evaluating the elements of graph-based modeling languages comprising 14 criteria – derived from standards, usability analyses, or the design theories ‘Physics of Notations’ and ‘Cognitive Dimensions of Notations’. The criteria come with measurement procedures and evaluation schemes based on reasoning, eye tracking, and questioning. The developed approach is demonstrated with a specific use case: three distinct sets of node elements for causal graphs are evaluated in an eye tracking study with 41 subjects.

References

[1]
Jorge Aranda, Neil Ernst, Jennifer Horkoff, and Steve Easterbrook. 2007. A framework for empirical evaluation of model comprehensibility. Proceedings of the 1st International Workshop on Modeling in Software Engineering (MISE ’07), 1–6.
[2]
Jörg Becker, Michael Rosemann, and Christoph Von Uthmann. 2000. Guidelines of Business Process Modeling. Springer, Berlin, Germany, 30–49.
[3]
Jaques Bertin. 1974. Graphische Semiologie: Diagramme, Netze, Karten (2 ed.). De Gruyter, Berlin, Germany.
[4]
Alan Blackwell, Carol Britton, Anna Cox, Thomas Green, Corin Gurr, Gada Kadoda, Maria Kutar, Martin Loomes, Crystopher Nehaniv, Marian Petre, Chris Roast, Chris Roe, Alan Wong, and Richard Young. 2001. Cognitive Dimensions of Notations: Design Tools for Cognitive Technology. Springer, Berlin, Germany, 325–341.
[5]
Alan Blackwell and Thomas Green. 2000. A cognitive dimensions questionnaire optimised for users, Alan Blackwell and Eleonora Bilotta (Eds.). Proceedings of the 12th Annual Workshop of the Psychology of Programming Interest Group (PPIG ’00), 137–154.
[6]
Grady Booch. 1994. Object-Oriented Analysis and Design with Applications (2 ed.). Addison-Wesley, Boston, MA, USA.
[7]
Robert Bowers. 2017. Causal Reasoning. Springer, Cham, Germany, 1–17. https://doi.org/10.1007/978-3-319-16999-6_3114-1
[8]
Vasiliki Diamantopoulou, Michalis Pavlidis, and Haralambos Mouratidis. 2017. Evaluation of a Security and Privacy Requirements Methodology using the Physics of Notation. Cham, Switzerland, 210–225.
[9]
Zdena Dobesova. 2013. Using the ‘physics’ of notation to analyse modelbuilder diagrams. Proceedings of the 13th International Multidisciplinary Scientific GeoConference Surveying Geology and Mining Ecology Management (SGEM ’13), 595–602. https://doi.org/10.5593/SGEM2013/BB2.V1/S08.039
[10]
Mohamed El-Attar. 2019. Evaluating and empirically improving the visual syntax of use case diagrams. J Syst Softw 156 (10 2019), 136–163. Issue 1. https://doi.org/10.1016/j.jss.2019.06.096
[11]
Gregor Engels. 2019. Modellierungssprache. Gito, Berlin, Germany. https://wi-lex.de/index.php/lexikon/technologische-und-methodische-grundlagen/sprache/modellierungssprache/
[12]
Andy Field and Graham Hole. 2002. How to Design and Report Experiments. SAGE Publications, Thousand Oaks, CA, USA.
[13]
Lisa Grabinger, Florian Hauser, and Jürgen Mottok. 2022. Accessing the presentation of causal graphs and an application of gestalt principles with eye tracking. Proceedings of the 29th IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER ’22), 1267–1274.
[14]
Thomas Green. 1989. Cognitive dimensions of notations, Alistair utcliffe and Linda MacAulay (Eds.). Proceedings of the 5th Conference of the British Computer Society (People and Computers V), 443–460.
[15]
Thomas Green and Alan Blackwell. 1998. Cognitive Dimensions of Information Artefacts: A tutorial (Version 1.2).
[16]
Thomas Green and Marian Petre. 1996. Usability analysis of visual programming environments: A ’cognitive dimensions’ framework. J. Vis. Lang. Comput. 7 (6 1996), 131–174. Issue 2. https://doi.org/10.1006/jvlc.1996.0009
[17]
Sandra Hart and Lowell Staveland. 1988. Development of NASA-TLX (task load index): Results of empirical and theoretical research. Elsevier, Amsterdam, Netherlands, 139–183.
[18]
Henrique Henriques, Hugo Lourenço, Vasco Amaral, and Miguel Goulão. 2018. Improving the developer experience with a low-code process modelling language. Proceedings of the 21th ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS ’18), 200–210. https://doi.org/10.1145/3239372.3239387
[19]
Miguel Hernán and James Robins. 2020. Causal Inference: What If. Chapman & Hall, Boca Raton, FL, USA.
[20]
Christopher Hitchcock. 2022. Causal Models. Metaphysics Research Lab, Stanford, CA, USA. https://plato.stanford.edu/archives/spr2022/entries/causal-models/
[21]
Kasper Hornbæk. 2006. Current practice in measuring usability: Challenges to usability studies and research. Int. J. Hum. Comput. 64 (2 2006), 79–102. Issue 2. https://doi.org/10.1016/j.ijhcs.2005.06.002
[22]
ISO 9186-3:2014(E) 2014. Graphical symbols - Test methods - Part 3: Method for testing symbol referent association. Standard. International Organization for Standardization, Geneva, Switzerland.
[23]
ISO ISO 9241-112:2017(E) 2017. Ergonomics of human-system interaction — Part 112: Principles for the presentation of information. Standard. International Organization for Standardization, Geneva, Switzerland.
[24]
Dominik Janzing and Bernhard Schölkopf. 2017. Elements of Causal Inference Foundations and Learning Algorithms. MIT Press, Cambridge, MA, USA.
[25]
Daphne Koller and Nir Friedman. 2009. Probabilistic Graphical Models: Principles and Techniques. MIT press, Cambridge, MA, USA.
[26]
John Krogstie, Guttorm Sindre, and Håvard Jørgensen. 2006. Process models representing knowledge for action: A revised quality framework. Eur. J. Inf. Syst. 15 (2 2006), 91–102. Issue 1. https://doi.org/10.1057/palgrave.ejis.3000598
[27]
Brian Lawler. 2008. Book review: Object-oriented analysis and design with applications, third edition. SIGSOFT SEN 33 (9 2008), 29. Issue 5.
[28]
Odd Lindland, Guttorm Sindre, and Arne Sølvberg. 1994. Understanding quality in conceptual modeling. IEEE Softw. 11 (3 1994), 42–49. Issue 2. https://doi.org/10.1109/52.268955
[29]
Jock Mackinlay. 1986. Automating the design of graphical presentations of relational information. ACM Trans. Graph. 5 (4 1986), 110–141. Issue 2.
[30]
Thomas Maurer. 2011. Zur Notengebung an Hochschulen. Retrieved January 22, 2023 from http://www.technikexpertise.de/files/Zur_Notengebung_an_Hochschulen.pdf
[31]
Babak Memarian and Panagiotis Mitropoulos. 2011. Work factors affecting task demands of masonry work, Tulio Sulbaran (Ed.). Proceedings of the 47th Annual International Conference of Associated Schools of Construction, 1–9.
[32]
Jan Mendling, Hajo Reijers, and Will Van der Aalst. 2010. Seven process modeling guidelines (7PMG). Information and Software Technology (Inf. Softw.) 52 (2 2010), 127–136. Issue 2. https://doi.org/10.1016/j.infsof.2009.08.004
[33]
Ana Molina, Miguel Redondo, Manuel Ortega, and Carmen Lacave. 2014. Evaluating a graphical notation for modeling collaborative learning activities: A family of experiments. Sci. Comput. Program. 88 (8 2014), 54–81. https://doi.org/10.1016/j.scico.2014.02.019
[34]
Daniel Moody. 2009. The ‘physics’ of notations: Toward a scientific basis for constructing visual notations in software engineering. IEEE Trans. Softw. Eng. 35 (12 2009), 756–779. Issue 6. https://doi.org/10.1109/TSE.2009.67
[35]
Brady Neal. 2020. Introduction to Causal Inference from a Machine Learning Perspective. Retrieved January 22, 2023 from https://www.bradyneal.com/Introduction_to_Causal_Inference-Dec17_2020-Neal.pdf
[36]
Susanne Patig. 2006. Die Evolution von Modellierungssprachen. Frank & Timme, Berlin, Germany.
[37]
Judea Pearl, Madelyn Glymour, and Nicholas Jewell. 2016. Causal Inference in Statistics: A Primer. John Wiley & Sons, Hoboken, NJ, USA.
[38]
Judea Pearl and Dana Mackenzi. 2018. The Book of Why: The New Science of Cause and Effect. Penguin Books, London, England.
[39]
Patrick Planing. 2022. Statistik Grundlagen. Retrieved January 22, 2023 from https://statistikgrundlagen.de/ebook/
[40]
Thomas Richardson and James Robins. 2013. Single World Intervention Graphs: A Primer. Proceedings of the 2nd Conference on Uncertainty in Artifical Intelligence (UAI 2013), 1–11.
[41]
Christian Schalles, Michael Rebstock, and John Creagh. 2010. Ein generischer Ansatz zur Messung der Benutzerfreundlichkeit von Modellierungssprachen, Gregor Engels, Dimitris Karagiannis, and Heinrich Mayr (Eds.). Proceedings of Modellierung 2010, 15–30.
[42]
Reinhard Schuette and Thomas Rotthowe. 1998. The guidelines of modeling - an approach to enhance the quality in information models, Tok W. Ling, Sudha Ram, and Mong L. Lee (Eds.). Proceedings of the 17th International Conference on Conceptual Modeling (ER ’98), 240–254.
[43]
Vasileios Skaramagkas, Giorgos Giannakakis, Emmanouil Ktistakis, Dimitris Manousos, Ioannis Karatzanis, Nikolaos Tachos, Evanthia Tripoliti, Kostas Marias, Dimitrios Fotiadis, and Manolis Tsiknakis. 2021. Review of eye tracking metrics involved in emotional and cognitive processes. IEEE Rev. Biomed. Eng. 14 (3 2021), 1–19. Issue 1. https://doi.org/10.1109/RBME.2021.3066072
[44]
Kenia Sousa, Jean Vanderdonckt, Brian Henderson-Sellers, and Cesar Gonzalez-Perez. 2012. Evaluating a graphical notation for modelling software development methodologies. J. Vis. Lang. Comput. 23 (8 2012), 195–212. Issue 4. https://doi.org/10.1016/j.jvlc.2012.04.001
[45]
Matthew Vowels, Necati Camgoz, and Richard Bowden. 2023. D’ya Like DAGs? A survey on structure learning and causal discovery. ACM Comput Surv 55 (4 2023), 1–36. Issue 4. https://doi.org/10.1145/3527154
[46]
Peter Wehrum and Dorothea Mehling. 1993. Rational Rose: Objektorientierte Analyse und objektorientiertes Design nach der Booch-Methode. Springer, Berlin, Germany, 208–212.
[47]
Johannes Zagermann, Ulrike Pfeil, and Harald Reiterer. 2016. Measuring cognitive load using eye tracking technology in visual computing, Michael Sedlmair and Petra Isenberg (Eds.). Proceedings of the 6th Workshop on Beyond Time and Errors on Novel Evaluation Methods for Visualization (BELIV ’16), 78–85.
[48]
Eloïse Zehnder, Nicolas Mayer, and Guillaume Gronier. 2018. Evaluation of the Cognitive Effectiveness of the CORAS Modelling Language. Springer, Cham, Switzerland, 149–162.
[49]
Michael Zimoch, Rüdiger Pryss, Johannes Schobel, and Manfred Reichert. 2017. Eye Tracking Experiments on Process Model Comprehension: Lessons Learned. Springer, Cham, Switzerland, 153–168.

Cited By

View all
  • (2024)Eye Tracking and Semantic Evaluation for Ceramic Teapot Product ModelingApplied Sciences10.3390/app1501004615:1(46)Online publication date: 25-Dec-2024
  • (2024)A study on an efficient OSS inspection scheme based on encrypted GMLHigh-Confidence Computing10.1016/j.hcc.2024.100279(100279)Online publication date: Nov-2024

Index Terms

  1. Evaluating Graph-based Modeling Languages

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    ECSEE '23: Proceedings of the 5th European Conference on Software Engineering Education
    June 2023
    264 pages
    ISBN:9781450399562
    DOI:10.1145/3593663
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 19 June 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. causal graphs
    2. cognitive dimensions of notations
    3. eye tracking
    4. modeling languages
    5. physics of notations

    Qualifiers

    • Research-article
    • Research
    • Refereed limited

    Funding Sources

    • BMBF
    • STMWI

    Conference

    ECSEE 2023

    Acceptance Rates

    Overall Acceptance Rate 11 of 16 submissions, 69%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)27
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Eye Tracking and Semantic Evaluation for Ceramic Teapot Product ModelingApplied Sciences10.3390/app1501004615:1(46)Online publication date: 25-Dec-2024
    • (2024)A study on an efficient OSS inspection scheme based on encrypted GMLHigh-Confidence Computing10.1016/j.hcc.2024.100279(100279)Online publication date: Nov-2024

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    HTML Format

    View this article in HTML Format.

    HTML Format

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media