Stefan Zugal
ABSTRACT
To empirically investigate conceptual modeling languages, subjects are typically confronted with experimental tasks, such as the creation, modification or understanding of conceptual models. Thereby, accuracy, i.e., the amount of correctly performed tasks divided by the number of total tasks, is usually used to assess performance. Even though accuracy is widely adopted, it is connected to two often overlooked problems. First, accuracy is a rather insensitive measure. Second, for tasks of low complexity, the measurement of accuracy may be distorted by peculiarities of the human mind. In order to tackle these problems, we propose to additionally assess the subject's mental effort, i.e., the mental resources required to perform a task. In particular, we show how aforementioned problems connected to accuracy can be resolved, that mental effort is a valid measure of performance and how mental effort can easily be assessed in empirical research.
- J. Aranda, N. Ernst, J. Horkoff, and S. Easterbrook. A Framework for Empirical Evaluation of Model Comprehensibility. In Proc. MISE '07, pages 7--12, 2007. Google Scholar
Digital Library
- A. Baddeley. Working Memory: Theories, Models, and Controversies. Annu. Rev. Psychol., 63(1): 1--29, 2012.Google ScholarCross Ref
- M. Bannert. Managing cognitive load---recent trends in cognitive load theory. Learning and Instruction, 12(1): 139--146, 2002.Google ScholarCross Ref
- J. C. Carver, E. Syriani, and J. Gray. Assessing the frequency of empirical evaluation in software modeling research. In Proc. EESSMod '11, pages 28--37, 2011.Google Scholar
- J. A. Cruz-Lemus, M. Genero, M. E. Manso, S. Morasca, and M. Piattini. Assessing the understandability of UML statechart diagrams with composite states---A family of empirical studies. Empirical Software Engineering, 25(6): 685--719, 2009. Google ScholarDigital Library
- F. Davies. A Technology Acceptance Model for Empirically Testing New End-User Information Systems: Theory and Results. PhD thesis, Sloan School of Management, 1986.Google Scholar
- A. M. Férnandez-Sáez, M. Genero, and M. R. V. Chaudron. Does the level of detail of uml models affect the maintainability of source code? In Proc. EESSMod '11, pages 3--17, 2011.Google Scholar
- A. Gemino and Y. Wand. A framework for empirical evaluation of conceptual modeling techniques. Requir. Eng., 9(4): 248--260, 2004.Google ScholarDigital Library
- D. Gopher and R. Brown. On the psychophysics of workload: Why bother with subjective measure? Human Factors: The Journal of the Human Factors and Ergonomics Society, 26(5): 519--532, 1984.Google ScholarCross Ref
- I. Hadar and U. Leron. How intuitive is object-oriented design? Communications of the ACM, 51(5): 41--46, 2008. Google ScholarDigital Library
- C. Houy, P. Fettke, and P. Loos. Understanding understandability of conceptual models: What are we actually talking about? In Proc. ER '12, pages 64--77, 2012. Google ScholarDigital Library
- R. Laue and A. Gadatsch. Measuring the Understandability of Business Process Models - Are We Asking the Right Questions? In Proc. BPD '10, pages 37--48, 2011.Google ScholarCross Ref
- R. Mayer and P. Chandler. When learning is just a click away: Does simple user interaction foster deeper understanding of multimedia messages. Journal of Educational Psychology, 93(2): 390--397, 2001.Google ScholarCross Ref
- D. L. Moody. Cognitive Load Effects on End User Understanding of Conceptual Models: An Experimental Analysis. In Proc. ADBIS '04, pages 129--143, 2004.Google ScholarCross Ref
- J. Mylopoulos. Information modeling in the time of the revolution. Information Systems, 23(3/4): 127--155, 1998. Google ScholarDigital Library
- F. Paas. Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4): 429--434, 1992.Google ScholarCross Ref
- F. Paas, A. Renkl, and J. Sweller. Cognitive Load Theory and Instructional Design: Recent Developments. Educational Psychologist, 38(1): 1--4, 2003.Google Scholar
- J. Parsons and L. Cole. What do the pictures mean? Guidelines for experimental evaluation of representation fidelity in diagrammatical conceptual modeling techniques. DKE, 55(3): 327--342, 2005. Google ScholarDigital Library
- J. Pinggera, M. Furtner, M. Martini, P. Sachse, K. Reiter, S. Zugal, and B. Weber. Investigating the Process of Process Modeling with Eye Movement Analysis. In Proc. ER-BPM '12, to appear.Google Scholar
- J. Pinggera, P. Soffer, S. Zugal, B. Weber, M. Weidlich, D. Fahland, H. Reijers, and J. Mendling. Modeling Styles in Business Process Modeling. In Proc. BPMDS '12, pages 151--166, 2012.Google ScholarCross Ref
- J. Pinggera, S. Zugal, and B. Weber. Investigating the process of process modeling with cheetah experimental platform. In Proc. ER-POIS '10, pages 13--18, 2010.Google Scholar
- K. E. Stanovich and R. West. Individual differences in reasoning: implications for the rationality debate? Behavioural and Brain Sciences, 23(5): 665--726, 2000.Google ScholarCross Ref
- J. Sweller. Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2): 257--285, 1988.Google ScholarCross Ref
- W. J. Tracz. Computer programming and the human thought process. Software: Practice and Experience, 9(2): 127--137, 1979.Google ScholarCross Ref
- W. P. Vogt. Dictionary of Statistics & Methodology: A Nontechnical Guide for the Social Sciences (Fourth Edition). SAGE Publications, 2011.Google Scholar
- C. Wohlin, M. Höst, and K. Henningsson. Empirical research methods in software engineering. In Empirical Methods and Studies in Software Engineering, volume 2765 of LNCS, pages 7--23. Springer, 2003.Google Scholar
- C. Wohlin, R. Runeson, M. Halst, M. Ohlsson, B. Regnell, and A. Wesslen. Experimentation in Software Engineering: an Introduction. Kluwer, 2000. Google ScholarDigital Library
- S. Zugal, J. Pinggera, J. Mendling, H. Reijers, and B. Weber. Assessing the Impact of Hierarchy on Model Understandability---A Cognitive Perspective. In Proc. EESSMod '11, pages 123--133, 2011. Google ScholarDigital Library
- S. Zugal, J. Pinggera, and B. Weber. Assessing process models with cognitive psychology. In Proc. EMISA '11, pages 177--182, 2011.Google Scholar
- S. Zugal, J. Pinggera, and B. Weber. Creating Declarative Process Models Using Test Driven Modeling Suite. In Proc. CAiSE Forum '11, pages 16--32, 2011.Google Scholar
- S. Zugal, J. Pinggera, and B. Weber. The impact of testcases on the maintainability of declarative process models. In Proc. BPMDS '11, pages 163--177, 2011.Google ScholarCross Ref
- S. Zugal, J. Pinggera, and B. Weber. Toward Enhanced Life-Cycle Support for Declarative Processes. Journal of Software: Evolution and Process, 24(3): 285--302, 2012.Google ScholarCross Ref
- S. Zugal, P. Soffer, J. Pinggera, and B. Weber. Expressiveness and Understandability Considerations of Hierarchy in Declarative Business Process Models. In Proc. BPMDS '12, pages 167--181, 2012.Google ScholarCross Ref
Index Terms
- Making the case for measuring mental effort
Recommendations
Measuring Mental Effort via Entropy in VR
IUI '20 Companion: Companion Proceedings of the 25th International Conference on Intelligent User InterfacesRecognizing changes in users' experienced mental effort is a perennial interest in human-computer interaction research particularly in the design of intelligent user interfaces built to adapt to different levels of mental effort. With virtual reality (VR)...
Physiological correlates of mental effort as manipulated through lane width during simulated driving
ACII '15: Proceedings of the 2015 International Conference on Affective Computing and Intelligent Interaction (ACII)Previous studies suggest that physiological effects of mental effort as manipulated trough cognitive task difficulty differ from effects of mental effort as manipulated trough a visuomotor task such as lane keeping in simulated driving. Most notably, ...
Quantitative Analysis of Mental Effort Investment using Photoplethysmogram
BIOSTEC 2014: Proceedings of the International Joint Conference on Biomedical Engineering Systems and Technologies - Volume 5Recent studies have shown a close relationship between the mental effort and the photoplethysmograms (PPG), but lack an index which can analyze the invested mental effort quantitatively during the cognitive tasks, which is notable for evaluating user ...
Comments