Full length articleLost in transition – Learning analytics on the transfer from knowledge acquisition to knowledge application in complex problem solving☆
Section snippets
Complex problem solving and knowledge transition
One 21st century skill that has received considerable attention in recent educational research is CPS (Greiff et al., 2013; Herde, Wüstenberg, & Greiff, 2016; Schweizer, Wüstenberg, & Greiff, 2013). Complex problem solving can be defined as the ability to solve problems with dynamic, hidden, or intertwined features. Such problems can be encountered on any societal level, from those with global impact such as, for instance, climate change (e.g., Urry, 2008), to individual ones referring to the
Sample characteristics
We analyzed the log files of a large-scale dataset of 9th grade students in Finland.1
RQ1: How many students who solved the knowledge acquisition phase fail in the knowledge application phase?
Our first RQ addresses the question to what extent the problem of being ‘lost in transition’ is generally occurring. Therefore, we analyzed the relative frequencies of students who, after successfully solving the knowledge acquisition phase, did not manage to perform successfully in the knowledge application phase. The results of this analysis can be seen in Fig. 3 below, indicating that, across all nine MicroDYN items, a significant proportion of students fell into this group (previously
Discussion
The overarching aim of this study was twofold. To begin with, we wanted to uncover the magnitude of what we call the ‘lost in transition’ phenomenon, which refers to the case that a participant successfully explores the hidden variable relations in the knowledge acquisition phase, but fails in reaching the predefined target goals in the following knowledge application phase (i.e., Group B as shown in Fig. 2), in the particular showcase of CPS. In addition, we wanted to investigate potential
Conclusion
The primary aim of this study was to uncover the magnitude of the ‘lost in transition’ phenomenon, referring to the case that a participant who successfully acquires knowledge fails to subsequently apply this knowledge, in CPS. In addition, we discussed some potential reasons for why this phenomenon arises. Overall, we can conclude that a significant amount of students experiences being ‘lost in transition’, and that the probability of this phenomenon occurring particularly depends on the
Credit author statement
Björn Nicolay, Conceptualization, Methodology, Formal analysis, Writing - original draft, Writing - review & editing, Visualization. Florian Krieger, Conceptualization, Methodology, Formal analysis, Writing - review & editing, Resources, Visualization. Matthias Stadler, Conceptualization, Methodology, Validation, Writing - review & editing. Janice Gobert, Conceptualization, Resources, Writing - review & editing. Samuel Greiff, Conceptualization, Software, Resources, Writing - review & editing,
Declaration of competing interest
Samuel Greiff is one of two authors of the commercially available COMPRO-test that is based on the multiple complex systems approach, which employs the same assessment principle as MicroDYN. He receives royalty fees for COMPRO.
Acknowledgments
This research was supported by a grant from the Fonds National de la Recherche Luxembourg (Luxembourgish National Research Found) [CORE ‘TRIOPS’]. We would like to explicitly thank our Finnish colleagues for the possibility to analyze this large-scale dataset, which is based on Vainikainen (2014).
References (86)
- et al.
Automatic coding procedures for collaborative problem solving
Acquisition of cognitive skill
Psychological Review
(1982)- et al.
Learning rapid and precise skills
Psychological Review
(2019) - et al.
Extending problem-solving procedures through reflection
Cognitive Psychology
(2014) - et al.
Mixed-effects modeling with crossed random effects for subjects and items
Journal of Memory and Language
(2008) - et al.
Thinking about applications: Effects on mental models and creative problem-solving
Creativity Research Journal
(2013) - et al.
Fitting linear mixed-effects models using lme4
Journal of Statistical Software
(2015) Basic topics and approaches to the study of complex problem solving
- et al.
Working memory, visual–spatial-intelligence and their relationship to problem-solving
Intelligence
(2008) - et al.
Cognitive flexibility and adaptability to environmental changes in dynamic complex problem-solving tasks
Ergonomics
(2003)
Developing and assessing ERP competencies: Basic and complex knowledge
Journal of Computer Information Systems
All other things being equal: Acquisition and transfer of the control of variables strategy
Child Development
Progression of mental models throughout the phases of a computer-based instructional simulation: Supportive information, practice, and performance
Computers in Human Behavior
The role of planning in complex problem solving
Computers & Education
Ability- or competence-based education: Bridging the gap between knowledge acquisition and ability to apply
Higher Education
An approach to training based upon principled task decomposition
Acta Psychologica
Microworlds based on linear equation systems: A new approach to complex problem solving and experimental results
Dynamic systems as tools for analysing human judgement
Thinking & Reasoning
Complex problem solving
Operationalizing and detecting disengagement within online science microworlds
Educational Psychologist
From log files to assessment metrics: Measuring students' science inquiry skills using educational data mining
The Journal of the Learning Sciences
Practice under changing priorities: An approach to the training of complex skills
Acta Psychologica
Problem solving in technical domains: How mental models and metacognition affect performance
Instruction: Theoretical and applied perspectives
Advancing the science of collaborative problem solving
Psychological Science in the Public Interest
Assessing complex problem- solving skills with multiple complex systems
Thinking & Reasoning
The systematic variation of task characteristics facilitates the understanding of task difficulty: A cognitive diagnostic modeling approach to complex problem solving
Psychological Test and Assessment Modeling
Students' exploration strategies in computer-simulated complex problem environments: A latent class approach
Computers & Education
Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioural data from computer-generated log files
Computers in Human Behavior
Sometimes less is more: Comparing the validity of complex problem solving measures
Intelligence
Computer-generated log file analyses as a window into students' minds?: A showcase study based on the PISA 2012 assessment of problem solving
Computers & Education
Dynamic problem solving: A new assessment perspective
Applied Psychological Measurement
Complex problem solving in educational contexts—something beyond g: Concept, assessment, measurement invariance, and construct validity
Journal of Educational Psychology
Mental models and problem solving in using a calculator
Designing knowledge-in-use assessments to promote deeper learning
Educational Measurement: Issues and Practice
The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans?
Computers in Human Behavior
Assessment of complex problem solving: What we know and what we don't know
Applied Measurement in Education
IBM SPSS statistics for windows, version 25.0
JASP
How to improve dynamic decision making? Practice and promise
Twenty-first century students need 21st century skills
Learning analytics to support self- regulated learning in asynchronous online courses: A case study at a women's university in South Korea
Computers & Education
What you train is what you get? Task requirements and training methods in complex problem-solving
Computers in Human Behavior
Cognitive learning efficiency through the use of design patterns in teaching
Computers & Education
Cited by (11)
Understanding transitions in complex problem-solving: Why we succeed and where we fail
2023, Thinking Skills and CreativityFocusing on eigendynamic effects promotes students’ performance in complex problem solving: A log-file analysis of strategic behavior
2022, Computers and EducationCitation Excerpt :This pattern of findings suggests that more intelligent (compared to less intelligent) students achieved higher CPS performance scores in the second CPS phase by adequately identifying eigendynamics (first comparison), but regardless of whether they applied the strategic behavior to identify eigendynamic effects early or not (second comparison). Possibly, the strategic behavior to adequately identify eigendynamics and integrate the gained information in the first CPS phase helped more intelligent students to transfer their knowledge acquired in the first CPS phase to the second CPS phase: a process at which surprisingly many students fail (Nicolay et al., 2021). An explanation for the non-significant mediation effect of the second comparison could be that the strategic behavior to identify eigendynamic effects early might be too distant to the second CPS phase to mediate the intelligence-CPS relationship in this CPS phase.
Towards strengthening links between learning analytics and assessment: Challenges and potentials of a promising new bond
2022, Computers in Human BehaviorInterdisciplinary frontiers: computer-based process data analysis in educational measurement
2022, International Encyclopedia of Education: Fourth EditionHow we explore, interpret, and solve complex problems: A cross-national study of problem-solving processes
2022, HeliyonCitation Excerpt :Sentences which were subject to different interpretations were further discussed among the researchers and translators. The CPS approach and the CPS tasks have been employed extensively at both the national and international levels (see Csapó and Funke, 2017; Eichmann et al., 2020; Greiff et al., 2013b; Greiff et al., 2015b; Mustafic et al., 2019; Nicolay et al., 2021; OECD, 2014a). The psychometric indices of the test proved to be good, independent of the cultures and nations (see Wüstenberg et al., 2014; Wu and Molnár, 2021).
- ☆
This research was supported by a grant from the Fonds National de la Recherche Luxembourg (Luxembourgish National Research Found) [CORE ‘TRIOPS’].