Full length articleUsing multi-channel data with multi-level modeling to assess in-game performance during gameplay with Crystal Island
Introduction
Game-based learning environments (GBLEs) have been touted as a solution for failing educational outcomes across several domains. Learning with GBLEs can be particularly effective for learning because they are designed to foster engagement during learning (e.g., Sabourin et al., 2013, Sabourin and Lester, 2014). Additionally, many games require self-regulated learning, in addition to other learning processes, such as scientific reasoning (Millis et al., 2011). Scientific reasoning involves generating and testing hypotheses, and therefore students use self-regulatory processes to assist in generating and testing these hypotheses. For example, during gameplay with Crystal Island, students are required to gather clues, and create and test hypotheses, to solve a mystery. It can thus be beneficial to situate theories of SRL with scientific reasoning to investigate learning with GBLEs.
Despite the widespread enthusiasm, many critics have raised serious issues regarding the effectiveness of GBLEs for learning and problem solving (Mayer, 2015, Shute and Ventura, 2013). Unfortunately, the majority of published studies suffer from conceptual, theoretical, methodological, and analytical issues, undermining the value of GBLEs for improving learning, problem solving, and transfer of knowledge and skills across domains and age groups. Recent calls have been made to improve the quality of GBLE research by using theoretically-driven approaches and interdisciplinary methods and analytical techniques to comprehend the cognitive, affective, metacognitive, and motivational processes simultaneously during gameplay to understand their roles and impact other than the typical approach of using pre-to post-test measures and self-reports of motivation and engagement (Mayer, 2014). In this study, we address some of these major issues by using eye movements and log files to examine the cognitive and metacognitive self-regulatory processes deployed by college students while playing Crystal Island, a GBLE that incorporates microbiology content and scientific reasoning to solve the mystery of what disease has spread through a fictional remote island.
Research on self-regulated learning (SRL) indicates that students are self-regulating when they adaptively respond to both internal (e.g., use cognitive strategies during scientific reasoning) and external conditions (e.g., navigate a game environment in search of evidence) as evidenced by accurate monitoring and effective regulation of their cognitive, affective, metacognitive, and motivational processes during learning, problem solving, and performance (Azevedo et al., 2011, Azevedo et al., 2015, Winne and Azevedo, 2014, Winne and Hadwin, 1998, Winne and Hadwin, 2008, Zimmerman and Schunk, 2011). Although research has shown that engaging in cognitive, affective, metacognitive, and motivational self-regulated learning processes can be beneficial for learning (Azevedo, 2009, Azevedo, 2014, Pintrich, 2000, Schunk and Greene, in press), research has also revealed that students do not typically deploy these processes effectively and efficiently during learning with advanced learning technologies such as intelligent tutoring systems, hypermedia, multimedia (see Azevedo et al.,, Azevedo et al., 2015, Azevedo et al., 2011, Graesser, 2015, VanLehn, 2016). Recent work on GBLEs and self-regulated learning has been conducted by Lester and colleagues (e.g., Sabourin & Lester, 2014) to examine if gameplay behaviors are predictive of learning, performance, engagement, and motivation using traditional statistics, data mining and machine learning. The current study extends this work by converging eye movements and log files to examine the underlying cognitive and metacognitive processes used by college students to solve the mystery on Crystal Island.
Winne and Hadwin, 1998, Winne and Hadwin, 2008 Information Processing Theory (IPT) was used as the theoretical framework for the current study, which posits that learning occurs through a series of four cyclical phases, and information processing can occur within each phase. In the first phase, task definition, students must develop task understanding that drives their planning, monitoring and regulatory processes. In Crystal Island, students must understand the overall goal for the task, which is to solve the science mystery. In the second phase (goals and plans), students set goals for how they will accomplish the task (e.g., gather clues in each building) and plan how they will accomplish those goals (e.g., read books, complete embedded assessments). The third phase, strategy-use, is when the students enact the plans to accomplish the goals they set in the previous phase (e.g., when students actually read the books and complete the embedded assessments). Strategy use and metacognitive monitoring can be inferred by analyzing in-game behaviors collected through eye movements and log files. The fourth phase (adaptation) is not addressed in this study. It is important to note that these phases are not necessarily sequential, and students can engage in multiple phases simultaneously, and in any order.
Information processing includes students engaging in cognitive, affective, metacognitive, and motivational processes to effectively self-regulate their learning (Azevedo et al., 2011, Azevedo et al., 2015, Azevedo et al.,), and these processes are related to monitoring and control. For example, students can monitor their use of strategies based on making metacognitive judgments (i.e., is completing the in-game assessment an efficient strategy if the student does not understand the material), and return to the goals and plans phase to dynamically monitor and control their use of strategies (i.e., re-reading a book as a cognitive learning strategy). Therefore, throughout all the phases of learning, self-regulation implies that students engage in monitoring and control of self-regulated learning strategies.
Our use of Winne and Hadwin, 1998, Winne and Hadwin, 2008 model is advantageous because even though it has yet to be empirically tested, it is the only model that assesses SRL as an event that temporally unfolds over time (Winne & Azevedo, 2014). The temporality of SRL is especially important during learning with GBLEs because students are presented with complex material, and their use of SRL strategies can change depending on the context. For example, during learning of complex text within a hypermedia-learning environment, we operationalize judgments of learning (JOLs) as assessing one’s understanding of the text by having them make that judgment, followed by a content quiz (Greene & Azevedo, 2009). When making these judgments, there is a valence associated with it, such that a high rating of understanding would be a JOL+, and a JOL-would be indicative of low understanding. Although this might seem specific to hypermedia-learning environments, this can be applied to learning with GBLEs as well. During gameplay with Crystal Island, one activity students can engage in is reading books, which are associated with embedded assessments called concept matrices (Rowe, Shores, Mott, & Lester, 2011). In each concept matrix, there are questions that test students on their understanding of the material in the book. Therefore, this can be seen as a JOL because students can self-evaluate their understanding of the text, and go on to complete the concept matrix to test if they did understand the text. Furthermore, we can investigate the valence of the JOL based on the correctness of the responses. Thus, as opposed to the valence being associated with the student’s judgment of their understanding, the valence can be associated with how well the student performed on the assessment, such that low performance has a negative valence, and high performance has a positive valence. Therefore, we can apply IPT, specifically metacognitive monitoring and control to gameplay and scientific reasoning with GBLEs.
Section snippets
Game-based learning environments
The effectiveness of GBLEs across domains (e.g., math, computer science, biology, psychology, etc.) has come into question as several meta-analyses (Clark et al., 2016, Connolly et al., 2012, Girard et al., 2012, Mayer, 2014, Wouters et al., 2013) have revealed different results. More specifically, they have revealed that learning with GBLEs results in small to medium effect sizes for knowledge acquisition (d = 0.29, p < 0.01; Wouters et al., 2013), yet moderate to large effect sizes for
Current study: assessing and converging multi-channel data with Crystal Island
The current study extends previous published research on GBLEs and the work on Crystal Island by Lester and colleagues by using eye tracking and log files to assess college students’ cognitive and metacognitive SRL processes during gameplay while using Crystal Island. More specifically, we converged specific in-game behaviors with eye tracking to assess how well students performed on in-game embedded assessments during learning and gameplay with GBLEs. As such, the goal of the current study was
Participants
Fifty (N = 501) non-biology majors (56% female) from a large public university located in the southeast region of the US
Results
We used SAS software 9.4 (SAS Institute Inc., 2012) to run our analyses, with a restricted maximum likelihood (REML) estimation method, and a variance components covariance structure. The first step in using MLM requires a fully unconditional model, which allows us to determine if there is sufficient between- and within-subjects variance in our dependent variable, and the intra-class correlation coefficient (i.e., the percentage of variance explained at the between- and within-subject levels),
Discussion
Overall, results from the log-file data indicate that the number of books read and the frequency of reading each book were both negatively related to concept matrix submission attempts, when assessing their unique associations with the number of concept matrix submission attempts. In addition to main effects of each variable, there was a significant interaction between both variables, such that reading fewer books, but reading each book more frequently was associated with fewer attempts, and
Acknowledgment
The research presented in this paper has been supported by funding from the Social Sciences and Humanities Research Council of Canada (SSHRC 895–2011–1006) awarded to the third and last authors. The authors would like to thank Robert Taylor and Andrew Smith for assisting with system development, and Megan Price for assisting with running participants.
References (81)
- et al.
The effect of competition on learning in games
Computers & Education
(2015) - et al.
A systematic literature review on serious games evaluation: An application to software project management
Computers & Education
(2015) - et al.
A systematic literature review of empirical evidence on computer games and serious games
Computers & Education
(2012) - et al.
Dynamics of affective states during complex learning
Learning and Instruction
(2012) - et al.
A macro-level analysis of SRL processes and their relations to the acquisition of sophisticated mental models
Contemporary Educational Psychology
(2009) - et al.
Learning to see: Guiding students’ attention via a model’s eye movements fosters learning
Learning and Instruction
(2013) - et al.
Designing game-based learning environments for elementary science education: A narrative-centered learning perspective
Information Sciences
(2014) Unique contributions of eye-tracking research to the study of learning with graphics
Learning and Instruction
(2010)The role of goal orientation in self-regulated learning
- et al.
Game-based learning and 21st century skills: A review of recent research
Computers in Human Behavior
(2016)
Situational interest in literary text
Contemporary Educational Psychology
Does agency matter? Exploring the impact of controlled behaviors within a game-based environment
Computers & Education
Visual behavior, blow and achievement in game-based learning
Computers & Education
Attention guidance during example study via the model’s eye movements
Computers in Human Behavior
Narrative games for learning: Testing the discovery and narrative hypotheses
Journal of Educational Psychology
Theoretical, methodological, and analytical challenges in the research on metacognition and self-regulation: A commentary
Metacognition & Learning
Multimedia learning of metacognitive strategies
Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues
Educational Psychologist
Using trace data to examine the complex roles of cognitive, metacognitive, and emotional self-regulatory processes during learning with multi-agent systems
Use of hypermedia to convey and assess self-regulated learning
Technologies supporting self-regulated learning
Inferring learning from gaze data during interaction with an environment to support self-regulated learning
Digital games, design, and learning: A systematic review and meta-analysis
Review of Educational Research
Location information within extended hypermedia
Educational Technology Research and Development
Giving eyesight to the blind: Towards attention-aware AIED
International Journal of Artificial Intelligence in Education
On the measurement of achievement goals: Critique, illustration, and application
Journal of Educational Psychology
Engagement as a volitional construct: A framework for evidence-based research on educational games
Simulation & Gaming
Serious games as new educational tools: How effective are they? A meta-analysis of recent studies
Journal of Computer Assisted Learning
Deeper learning with advances in discourse science and technology
Policy Insights from Behavioral and Brain Sciences
Goal recognition with Markov logic networks for player-adaptive games
Do adult readers know how they read? Evidence from eye movement patterns and verbal reports
British Journal of Psychology
Attention tool (version 6.0)
Predicting affect from gaze data during interaction with an intelligent tutoring system
Director agent intervention strategies for interactive narrative environments
A supervised learning framework for modeling director agent strategies in educational interactive narrative
IEEE Transactions on Computational Intelligence and AI in Games
Serious games get smart: Intelligent game-based learning environments
AI Magazine
Supporting self-regulated science learning in narrative-centered learning environments
Computer games for learning: An evidence-based approach
On the need for research evidence to guide the design of computer games for learning
Educational Psychologist
Cited by (57)
Progression of students' SRL processes in subsequent programming problem-solving tasks and its association with tasks outcomes
2023, Internet and Higher EducationCitation Excerpt :Highly effective learners are those who engage in SRL processes productively (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013). SRL skills help learners to decide when and how to find a solution to a complex learning task (Taub et al., 2017). Research indicates that people who plan, monitor, evaluate, and reflect on their problem-solving processes solve problems better than those who do not use them, which applies from a very young age up to college (Stillman & Mevarech, 2010).
Self-Regulated Learning and Working Memory Determine Problem-Solving Accuracy in Math
2022, Spanish Journal of PsychologyEffectiveness of AI-assisted game-based learning on science learning outcomes, intrinsic motivation, cognitive load, and learning behavior
2024, Education and Information TechnologiesAssessing the Complexity of Gaming Mechanics During Science Learning
2024, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)Towards scaffolding self-regulated writing: implications for developing writing interventions in first-year writing
2023, Metacognition and LearningAnalysing domain-specific problem-solving processes within authentic computer-based learning and training environments by using eye-tracking: a scoping review
2023, Empirical Research in Vocational Education and Training