Gamified performance assessment of collaborative problem solving skills

https://doi.org/10.1016/j.chb.2019.05.033Get rights and content

Highlights

  • Initial validation for the five skills underlying the design of the game.

  • Preliminary support for the use of the game for assessment of the intended CPS skills.

  • Results support the validity of the use of virtual agents in CPS assessments.

Abstract

In this paper we introduce a game-based approach for Collaborative Problem Solving (CPS) Skills assessment and provide preliminary evidence from a validation pilot study. To date, educational assessments have focused more heavily on the concrete, and accessible aspects of CPS with a diminished representation of the social aspects of CPS. We addressed this issue through the integration of our CPS construct into the game-based assessment “Circuit Runner” in which participants interact with a virtual agent to solve a series of challenges in a first-person maze environment (von Davier, 2017). Circuit Runner provides an environment that allows for controlled interdependence between a user and a virtual agent that facilitates the demonstration of the broad range of cognitive and social skills required for effective CPS. Tasks are designed to incorporate telemetry-based (e.g., log file, clickstream, interaction-based) and item response data to provide a more comprehensive measure of CPS skills. Our study included 500 participants on Amazon Mechanical Turk, who completed Circuit Runner, pre- and post-game surveys, and a CPS situational judgment test (CPS-SJT). These elements, in conjunction with the game-play, allowed for an expanded exploration of CPS skills with different modalities and types of instruments. The findings support and extend efforts to provide a stronger theoretical and empirical foundation for insights regarding CPS as a skillset, as well as the design of scalable game-based CPS assessments.

Introduction

The role of technology in advancing the field of education is both disruptive and promising. Technology continues to allow us to explore areas of education and learning that were previously obscured and to provide insights that were previously not accessible. This is especially true regarding the measurement of 21st century skills. These notoriously multidimensional and complex synchronizations of social and cognitive skills present a number of assessment design and measurement challenges (Graesser et al., 2018, Rosen, 2015), but also a number of opportunities to gain insights and make advancements in this area (Rosen, 2017, Schleicher, 2018, von Davier et al., 2017). It is technology, and specifically computer-based assessment, that allows us to make important advancements toward realizing those opportunities.

The collaborative problem solving (CPS) construct has been the focus of many efforts to address the challenges of teaching and measuring 21st century skills (von Davier, 2017). The complexities that arise around authenticity in both the assessment design and measurement of CPS (Care and Kim, 2018, Graesser et al., 2018), and around our ability to effectively measure both the cognitive and the social components that are required for successful CPS, require a well-defined construct that provides ample opportunity to address these challenges. The importance of these efforts was highlighted by the selection of CPS as a major area of focus by the Organisation for Economic Co-operation and Development (OECD) for the 2015 Programme for International Student Assessment (PISA) (OECD, 2017a). The most notable recent efforts to address these challenges have been through the frameworks developed by Assessment and Teaching for 21st Century Skills (ATC21s) (Hesse, Care, Buder, Sassenberg, & Griffin, 2015), and the PISA 2015 Collaborative Problem Solving Framework (OECD, 2017b).

In the case of collaborative problem solving, it is valuable to parse social skills from cognitive skills. Parsing social skills from cognitive skills allows for the ability to consider the role that these skills play across the full range of processes required for effective collaborative problem solving. For example, persistence as a social skill supports the social functions that require persistence such as challenging interactions, negotiations, and the pursuit of role or goal clarity. Persistence as a social skill also supports task-specific functions such as persisting through the execution of difficult tasks, the monitoring of task progress, and the evaluation of completed work. Additionally, acknowledging the variance that can occur in the demonstration of those skills and the degree to which the variance can be attributed to knowledge, ability, or propensity adds additional measurement complexity.

The interdependence required for collaborative problem solving also adds complexity to the measurement of CPS. Interdependence infuses the otherwise independent experience of problem solving, with the need to now negotiate, share, and capitalize on the resources that create the interdependence. This introduces the need for a range of inter- and intra-personal skills, processes, and behaviors to be employed to achieve successful collaborative problem solving.

Insights regarding the behaviors that support CPS are often limited to self-report and observation methods. Computer-based assessments (CBA), as opposed to traditional pencil-and paper based assessments, allow for new test–taking environments that have the potential to be less invasive and more authentic than pencil-and-paper based assessments for the demonstration and measurement of these skills. The measurement of CPS behaviors in these new environments is often captured through the review of commonly collected log file data (e.g. click-steam data) and often focuses on temporal and more specifically response time based criteria (Adams et al. (2015) Goldhammer et al., 2014, Zoanetti, 2010). CBA environments along with computational psychometrics are allowing for the blending of theory-driven methods with data-driven knowledge to allow for deeper knowledge discovery in this area (Graesser et al., 2018, Dordrecht Kirschner et al., 2018, Schleicher, 2018, von Davier, 2017).

To capitalize on the advantages presented by a CBA to create an interdependent collaborative problem solving environment in which we can address the social and cognitive skills involved in CPS we created Circuit Runner. Circuit Runner immerses participants in a first-person CPS experience requiring interdependence between a participant and a virtual agent to navigate and solve challenges in a 3D representation of a maze environment. Tasks are designed to elicit the demonstration of both explicit and implicit cognitive and social skills, which allows for both item response data (response selection) and telemetry-based (e.g., log file, clickstream, online environment interaction-based) data to be collected to support our ability to isolate and explore the skills, behaviors, and processes involved in CPS. These features also allow for improved possible insights into the complex factors surrounding the propensity to demonstrate these skills, particularly with respect to behaviors.

The tasks included in Circuit Runner explicitly employ the use of virtual agents to address issues surrounding generalizability. Through the use of a human-agent model, where a human participant interacts with a virtual agent, we are able to control for both the skills elicited from the participants and for the exploration of an improved range of skill level proficiencies demonstrated by participants (Rosen & Tager, 2013; Rosen, 2015). This range has the potential to more closely represent the range of skill levels both experienced in real-world teams and required for success in real-world teams and team environments. The virtual agents also allow for the creation of controlled interactions in which all participants are experiencing and responding the same social or task challenges. The level of challenge for these interactions can also then be controlled.

We have also capitalized on the CBA environment to address issues related to authenticity and propensity, particularly with respect to behaviors. The rich data trail available in CBA environments that are designed from an evidence elicitation perspective involves copious details regarding the explicit and implicit choices and demonstrations made by participants. Decoding the data trail elicited by these innovative task designs requires multiple and complex methods and psychometric approaches (Adams et al., 2015, Goldhammer et al., 2014, Zoanetti, 2010). One of the approaches that we have used to analyze subsets and aggregated information from this rich data stream is item response theory (IRT). An important feature of IRT is that it allows for handling incomplete data in a very efficient way. That is, for each respondent, we can use all the available data on those indicators that were observed for that respondent for the measurement of the CPS skills, and it is not required that an indicator is present for all respondents (under the usual assumptions for the IRT models). Furthermore, IRT allows us to combine indicators of different types, for example one indicator could be whether a certain action was taken (i.e., a binary indicator), while another indicator could be a selected response to a prompt from the computer agent where different responses refer to different levels of the measured skill (i.e., an ordinal indicator).

Telemetry data is collected throughout the game experience as a means to explore evidence of the demonstration of behaviors and processes. Some of the game interactions collected as telemetry represent interactions, and possible representations, of skills or behaviors, that are obvious to both the participants and those reviewing the log file data. For example, participants are aware that the clicking of specific buttons, or game features, and the duration of time spent in a room or on a task may indicate their persistence to complete the task. Some interactions, however, are designed to gather telemetry data to make inferences about a skill that the participant is not explicitly aware is being measured. These data, combined with response option data aligned with those same behaviors, allows insights regarding both the participants’ explicit demonstration of the appropriate application of those behaviors, and also their implicit demonstration of those behaviors as captured in the telemetry of processes, tasks and interactions. This extends beyond the more common practice of applying a-priori expectations for common telemetry and observable behaviors to CPS log file data (Adams et al., 2015, Zoanetti, 2010), to instead explore the demonstration of those behaviors through tasks and telemetry-based log file data collection methods designed specifically to elicit and measure varying levels of those behaviors.

These elements allowed us to derive insights and skill level data based on response data and telemetry collected through the normal game play interactions. Game and scoring design elements are discussed in this paper, as are findings from the study. A discussion regarding the impact of this work on the direction for future research is also included.

Section snippets

Collaborative problem solving skills

CPS is a critical competency for college and career readiness and is generally considered to be one of the critical components of a 21st century skill set (Fiore et al., 2018, Graesser et al., 2017, OECD, 2017a, O'Neil and Chuang, 2008, Rosen and Rimor, 2012). Our operational definition of CPS refers to the knowledge, skills, and behaviors required to effectively participate in a joint activity requiring interdependence among participants to transform a current state to a goal state.

The CPS

Game design

In the Circuit Runner game environment, a participant collaborates with a virtual agent to navigate a circuit board maze from a first-person perspective. The participant enters the maze while the virtual agent is understood to stay at a base location which contains additional resources that the team will need to navigate the maze. This creates an interdependence between the participant and the virtual agent based on unequal distribution of knowledge and resources. Within the maze, the

Research questions

The study was focused on exploring feasibility and validity of measuring CPS skills via a CPS gamified assessment – Circuit Runner. More specifically, the following research questions were examined empirically:

To what extent does a CPS gamified assessment provide valid and reliable data on a broad range of CPS skills, including: Reaching the Goal, Persistence, Problem Feature Awareness, Perspective Taking, and Strategy? How does the quality of measurement change with the inclusion of the

Methods

A total of 500 unique users participated in the study through the Amazon MTurk platform. Of the 500 study participants, 379 provided a complete dataset (mean age = 33.76, SD = 9.14; 45% female). Participants ranged in age from 18 to 68 with the majority of participants (63.32%) falling between the ages of 25 and 40. The majority of the participants had completed some form of post high-school educational training, with only 28% completing high school or below. The majority of participants were

CPS assessment instruments

Participants were directed that the study should take less than 3 h to complete their interaction with the CPS gamified assessment, CPS-SJT and CPS self-report questionnaire. The CPS-SJT was designed to measure three skills from the CPS construct: Strategy, Perspective Taking, and Problem Feature Awareness. The format of the CPS-SJT includes three sections. Each section begins with a video that provides situational context for a workplace scenario that features elements relating to CPS skills.

Results

On average, participants interacted with the Circuit Runner game for 31 min. As described above, the game has been designed to measure five key skills within the HF CPS construct. Factor and reliability analysis of the common node game data revealed initial validation for the five skills underlying the design of the game: Reaching the Goal 9 items (0.72), Persistence 9 items (0.65), Problem Feature Awareness 8 items (0.49), Perspective Taking 9 items (0.59), and Strategy 8 items (0.63). The

Discussion

The importance of facilitating learners’ ability to acquire and cultivate these skills over time is essential and depends on our ability to identify, measure, and track proficiency with the skills that support these critical competencies. This is an ambitious endeavor, as these notoriously multidimensional and complex synchronizations of social and cognitive skills present a number of construct design, assessment design, and measurement challenges.

Our current study is focused on some key

Conclusions

CPS is a construct that is difficult to measure. Our current work contributes to the exciting progress being made to explore the demonstration and measurement of CPS skills across a range of environments. Progress is currently being made in both the collection and understanding of response data in these environments, and the value of incorporating the analysis of process data made available by these environments to inform insights regarding CPS. Our understanding of the measurement of these

References (26)

  • R. Adams et al.

    Automatic coding procedures for collaborative problem solving

  • W. Camara et al.

    Beyond academics: A holistic framework for enhancing education and workplace success

  • E. Care et al.

    Assessment of twenty-first century skills: The issue of authenticity

  • R.P. Chalmers

    Mirt: A multidimensional item response theory package for the R environment

    Journal of Statistical Software

    (2012)
  • A.A. von Davier

    Computational psychometrics in support of collaborative assessments

  • P.A. Dordrecht Kirschner et al.

    Editorial to the special issue: Current innovations in computer-based assessments

    (2018)
  • S.M. Fiore et al.

    Collaborative problem-solving education for the twenty-first-century workforce

    Nature Human Behaviour

    (2018)
  • F. Goldhammer et al.

    The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment

    Journal of Educational Psychology

    (2014)
  • A.C. Graesser et al.

    Challenges of assessing collaborative problem solving

  • A. Graesser et al.

    Complex problem solving in assessments of collaborative problem solving

    Journal of Intelligence

    (2017)
  • F. Hesse et al.

    A framework for teachable collaborative problem solving skills

  • OECD

    Assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving, PISA

    (2017)
  • Cited by (0)

    View full text