Specification and evaluation of an assessment engine for educational games: Empowering educators with an assessment editor and a learning analytics dashboard

Highlights

• Evaluation of an innovative tool for assessment in educational games (EngAGe)

• Used by developers to build assessment and learning analytics into games.

• Used by educators to modify educational games and visualise learning analytics.

• The engine’s usability and usefulness was positively evaluated by 31 educators.

• Findings suggest that EngAGe increases educators’ trust towards a game’s assessment.

Abstract

Assessment is a crucial aspect of any teaching and learning process. Educational games offer promising advantages for assessment; personalised feedback to students and automated assessment process. However, while many teachers agree that educational games increase motivation, learning and retention, few of them are ready to fully trust them as an assessment tool. We believe there are two main reasons for this lack of trust: educators are not given sufficient information about the gameplays, and many educational games are distributed as black-boxes, unmodifiable by teachers. This paper presents an assessment engine designed to separate a game and its assessment. It allows teachers to modify a game’s assessment after distribution and visualise gameplay data via a learning analytics dashboard. The engine was evaluated quantitatively by 31 educators. Findings were overall very positive: both the assessment editor and the learning analytics dashboard were rated useful and easy to use. The evaluation also indicates that, having access to EngAGe, educators would be more likely to trust a game’s assessment. This paper concludes that EngAGe can be used by educators effectively to modify educational games’ assessment and visualise gameplay data, and that it contributes to increasing their trust in educational games as an assessment tool.

Introduction

Games-based learning (GBL) is increasingly used as a supplementary tool for education. GBL offers a variety of advantages to assist traditional teaching. They can, for instance, allow students to learn at their own pace and they are a safe and controlled environment for students to learn through trial and error. Various institutions use GBL for learning and training, ranging from schools [32] and higher education institutions [13] to healthcare [34], [51] and the army [29], [55].

However, while many teachers agree that GBL increases motivation towards learning [43] and despite the evidence that games are valid assessment tools [24], there seems to be a lack of trust in an educational game’s assessment [43], [48]. Teachers need to feel in control before introducing a new tool in the classroom and there is a need for ownership over the game [30]; without control, educators might feel threatened by a game rather than supported by it.

One of the main limitations of GBL is that educational games are too often distributed as “black-boxes”; they are closed and self-contained systems, making it difficult to modify or retrieve data from [47]. This can mean that the potential of the game and its attractiveness to educators are reduced. Indeed, in traditional teaching, improvisation and adaptation to students represent a key aspect of the educator’s role [27], however, teachers tend to lose this capacity with the introduction of a tool they cannot modify to suit the needs of their students. Then, they cannot retrieve data about the gameplays to appreciate whether their teaching goals have been met. Educators and researchers have very little insight about what the students learn through a computer game and how they interact with it. Learning Analytics (LA) is an emerging field based on data mining processes [50] that can provide such detailed reports about the gameplays; data from the gameplays of several educational games are collected and data mining algorithms allow conclusions to be drawn about the games and the players. However, due to the novelty of the field, presently very few papers exist on LA and its application in GBL and LA is still beyond the reach of most teachers [28].

Various platforms such as <e-Adventure> [53] or e-CLIL [21] provide educators with the ability to create and modify their own computer games; <e-Adventure> even includes a learning analytics module [36]. These games engines externalise content and assessment integration from the game’s code and partially address the problems identified previously. However, these engines were created for educators alone; they are not meant to be used when working with game developers and therefore only provide limited options in terms of game genres and assessment integration. Teachers sometimes lack the time to develop the games themselves or there is a need for a type of game not offered by such platforms.

To summarise, computer games are a powerful tool for learning and assessment but they are often underused by educators, particularly for assessment. We propose three key improvements that could be made for GBL to be more teacher-friendly. First, teachers should be given more control over the game and they should feel a sense of ownership toward the game. Second, the games should be made more flexible, allowing educators to modify and adapt them. The third improvement is the introduction of more detailed reports on the gameplays through LA that will provide teachers with an insight into the appropriateness of the assessment regime used in the game and their students’ learning outcomes. It would be optimal to look at all three improvements from a general point of view, addressing assessment integration as well as all other facets of a game such as content integration, story line, graphics and sounds. However, this paper focuses on assessment as it is central in the learning and teaching process.

The aim of this study is to develop and evaluate an assessment engine that would facilitate integrating these three improvements to educational games. In this paper, we present an assessment engine, EngAGe (an Engine for Assessment in Games), that is used by developers during the development of an educational game and it provides tools for educators after distribution of the game. Our approach is based on the externalisation of the assessment. The resulting modularity offers the possibility to modify the assessment logic via an online editor without interfering with the game mechanics and to retrieve information about the gameplays through an LA dashboard.

This paper is divided into five sections as follows. In Section 2, we present a summary of the literature on LA associated with educational games. In Section 3, we explain how EngAGe is used by educators, detailing the design for the assessment editor and the LA dashboard. In Section 4, we present the findings of an evaluation of the tool carried out with 31 educators. Finally, Section 5 draws conclusions and discusses future directions of our research.