Specification and evaluation of an assessment engine for educational games: Empowering educators with an assessment editor and a learning analytics dashboard
Introduction
Games-based learning (GBL) is increasingly used as a supplementary tool for education. GBL offers a variety of advantages to assist traditional teaching. They can, for instance, allow students to learn at their own pace and they are a safe and controlled environment for students to learn through trial and error. Various institutions use GBL for learning and training, ranging from schools [32] and higher education institutions [13] to healthcare [34], [51] and the army [29], [55].
However, while many teachers agree that GBL increases motivation towards learning [43] and despite the evidence that games are valid assessment tools [24], there seems to be a lack of trust in an educational game’s assessment [43], [48]. Teachers need to feel in control before introducing a new tool in the classroom and there is a need for ownership over the game [30]; without control, educators might feel threatened by a game rather than supported by it.
One of the main limitations of GBL is that educational games are too often distributed as “black-boxes”; they are closed and self-contained systems, making it difficult to modify or retrieve data from [47]. This can mean that the potential of the game and its attractiveness to educators are reduced. Indeed, in traditional teaching, improvisation and adaptation to students represent a key aspect of the educator’s role [27], however, teachers tend to lose this capacity with the introduction of a tool they cannot modify to suit the needs of their students. Then, they cannot retrieve data about the gameplays to appreciate whether their teaching goals have been met. Educators and researchers have very little insight about what the students learn through a computer game and how they interact with it. Learning Analytics (LA) is an emerging field based on data mining processes [50] that can provide such detailed reports about the gameplays; data from the gameplays of several educational games are collected and data mining algorithms allow conclusions to be drawn about the games and the players. However, due to the novelty of the field, presently very few papers exist on LA and its application in GBL and LA is still beyond the reach of most teachers [28].
Various platforms such as <e-Adventure> [53] or e-CLIL [21] provide educators with the ability to create and modify their own computer games; <e-Adventure> even includes a learning analytics module [36]. These games engines externalise content and assessment integration from the game’s code and partially address the problems identified previously. However, these engines were created for educators alone; they are not meant to be used when working with game developers and therefore only provide limited options in terms of game genres and assessment integration. Teachers sometimes lack the time to develop the games themselves or there is a need for a type of game not offered by such platforms.
To summarise, computer games are a powerful tool for learning and assessment but they are often underused by educators, particularly for assessment. We propose three key improvements that could be made for GBL to be more teacher-friendly. First, teachers should be given more control over the game and they should feel a sense of ownership toward the game. Second, the games should be made more flexible, allowing educators to modify and adapt them. The third improvement is the introduction of more detailed reports on the gameplays through LA that will provide teachers with an insight into the appropriateness of the assessment regime used in the game and their students’ learning outcomes. It would be optimal to look at all three improvements from a general point of view, addressing assessment integration as well as all other facets of a game such as content integration, story line, graphics and sounds. However, this paper focuses on assessment as it is central in the learning and teaching process.
The aim of this study is to develop and evaluate an assessment engine that would facilitate integrating these three improvements to educational games. In this paper, we present an assessment engine, EngAGe (an Engine for Assessment in Games), that is used by developers during the development of an educational game and it provides tools for educators after distribution of the game. Our approach is based on the externalisation of the assessment. The resulting modularity offers the possibility to modify the assessment logic via an online editor without interfering with the game mechanics and to retrieve information about the gameplays through an LA dashboard.
This paper is divided into five sections as follows. In Section 2, we present a summary of the literature on LA associated with educational games. In Section 3, we explain how EngAGe is used by educators, detailing the design for the assessment editor and the LA dashboard. In Section 4, we present the findings of an evaluation of the tool carried out with 31 educators. Finally, Section 5 draws conclusions and discusses future directions of our research.
Section snippets
Previous research
This section presents the findings of a literature review performed for this research and reviews the different approaches to using LA in GBL. No restriction was imposed on the dates of the papers, however, the oldest relevant study identified was published in 2011 reflecting how recent the topic of LA in games is. The following search terms were used: “learning analytics” AND game. The search was performed on 15 databases relevant to education, information technology and/or social science: ACM
Proposed approach: EngAGe
Section 1 identified two key problems related to educators’ use of GBL as an assessment tool. First, the assessment is embedded into the game’s code and educators cannot modify it to suit their students’ needs. Second, there is a lack of detailed reports about the students’ interactions during the gameplays and their assessment. Authoring tools [21], [52] partially addressed these problems by bypassing game developers and allowing educators to create and modify educational games themselves.
Evaluation of the tool
This section presents the evaluation of the engine with educators. A user study was carried out to collect educators’ opinions on EngAGe for visualising LA through the dashboard and modifying a game’s assessment using the editor. A summative evaluation was performed with 31 educators from various backgrounds to evaluate the tool’s usability and usefulness. The evaluation also allowed preliminary conclusions to be drawn on the educators’ change of opinion regarding GBL as an assessment tool
Conclusion and future work
Assessment is a crucial part of any teaching and learning process; it must be carefully integrated in educational games. This paper has presented the research project EngAGe (an Engine for Assessment in Games), its background and motivations. The engine is used by developers when creating educational games resulting in a separation of the assessment from the game’s mechanics. As a result of this modularity, educators can modify the game’s assessment and adapt it to their players via an online
References (55)
- et al.
Teachers and game-based learning: improving understanding of how to increase efficacy of adoption
Comput. Educ.
(2011) - et al.
Serious game based personalized healthcare system for dysphonia rehabilitation
Pervasive Mob. Comput.
(2017) - et al.
Visualisation of student learning model in serious games
Comput. Hum. Behav.
(2015) - et al.
Applying standards to systematize learning analytics in serious games
Comput. Stand. Interfaces
(2017) - et al.
Tracing a little for big improvements: application of learning analytics and videogames for student assessment
Procedia Comput. Sci.
(2012) - et al.
Application of learning analytics in educational videogames
Entertain. Comput.
(2014) - A. Bader-Natal, T. Lotze, Evolving a learning analytics platform, in: Paper Presented at the 1st International...
- et al.
An empirical evaluation of the system usability scale
Int. J. Human-Comput. Interact.
(2008) - P. Blikstein, Using learning analytics to assess students' behavior in open-ended programming tasks, in: Paper...
- P. Blikstein, Multimodal learning analytics, in: Paper Presented at the 3rd International Conference on Learning...
SUS-A quick and dirty usability scale
Information visualization
Readings in Information Visualization: Using Vision to Think
Integrating assessment, feedback and learning analytics in educational games: literature review and design of an assessment engine
Visualizing information on mobile devices
Computer
Game-based learning and gamification in initial teacher training in the social sciences: an experiment with MinecraftEdu
Int. J. Educ. Technol. Higher Educ.
Sample size estimation: how many individuals should be studied? 1
Radiology
From data mining to knowledge discovery in databases
AI Mag.
Serious games as edX MOOC activities
Paper Presented at the Global Engineering Education Conference (EDUCON)
Some psychometric and design implications of game-based learning analytics
Learning analytics: from theory to practice-data support for learning and teaching
Cited by (8)
A systematic review on the impacts of game-based learning on argumentation skills
2020, Entertainment ComputingCitation Excerpt :Research on digital GBL with media-enhanced narratives has recently become a topic of interest to learning scientists and educational technologists (e.g. [1,9,14,20,40,48,58,79]). GBL is known to be motivating and engaging to students across various educational levels [26], and they have rapidly become an important area of investigation within technology-enhanced learning environments due to their positive impacts on students’ engagement, motivation, and learning (see [15,20,39,54,57,107]). Many scholars have indicated the potential of digital GBL to support learners in terms of conceptual understanding, process skills and practices, epistemological understanding and player’s attitudes identity, and engagement [20,72,66].
Specification and evaluation of an assessment engine for educational games: Integrating learning analytics and providing an assessment authoring tool
2019, Entertainment ComputingCitation Excerpt :EngAGe was designed to be used by two target users: developers creating the games and educators using and modifying them. The authors evaluated EngAGe quantitatively with 31 educators [8]. The study shows encouraging results with both the assessment editor and the LA dashboard rated useful and easy to use.
Online Assessment in Higher Education: A Systematic Review
2023, Online Learning JournalRediscovering the Uptake of Dashboard Feedback: A Conceptual Replication of Foung (2019)
2022, Sustainability (Switzerland)A Tool Supported Approach for Teaching Serious Game Learning Analytics
2021, Proceedings - Frontiers in Education Conference, FIE