Keywords

1 Introduction

Usability refers the extent to which a product can be used with efficiency, effectiveness and satisfaction in a specific use context and where the user meets certain goals with the use of this product [1]. This topic is very important in educational applications, given that if it has a high level of usability, it would effectively support learning and cause a positive impact by motivating students to learn, therefore the success or failure of these applications [2].

The challenge of educational games is the fact that the subconsciousness of people normally transmits the feeling that something related to education will not be entertaining even if it is related with any word that sounds fun by itself, as the word “video game”, and that discourages people from using them [3]. One of the crucial factors is because usually, it is often to omit characteristics or features related to playability in most of the educational video games, as they pretend to have a strong educational intention [4]. For this reason, usability plays a significant role in those kinds of games, helping the developers improve the visual aspects or the playability of them, in order to achieve higher acceptance of the players.

In this study, we present a systematic literature review to identify guidelines of usability evaluation on mobile educational games for primary school students and the aspects that are considered relevant in the usability evaluation of mobile educational video games. Through this procedure, we can find out the state of the art of mobile educational games guidelines for usability evaluation. It is intended to serve as a literature base for future work that want to create new set of mobile educational games guidelines of usability evaluation.

The paper has the following structure. In Sect. 2, we describe the main concepts related to our topic. In Sect. 3, we present the methodology used to undertake this study. In Sect. 4, we present the results of our research. Finally, we present the conclusions in Sect. 5.

2 Background

2.1 Usability

According to the ISO/IEC 9126-1 standard [5], usability is defined as “the capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions.”

Another definition of usability is given by Nielsen [6], who defined it as “a quality attribute that assesses how easy user interfaces are to use.” This author mentioned that the word usability also refers to “methods for improving ease-of-use during the design process”, and defined it by 5 quality components:

  • Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?

  • Efficiency: Once users have learned the design; how quickly can they perform tasks?

  • Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency?

  • Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?

  • Satisfaction: How pleasant is it to use the design?

2.2 Usability Evaluation

An evaluation method is a procedure composed by a series of activities well defined with the purpose of collect user data related to the interaction of a final user with a software product and to understand how specific features of this software contributes to achieve a certain degree of usability [7].

Although there are several taxonomies to classify the usability evaluation methods, these can be classified broadly into two main groups: empirical methods and inspection methods [8].

The empirical methods are based on capturing and analyzing the usage data from a group of representative users. While these users perform a series of predefined tasks, an evaluator, that could be human or a specific software, is registering the results of their actions. From the analysis of these collected results, it can provide us valuable information to detect usability problems [7].

On the other hand, the inspection methods are executed by expert evaluators or designers, which do not require the participation of real end users. These methods are based on examining the usability aspects of the user interface with a set of guidelines. These guidelines not only can review the compliance level of certain usability attributes, but also can predict problems related to software interfaces into a heuristic evaluation [9].

2.3 Game-Based Learning

According to Pho and Dinscore, game-based learning is a trend that has been implemented in many settings including workplace training, education, and social media [10]. It converts users into designers of their own learning environment using video games as a means. Many researches show successful results from innovative educational practices mediated by video games. Also, these studies highlight the positive impact in reasoning ability of children’s education and the development of complex capabilities, such as leadership or cooperation using video games in Primary school students [11].

3 Systematic Literature Review

A systematic literature review is a methodology that identify, synthesize and interpret all available studies that are relevant to a research question formulated previously, or topic area, or phenomenon of interest [12]. Although systematic reviews require more effort than traditional reviews, the advantages undertaking this method are greater. It can identify any gaps in current research and summarize the existing evidence in the literature in order to help further investigations. The aim of this work is to identify relevant studies about guidelines of usability evaluation applied to mobile video games of the educational domain focused on primary school children. In addition, to identify educational games aspects that are commonly considered part of usability evaluation criteria. This work was based on the guidelines proposed by Kitchenham and Charters [13] for performing systematic literature reviews in the field of Software Engineering. The steps of this methodology are documented below.

3.1 Research Questions

The research questions formulated to this study are:

  • RQ1: What guidelines are used to measure the usability of educational video games for smartphones?

  • RQ2: What aspects of educational video games are considered in the usability evaluation?

In order to elaborate the search string, we defined general concepts using PICOC method. The “comparison” criterion was not considered, because the focus of this research was not comparing “interventions”. The definition of each criterion is detailed in Table 1.

Table 1. Definition of the general concepts using PICOC.
Full size table

4 Search Process

Based on our research questions, we determined a set of terms and grouped them according to each criterion of PICOC. In order to get current studies in the review and relevant to the state of the art of usability guidelines for mobile educational games, we only considered studies whose publication year was after 2014. The search string defined was the following one:

(“Educational game app” OR “educational video game” OR “mobile educational game” OR “educational game” OR “educational touchscreen application” OR “educational smartphone application” OR “educational smartphone game” OR “mobile games for learning” OR “game app for learning” OR “teaching with mobile games” OR “teaching with game apps”) AND (“children” OR “primary school student” OR “primary school”) AND (“Usability” OR “Interface” OR “User Interface” OR “UX” OR “User Experience”) AND (“Methodology” OR “Method” OR “Framework” OR “Guidelines” OR “Principles” OR “design” OR “evaluation” OR “user interface” OR “study”) AND (publication year > 2013)

The search process was performed by using three recognized databases in order to obtain the relevant studies: SCOPUS, Springer y IEEExplorer. The search string was adapted according to the instructions of each search engine. No additional study was considered.

4.1 Inclusion and Exclusion Criteria

Every article that was obtained from the search, was analyzed by its title, abstract and keywords, in order to determine its inclusion in the review, if the proposal presented was focused on educational video games developed in smartphones and had been applied for children of primary school. Additionally, we analyzed whether their content was about guidelines of usability evaluation or usability in general to decide if it had to be included as relevant studies in the context of the systematic review.

Articles that match any of the following items were excluded from this review: (1) the study is not apply for mobile video games, but for computers or in-person games, (2) the study is not about usability, and (3) the study is not written in English.

4.2 Data Collection

After the application of the procedure in the databases, 910 results were found, in which 36 studies were selected for the review process. The obtained studies were filtered based on our inclusion and exclusion criteria. Table 2 shows the summary of the amount of studies that were found in the search process, and Table 3 shows the list of selected studies.

Table 2. Summary of search results.
Full size table
Table 3. List of selected studies.
Full size table

5 Data Analysis and Results

In order to determine the relevant studies for the present work, we have identified those which have their main topic as educational video games and divide into subtopics that are relevant for our analysis. The selected results are presented below in Table 4.

Table 4. Frequency of topics related to our research.
Full size table

5.1 Usability Evaluation Methods

Based on the articles obtained in the systematic review, it was observed that the most common methods for usability evaluation of mobile applications were through heuristics, metrics and questionnaires. Most of the usability evaluations were based on a specific application, with an emphasis on the user interface aspect.

A study reveals that they have found evaluation models specifically for the aspect of playability, such as the SEEM model and entertaining factors evaluation metrics proposed by Read, Macfarlane and Casey [14].

USERBILITY model, a model that evaluates the user experience (UX) and usability of mobile applications in general, using generic heuristics based on the Nielsen model. Although this model has been designed for mobile applications, it does not take into account the distinctive characteristics of the mobile environment. For the evaluation of user experience, Userbility uses the 3E model (Expressions, Emotions and Experiences), this model is also a generic model for evaluating the user experience, and is not designed especially for mobile applications [15].

HECE, a model to evaluate the dimensions of playability for children, that uses the Nielsen model as a basis and adds aspects of usability for children, where it also assesses aspects such as learning ability for children and if it is appropriate for them [16]. The authors use this model to apply it in the development of usability evaluation for m-GBL in primary schools.

A study has established guidelines for Graphical User Interface (GUI) for the design and development of mobile application prototypes focused on children with hearing impairment. Additionally, they applied an usability evaluation using the inspection method, taking into account three types of user profiles: specialists in children with hearing disabilities, designers and developers. They take into consideration aspects of identity criteria, design, accessibility and the development of the GUI design guide for mobile applications aimed at children with hearing disabilities [17].

The MEEGA model is intended to be used in case studies that begin with the treatment of educational videogames, and after playing the game, the MEEGA questionnaire is answered by the apprentices in order to collect the respective data [18].

The Usability Testing, consists in Nielsen’s model of evaluating the aspects of ease of learning, efficiency, memorability, errors and satisfaction [19].

5.2 Aspects Considered in the Usability Evaluation of Mobile Educational Video Games

We found that in addition to the five main aspects in the traditional usability evaluation [6], aspects such as visibility, game logic, playability, simplicity and learning capacity are also taken into consideration. Table 5 shows the number of studies found in this research of each aspect that are relevant to our domain.

Table 5. Aspects considered in the usability evaluation of mobile educational video games.
Full size table

Below are the studies that substantiate the importance of the aspects selected as relevant:

Since it is about gaming applications, playability would be the fundamental characteristic for the usability evaluation. This plays a very important role in children’s learning, since their natural way of learning is through experience [14].

Mozelius indicates a set of key factors for the design of mobile educational games, which are: simplicity, mobility, usability, playability, gradual increase of game levels, practical and conceptual understanding, collaboration, competition, among others [20].

From Padilla’s point of view, it must take into consideration the aspect of playability for an educational video game to be really effective, which makes it attractive to users, and the learning capacity, which allows users to obtain an educational benefit from the game [3].

According to Maqsood, Mekhail and Chiasson, they take into consideration aspects such as the length of the game content, the relevance of the themes, the visual design and the learning capacity, important for an educational game. They mention that the group of children who participated in the study case, preferred designs with characters that look older, because they felt that they can teach them about situations that may be found in the future. In addition, the colors used in the game also play an important role, since they can influence the perceptions that the players have [21].

According to the results of Cruz‘s study, who applied a questionnaire to evaluate usability in his mobile video game, which involves practicing reasoning of arithmetic operations. It is observed that aspects such as visibility of the system state and the consistency of elements such as buttons are not so relevant for the players. However, aspects such as the competitiveness of multiplayer and sound effects are important factors, as it motivates the player to continue playing and therefore generate satisfaction in the players. Another point that highlights the game is that not only has the logical reasoning part, but also has help with strategies, so players can have easier situations to win the game, in which this is related to the appearance of simplicity of the game [22].

In a study of Drosos, applied to a serious 3D game, the students who tried the game mentioned that it was very flat and that it did not have characteristics that increased the pleasure of playing it. So, it can be said, that the playability plays an important role in educational games. Although, the majority has mentioned that they like the 3D design of the game, and that it was a pleasant educational experience learning new concepts about El Greco, which was the central theme of the game [23].

6 Conclusions and Future Works

Based on the information obtained from the systematic literature review, we present the state of the art of usability evaluation methods for mobile educational video games. In addition, we identify the impact that these mobile applications have on primary school children lives nowadays. Studies related to m-learning have been found, either in a general way or applied to a specific educational level. However, there are not many studies that link the importance of m-learning with the game usability. The existing usability evaluation methods seems not to concern all the aspects about a mobile educational game. Some studies about the development of new mobile educational game evaluate it usability adapting general usability heuristics. The results indicate a need for new sets of guidelines of usability evaluation for mobile educational games, especially focused on primary school children, since they are mainly the target audience of this kind of applications.