Keywords

1 Introduction

The usability of a software product, which refers to the ease that a system can be learned and used, greatly influences the user experience. This aspect is a critical condition for the system, because if it is not understandable and difficult to use, users will stop using it [1].

There are different methods, techniques and tools related to the treatment of usability issues in software products. A usability evaluation method is a procedure composed of a set of well-defined activities for the collection of data related to the interaction between a user and the software product; and how these specific properties of the product contribute to achieve a certain degree of usability [2]. There are several methods for evaluating usability, which vary according to the time/benefit cost, rigor, number of users, number of evaluators and the knowledge they possess. Holzinger proposes a classification for these methods using two categories: inspection methods and test methods [3].

Among the methods of inspection is the heuristic evaluation, which, according to Nielsen [4], is “a discount engineering method that involves having usability specialists to judge whether each item evaluated follows established usability principles”. Heuristics are general design rules and not specific guidelines [5].

Nielsen proposed the ten usability heuristics for the design of user interfaces [6]. Other authors designed specific heuristics for some types of software [7, 8]. In a previous study [9], from the literature some usability and playability heuristics were gathered for video games on mobile devices and others for augmented reality applications.

The aim of this work is to formalize the development process of a proposal of usability and playability heuristics for augmented reality video games for smartphones. This article is structured as follow: In Sect. 2, we present the methodology used in this study, then in Sect. 3, we show a survey made for end users and usability experts with the purpose of validate the preliminary set of heuristics from a previous study, after that in Sect. 4, we analyze a case study of the videogame Park AR where the set of heuristics was used as an evaluation tool, and finally in Sect. 5, we present the heuristic set formally.

2 Methodology

To formalize the development of the heuristic proposal, a methodology was used. This methodology was created by Daniela Quiñones [10] to establish usability heuristics and control lists associated with usability. It has six defined and ordered stages: Exploratory Stage, Descriptive Stage, Correlational Stage, Explanatory Stage, Validation Stage and Refining Stage.

2.1 Systematic Review

In a previous work [9], a study of systematic literature mapping of the use of augmented reality in educational video games was carried out. In this systematic review, after applying the inclusion and exclusion criteria mentioned in the article, six studies were selected, which are shown in Table 1.

Table 1. Selected studies [9].
Full size table

Finally, from the six selected studies, 45 heuristics were taken as a result of a comparative analysis. In this analysis, the authors filtered the preliminary set by taking some criteria such as discarding those heuristics related to multiplayer video games or some very general guidelines about game design, and comparing those ones with similar approach. The set of heuristics as result of the previous study can be seen in Table 2.

Table 2. List of heuristics from a previous study [9].
Full size table

3 Surveys for End Users and Usability Experts

A survey is a method in which information about a sample of individuals is collected [17]. For this research, two surveys were carried out whose objective was to validate and evaluate the set of initial heuristics of Table 2. One survey was focused on usability experts and the other, on end users (with a player profile). Before starting the surveys, a usability heuristic was added to the initial set to make a total of 46 heuristics.

Initially, a basic design of the surveys was carried out, that is, some questions were found in both surveys. The surveys had two parts: the first was used in order to highlight the profile of the participant and the second part for the assessment of the set of heuristics. For the assessment of the heuristics, the LIKERT measurement scale was applied using three levels (Not very important, Important, Very important).

The survey for players had 58 participants who answered it voluntarily. The participants were contacted in various ways: mail, forums, social networks, etc. Some characteristics of the participants were the following: the majority were male, they were between 19 and 42 years old, most were university students (in process), some participants knew about Augmented Reality video games like Pokemon Go and Ingress and almost all the participants use to play video games a maximum of 2 h a day. With respect to the assessment of the heuristics, the average of the rating awarded by each player was extracted for each heuristic and 3 of the heuristics obtained an average score lower than 2 (Important). The heuristics with the score lower than the average in this survey can be seen in Table 3.

Table 3. Heuristics with the score lower than the average in the end user surveys.
Full size table

The survey for experts had 5 participants. The participants were contacted by mail. Some characteristics of the participants were the following: the majority were female, they were between 34 and 40 years old, most with the Master’s degree, all had experience performing heuristic evaluations, everyone knew the augmented reality video game Pokemon Go, and most use to play video games a maximum of 2 h a day. With respect to the assessment of the heuristics, the average of the rating awarded by each usability expert was extracted for each heuristic and 8 of the heuristics obtained an average score lower than 2 (Important). The heuristics with the score lower than the average in this survey can be seen in Table 4.

Table 4. Heuristics with the score lower than the average in the usability expert surveys.
Full size table

The results of the surveys were compared and it could be observed that 2 of the heuristics of the set were rated with a score lower than average. The two heuristics that were eliminated from the set are shown in Table 5.

Table 5. Heuristics eliminated after the surveys.
Full size table

Before the survey had 46 heuristics, after this, the whole was reduced to 44 due to the coincidental assessment by experts and players regarding 2 heuristics. At the end of the surveys, a new heuristic was proposed related to the way in which users interact with the controls in video games.

4 Case Study - Park AR

After the surveys, a case study was conducted in order to use the set of heuristics obtained as an evaluation tool. The videogame Park AR was selected for this case study because it fits into the profile of the type of software covered in this study.

The case study was conducted by 5 usability experts. The objective of this study was to identify aspects that are not being covered by the set of heuristics and, if the situation arises, to formulate new heuristics to add them.

The analysis was started by verifying that the problems identified were correctly associated with a heuristic. This result can be seen in Table 6.

Table 6. Results from the case study - Park AR.
Full size table

Then, a unique list of problems was developed based on those identified by each evaluator. In total, the list contained 61 usability problems. From these problems and their associated heuristics could be described in a better way some proposed heuristics in the formal specification of these.

5 Formal Specification of the Heuristics for Augmented Reality Videogames on Smartphones

Finally, some definitions of the heuristics were corrected as a result of the usability problems identified in the case study. There were a total of 45 heuristics, which will be formally presented with the help of the template proposed by Quiñones and Rusu [10]. The resulting set was named HARVS due to its acronyms “Heuristics for Augmented Reality Videogames on Smartphones”. Table 7 shows the proposed set of heuristics.

Table 7. Heuristics for Augmented Reality Video games on Smartphones (ID, name).
Full size table

Next, the descriptions of each heuristic will be placed:

  • HARVS01: The music and the game are in sync.

  • HARVS02: The design of the interface provides enough information to correctly understand and generate a correct interaction between the player and the game.

  • HARVS03: The user interface of the game has the necessary options to not depend on the user interface of the device.

  • HARVS04: The indicators referred to the game screen are correctly located and are distinguished in the user interface. In addition, they are easy to understand for the player.

  • HARVS05: The player is capable to understand the context of the video game.

  • HARVS06: Navigation through the application provides logical and appropriate guidance for the understanding of the game.

  • HARVS07: The control keys are not repeated in the interface, perform the expected action and maintain a standard convention. These control keys must be self explanatory.

  • HARVS08: The game controls are easy to recognize and use, they are also in an appropriate position on the interface and do not hinder the interaction with the game.

  • HARVS09: The game reacts efficiently and quickly to the interaction of the player and the controls of the game.

  • HARVS10: The player cannot fall into infinite loops where the game cannot continue.

  • HARVS11: The game should provide guides on what things to do or see in the game so that the player does not have to overload his memory remembering this information.

  • HARVS12: The game has an instruction manual and frequently asked questions (FAQ).

  • HARVS13: You should not go through many user interfaces to start the game or the game sessions.

  • HARVS14: The game is portable.

  • HARVS15: The game provides clear objectives or supports the objectives created by the player providing advice or instructions to achieve them.

  • HARVS16: The player has access to the results obtained in each game.

  • HARVS17: Players get rewards that motivate, loyalty and can be useful in the game.

  • HARVS18: The player always makes decisions based on the main objective of the game.

  • HARVS19: The elements and mechanics of the game are in balance so that the game can be finalized by a player.

  • HARVS20: The first experience with the video game should give a good impression of everything that the game has: story, characters, enemies, difficulty, etc.

  • HARVS21: The defined story must be portrayed in the objectives of the game and the form of victory.

  • HARVS22: The tasks defined by the game do not become repetitive, so each game is different from the previous one.

  • HARVS23: Players can express themselves as they are.

  • HARVS24: There are different ways to achieve the objectives of the game.

  • HARVS25: It is not related to the performance of the device. That is, there is no way for the player to be “stuck” at some point within the application.

  • HARVS26: Players should not ask themselves if the different words used in the game, situations or actions of different options mean the same thing.

  • HARVS27: The game uses orthogonal differentiation of units, that is, each virtual unit has different functionalities.

  • HARVS28: The player is capable to collect different possessions won during the game without fear of losing them.

  • HARVS29: The player must be capable to save the game at any time and at any stage of the game, due to the limited resources of the mobile phone, the player must be able to save the game each time he wants to save it, and then the player can continue from the saved stage.

  • HARVS30: The player is capable to skip non-playable content such as films or introductory images, the player must have control over whether to watch those contents or not, they should be able to avoid if the player wishes.

  • HARVS31: The game should allow user customization, so users can play the game at the desired difficulty and at the desired speed. In mobile phones it is difficult to play at speed because of the inconvenience of the control keys. The control keys, sometimes, are not convenient to play easily, to react to quick decisions, that is, to respond right away.

  • HARVS32: The game should be able to handle internal interruptions. These interruptions can be generated by calls, text messages, emails, etc. The game must be paused only so that the player can continue playing after the interruption, otherwise the player will feel frustrated and turn off the game.

  • HARVS33: To handle external interruptions, these interruptions are those that occur in the environment of the player for example if someone speaks to the player while playing, or the player is waiting for the train and this arrives so the player needs to stop the game to be able to continue it at the exact moment in which it stopped him.

  • HARVS34: First design the experience generated by the game, then decide the appropriate technology.

  • HARVS35: Allow players to interact with virtual characters and other players.

  • HARVS36: Do not completely convert the real environment into a virtual one, because the components of reality will be overshadowed.

  • HARVS37: Not only convert existing video games to augmented reality. After a few times of playing it, the initial emotion disappears, since the game only tries to be visually more attractive than the originals, but does not include really genuine game mechanics.

  • HARVS38: The 3D content in the game should add something interesting to the game.

  • HARVS39: Evaluate how the system is viewed by the user. Users should receive feedback on what is happening in the system. The augmented reality applications use tracking systems to determine the position of the virtual content in the real scene, which must be fast and reliable, otherwise, the users will be lost when interacting with the application.

  • HARVS40: How accurate is the system during interactions. The position of the virtual content in the interface is determined by the tracking system and should not vary.

  • HARVS41: Augmented reality applications require special devices such as sensors and/or cameras. In addition, bookmarks may be necessary, such as reference markers. The configuration of the environment should be as simple as possible.

  • HARVS42: This measures the degree to which the augmented reality application exceeds the user’s expectations. Interaction is an important aspect in augmented reality applications, and the user must have positive attitudes towards the system.

  • HARVS43: It keeps the user informed about what is happening when interacting with the game controls.

  • HARVS44: If the game runs commercials with advertising, you must choose non-playable moments to add them.

  • HARVS45: The controls must have an option for left-handed people in which the main controls are inverted for the player’s comfort, this mode in spite of having a defined order can be edited by the player.

6 Conclusions and Future Works

The present work tries to cover the different aspects of usability and playability when evaluating augmented reality video games. The authors designed this set of heuristics based on a previous study, which was evaluated and tested through surveys and a case study. However, more tests and studies are needed to verify that the set covers all aspects of applications in this domain.

It was possible to determine, from the surveys carried out, that both experts and users with a profile of players accept the majority of the initial set of heuristics. However, then some heuristics were added and removed, so in a future work we must re-survey people with these profiles using the final set proposed to confirm their acceptance. Besides, we will use this results to build a numerical analysis.

With the case study it was possible to verify that the set can be employed as an evaluation tool for video games of augmented reality in smartphones. Furthermore, we discovered some mistakes in the description of some of the heuristics that caused confusion in the preliminary set.

As part of a future study, an augmented reality video game for smartphones will be designed using the heuristics proposed in this work in order to validate that the set also can be used to design applications of this type. The design of this video game will be employed to add or remove more heuristics to the whole of the present proposal.