1 Introduction

Currently, there is growing interest in mobile applications and geo augmented reality applications for various purposes, such as marketing, tourism, or serious games [1]. In order to create effective applications, a key aspect is taken into account in the design users design their own experience with the application. Therefore, in this paper we will focus on studying the user experience (UX) [3] and the player experience (PX) [3] to contribute to the design of serious games geolocated augmented reality (RA) [4].

The UX is a developing concept, interdisciplinary, requiring research and study, applied to different systems, devices, contexts and people [2]. Also, the player experience (PX) is influenced by various external and internal factors to the subject [3]. For example, as external factors we find those related to (social, cultural, time, space) context or video game system (gameplay, mechanical, motor, narrative, interface devices, immersion sensors, etc.) [5, 6]. As the player internal factors we can find its own characteristics (e.g., chronological age and gender), psychological and/or physical, their preferences, their playing styles, their mental and cognitive models, among others [6]. If we want to model and evaluate this experience we find variables and metrics of different types: qualitative and quantitative or concrete and abstract [7]. All this makes the selection tools, methods and suitable for the present case assessment techniques: serious games geolocated of RA.

This paper presents a methodology for evaluating serious games geolocated with RA. This methodology is organized at different times and includes different instruments and mixed evaluation methods. A guide is presented for test configuration. This guide includes the selection of the user group to evaluate, configuration-test/s for geolocation areas, the test script and the different instruments to be used before, during and after the test. The instruments that have been selected and used are types of players’ test [8], playability test [6], questionnaires gaming experience [9], focus groups and analysis of logs. This methodology has been validated using a serious game created by the research group, called Progrezz [10–13], which is described in the next section.

2 Progrezz: Serious Geolocated Game with AR

The Progrezz platform is born as a project of free and open-source software (MIT license) that seeks to create a video game designed for smartphones and designed as a web application and it stills in a development state. It aims to intertwine reality with their own history and game mechanics, so there’s an energy called “entropy” symbolizes the social imbalance, which has been generated by all those negative actions that manifest in today’s society. Starting from this reality, the player is invited to embody a member of a clandestine network of volunteers, seeking to stop civilization’s collapse and restore harmony to the world. Using geolocation, the user is provided a stage to explore, where the possibility of finding messages and other items located in the real map is one of the basic mechanics, whose collection is linked to the physical movement of the player to place concrete (Fig. 1).

Fig. 1.
figure 1

Progrezz map

Full size image

The utility of the collected objects can advance the story (discovering new mechanical or increasing the level and privileges player), promote events, contain texts advanced players, etc. In addition, you must complete a series of mini games at every step to achieve different objectives. It also seeks that all players can contact other volunteers and identify critical points of the environment through geolocated messages and other resources. Likewise, it is possible to detect and enhance the places where positive actions take place (even contribute economically). That is, the key feature is that users will have the ability to help society while they play and advance in Progrezz. According to the above, it is intended to function as a supportive social network and platform for the dissemination of actions and social movements and voluntary, allowing somehow, visualizes positive actions that happen in our environment both locally and globally. These actions will be reinforced with gamification component surrounding each action carried Progrezz players. Some gameplay elements of Progrezz include territoriality and the feeling of belonging to a global group, adding a sense of solidarity to the game, because the impact is quantified in reality made by the player.

In the absence of defined standards or guidelines for evaluating UX geolocated mobile games, this work has been evaluated, selected and adapted different techniques and instruments that allow the evaluation of this platform. Following evaluation designed specifically for this type of mobile games described.

3 Evaluation Proposal

Table 1 shows the phases and activities in which evaluation has organized a mobile game geolocated with RA, as well as the people involved in them are presented.

Table 1. Organization Assessment
Full size table

We will describe each of the activities carried out to evaluate the geolocated game with RA.

3.1 Pre-test

3.1.1 Set the Group, Objectives and Test Area

The ideal number of participants for the test was determined to be between 10 and 12 people, in order to ensure the proper development of it in a 3-hour session. In addition, the group must be balanced in the number of men and women to avoid bias in gender.

As for the ages, they will be chosen depending on the target audience of the game. For example, in the case of Progrezz, it will be between 18 and 45 years.

The user profile should be related to your level of experience with mobile technologies and geo games. In the case of expert users in geolocated games, they will be asked to act as expert evaluators to inspect the playability of the game.

Also, the goals to be achieved in the test must be correctly located on a map, so that are achieved in a walking tour of the city chosen for the test, the duration of the testing session. In the case of Progrezz, we proceeded to design distribute messages in different streets of the city of La Laguna in Tenerife.

3.1.2 Prepare the Test Script

The test script includes instructions should be given to participants, as well as everything that should be considered for conducting the test: necessary tools, links, resources, etc.

Before starting the test, players must complete the questionnaire on player profile (following Marczewski proposal [8]) so on-line, save the result obtained from it and send it via email to an address given by the evaluators. Here are accessed from the mobile device to the serious game.

In the case of Progrezz, access is performed by using the same browser and login request from a personal account (Facebook, Twitter, Google, Github or Steam). It shall authorize the application to use the GPS location and use your phone camera, in addition to activating the gyroscope (screen rotation). Once the user is in Progrezz account, you can choose from:

  • Messages, where you can see unread messages, unread messages, which are available to unlock and fragments of incomplete messages.

  • To unlock a message, you must complete a mini game.

  • Profile, where the user level and data on the range of vision that is available, according to their level, the distance that must be the fragment to collect to collect it and the accuracy of detecting them will be displayed. In the first few levels players cannot even, post messages on the platform.

  • Map where the location of the player and the possible locations of messages shown around. The blue circle indicates the vision range, the red squares areas where you can find messages. The red squares triangles linking together fragments of the same message.

  • Scanner, where you can access the viewer augmented reality game. In it they appear blue-green diamonds when we stand before a message fragment to collect.

  • The scanner has three modes: camera viewfinder, where you show what is seen directly from the phone’s camera viewer augmented reality, where the world increased to the original camera image overlays, and viewer for reality glasses increased, designed for use with glasses Oculus Rift like.

Once inside the game, it will be inserted into the narrative of it. Thus, in Progrezz the player will propose an adventure that is part of a secret society that tries to save the world doing positive social actions. Specifically, for testing has been introduced in the system a total of 17 messages, some of which are divided into fragments as a tutorial. The player’s goal is to collect the different messages. To do this, you must move around the map to the red areas and, once there, use the augmented reality viewer to find messages. When the player has the message, you must “capture” for later, you can unlock your reading. Once you have all the fragments of each message, you can access a panel where you collected so far will be displayed. If you want to read one, simply select it, after overcoming a small mini game, its content is displayed onscreen. They have to collect 17 messages to complete the mission.

The test has been designed to be carried out in one day, although the duration will depend largely upon the involvement of the player.

During the game session, the player must pause to complete the questionnaire that will measure the experience during the game. The hyperlink to the questionnaire will be sent to the user during the course of the evaluation.

Once the game session, the user must complete the remaining two modules of the questionnaire gaming experience: the basic module, which will measure the player’s experience during the game and post-game module that will measure the feelings of the player once he has stopped playing.

In addition, expert players, who are those who already have experience with similar, such as Ingress, also must complete a questionnaire heuristic that measures factors Playability in different facets, as the intrinsic gameplay, gameplay mechanics, Artistic Playability games the Personal or Social Playability Playability.

Finally, it will be held a Focus Group with test participants to discuss the experience, achieved and frustrated goals, the difficulty of the proposed mission, satisfaction with the goals achieved, possible errors of the platform, the main problems, etc.

3.2 Test: Game Experience Questionnaire (GEQ)

The Gaming Experience Questionnaire (GEQ) [9] has a modular structure comprising: a basic questionnaire, social presence module and a module for post-game.

The three modules are intended to be evaluated immediately after you finish a game session, in the order listed above. The first and second parts are polls feelings and thoughts Player while playing the game; the third part, the post-game module evaluates how players after finished playing feel.

In our case we have developed an in-game version of GEQ, in order to assess the gaming experience in multiple intervals during a game session.

Thus, the following modules are proposed:

  • GEQ Basic module: We could say that this is the central part of the GEQ. We will serve to evaluate the gaming experience through seven components: Dipping, flow, competition, positive affect, negative affect, tension and challenge.

  • GEQ module In-Game: In-Game GEQ module is the minimum version of the basic questionnaire is used to assess the gaming experience as it takes the game session, which will facilitate the validation of continuous indicators in real time.

  • GEQ module Post-Game: The post-game module evaluates how players after having stopped playing felt.

3.3 Post-Test

3.3.1 Focus Group

After users will run the test and, after making the corresponding questionnaires, it is conducted Focus Group session where the highlights of the test are discussed. After the session, a transcript of it is done and proceeds to its qualitative analysis. The phases of this analysis are:

  1. 1.

    Read the transcription and identification of the issues that are most relevant in accordance with the purpose of observation.

  2. 2.

    The process of categorization and coding.

  3. 3.

    Representation of the information collected for each category in a matrix.

  4. 4.

    Obtaining results and drawing conclusions.

3.3.2 Playability

The evaluation of the gameplay should be done by expert users, since a validated questionnaire about the player’s experience with different heuristics that allow inspection of different criteria organized into dimensions [2], which measures the quality of use is use a videogame. These quality measures are:

  • Satisfaction: Liking or complacency player before the full game or in some specific aspects of it. It can be measured by the percentage of game discovered or unlocked by the number of challenges (primary and secondary) resolved.

  • Learning: Easy to understand and master the system, the mechanics of the game and how to interact with it. It can be measured by the number of attempts to challenge and invested according to the intended time.

  • Effectiveness: Time and how to use resources to provide fun to the player while it achieves the objectives. It can be measured by the number of shares not carried out according to the time between objectives, goals and challenges.

  • Immersion: Ability to believe what is at stake and integrated into the virtual world shown in the game. It can be measured by the time and attempts to challenge and by SAGAT technique [14].

  • Motivation: Feature game that moves the person to perform certain actions and persist in them for completion. The percentage of unlocked game and the number of items and secondary objectives achieved can measure it.

  • Emotion: Impulse originated involuntary response to stimuli that induce feelings and video game triggers automatic reaction behavior. It can be measured by the Test of emotional grid [15] and by biometric observation and thinking aloud.

  • Socialization: Measurement of the elements that promote social factor or group experience; thanks to the reactions engage with other players or with other game characters. The number and type of messages between group members can measure it and the number of shared and used resources.

3.3.3 Logs Analysis

Records of the test participants are analyzed for different regarding the effectiveness and efficiency in achieving the goals of the game metrics, such as:

  • Effectiveness on the goal: Number of goals that have been achieved correctly.

  • Completed Challenges: Completed challenges number.

  • Frequency attempts in achieving goals: Number of attempts to achieve the goal.

  • Goal time: Time required by the user to achieve a goal.

  • Optimal route: Variability of the route used by the user to achieve the goal against the optimal path.

4 Conclusions

This article has presented a guide for evaluating the user experience (UX) and the player experience (PX) in a geolocated video game that uses augmented reality organized so that to analyze the gameplay as a measure of quality use. This guide was then proposed to evaluate different techniques and methods UX and PX for video games. In addition, it has been applied to a specific case study: Progrezz.

The guide consists of three phases: a phase of pre-test, a test and a posttest. The pre-test phase, comprising various activities, such as setting up the group of participants who will perform, objectives or goals to be achieved and the area of ​​the test. In addition, it should develop the script for the test and identify the player profile, which in our case is selected profiles described by Marczewski. Then, in the test phase, it is important to evaluate the experience in the game. For this we have chosen to select the GEQ questionnaire and an adaptation of it for a test at different intervals during the course of the test. Finally, as the close of the assessment test, conducting a focus group to analyze the main difficulties encountered in performing the test as well as suggestions and proposals for improvement it is proposed. Moreover, expert users should perform a test inspection of the gameplay of the game. Finally, evaluators review the logs obtained from test development in order to obtain metrics effectiveness and efficiency of the game. Following this proposal as a guide for evaluating geolocated mobile games with RA is possible to obtain qualitative and quantitative metrics that allow multidimensional assessment UX/PX as quality video game use. Finally, say that this guide is an initial assessment proposal for a case study where there are standards, but that it should be revised and refined metrics for tighter UX/PX geolocated mobile games with RA.