Keywords

1 Introduction

Hand-eye coordination is one of the essential steps of the human learning process. It consists of the ability to coordinate and track hand movements with the eyes, which is essential to reading and writing skills. The performance of various activities requires careful eyes for the brain to know the accuracy of space, along with the use of hands for visual capture. Some examples of hand-eye coordination include writing, driving a car, playing sports, locking and unlocking a door, and so on.

However, some individuals (children, adults, and older people) have difficulties with hand-eye coordination, showing problems in performing specific tasks. There are clinical exercises for stimulating cognitive action as a treatment stage to improve people’s hand-eye coordination. Among these, in the Grooved Pegboard Test (PegB), a patient should insert 25 cylindrical pins as quickly as possible into each of the 25 holes of a plate, one at a time. The time to complete the task is twice each hand. The stick capture test (StickC) is another clinical exercise where the reaction time is checked [5].

With the advancement of technology, traditional clinical exercises can be improved so that their use does not become something monotonous for the participant; instead, the user feels engaged in carrying out the tasks, such as serious games [3, 13]. Serious games can be classified as a playful interactive tool capable of addressing different thematic areas and concepts through its resources, becoming a facilitator in the teaching process [14]. Studies on eye-hand coordination are explored through gamification in serious games, where the exercise environment can adapt to players, depending on their age group and the intensity of the problem.

Serious games can help to develop motor coordination, attention, memorization, among other physical and cognitive skills. Their intrinsic characteristics, such as rewards, assets, sounds, goals, and achievements, allow players to engage and stimulate their senses. However, to create a serious game that meets players’ skills is a big challenge, as each user has a different skill level [7]. Offering a challenging game can cause players to be frustrated and disinterested. By contrast, leaving a more accessible level can bore and displease others [6]. One way to solve the difficulty adjustment problem in a game is to employ the Dynamic Difficulty Adjustment (DDA) technique. It consists of making modifications to parameters, scenarios, and game behaviors in real-time to avoid the player’s frustration when facing the game challenges [2].

Studies have shown that employing the DDA approach in games generates a high level of overconfidence in the player [7]. Therefore, using it for visual-haptic attention training, for example, ensures comfortable training with significant improvements for training participants [17]. In [18], the authors analyzed executive function skills training with both adaptive and non-adaptive methods, obtaining that DDA was more effective than the non-adaptive method.

In this study, we propose an adaptive serious game using the DDA technique. The adaptive difficulty of the game is implemented through a Reinforcement Learning algorithm. Our game is based on Stick Catching test [5], which consists of verifying the reaction time of players during the task of collecting the corresponding color balloons to prevent that they fly. Players use a Leap Motion controller for interaction with the game. We evaluate the game’s adaptivity by comparing velocity by episodes and accumulated rewards for each player. We also conducted experimental tests involving ten participants for usability evaluation.

The rest of this paper is organized as follows. In Sect. 2, we present the related work. In Sect. 3, we detail the proposed solution, which includes a description of the DDA technique, the steps followed for implementing adaptive difficulty, and the Leap Motion Controller. The evaluation and results are described in Sect. 4. Finally, in Sect. 5, we present the conclusions of this study.

2 Related Work

We found in the literature several studies using serious games, such as a tool to assist in therapeutic processes that involve coordination as well as works that include DDA techniques.

Stavros et al. [20] present a study on how to design serious games for individuals with intellectual disabilities (ID) from existing design frameworks. The authors structure a survey into two categories. In the first category, they presented the structure of serious games, suggesting the essential elements and principles for their construction. The second category presents design guidelines for serious games intended for people with ID. The authors conducted a survey identifying the elements needed to create an SG for people with ID. The authors conclude that no design framework includes all design guidelines from the literature for implementing an SG for DI. Still, it is possible to make some changes in the construction of such games. Although our work is not focused on designing SG for people with DI, we have adopted two features presented in that study: Graphical Interface and Game Difficulty, which use machine learning techniques to create a Game with a self-adaptive difficulty adjustment.

The study by Hendrix et al. [10] shows how games can dynamically adapt according to the player’s ability. The authors propose a six-step method for implementing a game with adaptive difficulty. They evaluated that method with users through experimentation in two games. The results show that the pragmatic six-step plan presents a relatively good and straightforward set of steps that a game developer can follow regardless of the genre and style of the game as well as the tool used. In our work, we use the pragmatic six-step plan for implementing adaptive difficulty showing from the above authors.

Hidalgo et al. [11] propose a serious game to support therapy for children with motor difficulties. The game consists of associating different figures-colors of their environment, and with each new level of play, the difficulty is increased so that the child works yet more his hand-eye concentration. The game uses a Leap motion controller as an input device. The accompanying therapist adjusts the difficulty level of the game. The authors evaluated their work with 20 participants. They evidenced that the game was surprising among some children. They were able to adapt quickly within the space and depth of the virtual game.

The study by Lin et al. [13] present a serious game to encourage physical and cognitive functions in the elderly. The serious game consists of a rhythm game where each phase works some cognitive ability such as response speed, short term memory, or working memory. User interaction with the game is via a motion sensor controller. This device is like a set with some buttons scattered around it, and when the user wears them, these buttons line upon his body. To interact, one need to press these buttons that also estimate acupuncture points. Therefore, the player must observe in the game interface, the position and the colors of the notes and then press the buttons of the corresponding motion sensor controller. During gameplay, player data collected for analysis of cognitive skills. To validate the game, the authors conducted experiments with older individuals (age between 63–93 years) that presented a viable game with improvement points.

Regarding studies that use DDA techniques, Zohaib [21] reports that in the last decade (2009–2018), several research studies have been carried out on DDA, both to develop new methods, improve methods, and to apply them in several areas. There are several approaches to working with DDA, being the reinforcement learning one of them. For digital games the creation of an adaptive game allows a more exceptional experience for each style of player. As there are players with different profiles, static challenges sometimes do not take into account the player’s reality. Some work shows that using reinforcement learning with the DDA technique, players feel comfortable in a balanced environment, as parameter adjustments are made in real-time in response to their skills. In addition, studies are presented that create adaptive algorithms using concepts of evolutionary computing and reinforcement learning. We use the DDA technique in the implementation of the game presented in the present work. Details can be seen in the section below.

3 Proposed Solution

Our method consists of the combination of three different elements: the Dynamic Difficulty Adjustment method to create a game where the difficulty levels are adjusted according to the player’s skills; pragmatic six-step plan to implement adaptive difficulty and the Leap Motion controller as interaction device with the game interface.

3.1 Dynamic Difficulty Adjustment (DDA)

Dynamic Difficulty Adjustment (DDA) allows adjusting the entire scenario of the game, including design, gameplay, features, components, behaviors, and the different elements that compose the game. This adjustment is made at runtime according to the performance of each player. The modeling of the scenario must take place in real-time according to the player’s abilities so that it presents a challenging difficulty to the point of not frustrating him, and which do not be so easy as to disappoint him. We use the DDA technique in the game, aiming to keep the players engaged in a task with an appropriate level of difficulty according to their performance. We follow the recommendations by Andrade and colleagues [1] to apply the DDA technique in our game. The authors [1] suggest that DDA must comply with three basic needs, as follows:

  • The game needs to automatically track the player’s ability and rapidly adapt to it.

  • The game must follow the player’s improving or falling level and maintain a balance according to the player’s skill.

  • The players must not perceive the process of adaptation, and successive game states need to have coherence with the earlier ones.

Csikszentmihaly [15] proposes the flow channel for creating the DDA (Fig. 3). This model indicates how the difficulty level of a task directly implicates the player’s perception. The flow channel prevents frustration under challenging tasks and boredom under simple ones (Fig. 1).

Fig. 1.
figure 1

Source: Image from [21]

Flow channel model.

We adopted the above flow channel to create an adaptive game that is not difficult to the point of being frustrating and not easy to the point of being boring.

3.2 Pragmatic Six Step Plan for Implementing Adaptive Difficulty

The primary key for a game to be adaptable is through user performance. We use the method proposed by Hendrix et al. [10] for creating an adaptive game. It consists of the following six steps:

  1. 1.

    Identify variables in the game that are good indicators of the player’s performance.

  2. 2.

    Determine variables that influence the difficulty of the game, and that can be changed.

  3. 3.

    If an implementation of the game exists, locate the performance and difficulty indicating variables.

  4. 4.

    Consider whether the game features multiple mechanics and if so, to which mechanic do performance and difficulty variables relate.

  5. 5.

    Decide how the performance variables will be used to calculate difficulty.

  6. 6.

    Decide upon sensible starting values for the identified variables, impacting the difficulty balancing.

3.3 Leap Motion Controller

In this work we use the Leap Motion ControllerFootnote 1as the user interaction device. Leap Motion is a device that connects to a computer and allows one to track gestures with hands and thus interact with virtual objects intuitively and naturally. It consists of two cameras and three infrared LEDs, forming an area of interaction, as shows Fig. 2. It has been used in several fields, such as virtual simulations, learning environments, rehabilitation, entertainment, among others. This device allows creating a more natural and intuitive interaction, contributing, thus, to motor coordination required in the eye-hand coordination training.

Fig. 2.
figure 2

Source: Image from [9]

Leap Motion interaction area.

To fit our game, we used the leap motion interaction area divided into two zones, one for the player to navigate using the hands through the environment and thus position the hands on the objective and one other that detects the user’s interaction with the game objects.

Fig. 3.
figure 3

Source: Image from [16]

Interaction space is split into two zones.

3.4 Implementation

In this section, we present the proposed game implementation process, including the game design principles, the adaptive difficulty process, and the reinforcement learning algorithm implementation for the self-adaptive game difficulty functionality.

Proposed Game. Using the Unity 3D engineFootnote 2, we construct our proposed game in two dimensions (2D). The goal of the game is popping balloons the same color that appears in the indicative panel, as shows Fig. 4.

Fig. 4.
figure 4

Game Interface with points and hands for the interaction of balloon (Color figure online)

The colors used are the basics of the RGB (Red, Green, Blue) additive color system. The difficulty level of the game is given by speed the balloons to rise, forcing the player to memorize the color and pop the balloons quickly. The users use a leap motion controller as the input device.

Implementation of the DDA Technique. We mapped our game by following the six implementation steps suggested by Hendrix et al. [10] to make it adaptive game difficulty.

Initially, We identified the following: number of the score, number of losses, and total of balloons as variables in the game that are good indicators of player performance.

As second step, we identify variables that influence the difficulty of the game, and that can be changed: speed of balloons and Quantities of balloons.

In the third step, we identify the variables score, loss, performancePlayer and balloon.velocity in the script denoted AgentScript.

In the fourth step, we define the game design by including a single mechanic.

In the fifth step, we used the reinforcement learning (RL) technique for that rewards are obtained according to the performance variable [19] explains, RL is a technique that allows learning what should be done, mapping situations to actions, aiming to maximize rewards obtained by actions. The agent does not know which steps to take, so he must find out the steps that bring the most rewards.

Finally, in the sixth step, we chose to empirically the starting variables. The default values can be adjusted if required.

Reinforcement Learning Approach to Include Game Self-adaptive Functionalities. We implemented the adaptive difficulty of the game through Q-Learning, a Reinforcement Learning algorithm. Q-Learning is an off-policy temporal difference algorithm that focuses on state-action values. The action value in each state is obtained using a table that is updated in each interaction with the environment, denoted Q-values, as shown in the Eq. (1).

$$\begin{aligned} Q(s,a) = Q(s,a) + \alpha [r + \gamma .maxQ(s',a') - Q(s,a) ], \end{aligned}$$
(1)

where s is the current state and a is an action taken in this state. When each action a is taken, a new s’ state is selected, and a reward issued for that pair of (s, a). For the new selected state, a new action a’ is taken, chosen randomly using a predefined probability (a method called Epsilon-Greedy Policy). \(\alpha \) is the learning rate, r is the reward for an action taken in a given state and \(\gamma \) and the factor of discount. We are working with the velocity v of the balloon as a parameter for game difficulty. Therefore, the states are the respective velocities that the balloon can assume:

$$\begin{aligned} S_{velocity} = \{v_{1},v_{2},v_{3},v_{4},v_{5},v_{6},v_{7},v_{8},v_{9},v_{10}\} \end{aligned}$$
(2)

The three possible actions that change the game’s difficulty are increasing, decreasing, and keep up speed.

$$\begin{aligned} A = \{v_{+1},v_{-1},v_=\} \end{aligned}$$
(3)

We chose to implement our RL algorithm, using the ML-Agents toolkit as a facilitator for game design and communication with the algorithm through the Python API (Fig. 5).

Fig. 5.
figure 5

Block diagram of ML-Agents toolkit adapted for our work.

An agent observes the environment so that he can make a better decision. Our agent is the entire game environment, including heads-up display (HUD), balloons, and color panel.

The brain is linked to one or several agents. He receives remarks and rewards from the agent and returns an action. Our brain is connected to the game environment. It has a vector of observation with space size two, allowing the observation of the current balloon speed as well as the player’s performance. It also has a vector with the branches size of two, so that the agent receives two possible actions, balloon velocity increase and decrease.

For communication between the Python API part and Unity, it is necessary to use the academy, which will orchestrate all of the Agent and Brain objects in a Unity scene, including observations, decisions, rendering quality, and other features. This communication is accomplished through external communicator (feature called broadcasting).

4 Evaluation and Results

We evaluated the adaptive of the game through experiments involving ten participants (ages between 16 and 23 years). These users are all-male, whose selection was carried out at random. Each participant performs a ten-minute experiment to assess the perception of the use of the proposed game and whether the environment is adjusted according to their capacity. For each participant, the game was adapted according to their ability, presenting a difficulty that was challenging and achievable. The tests sessions were conducted using a notebook of 1 TB HD, 8 GB of RAM, processor of 2.70 GHz intel core i5 and Intel HD Graphics 620. Each test session consisted of one participant interacting with the game through Leap Motion (Fig. 6). The test for each participant lasted between 10 min. Before the tests, we introduce participants to the use of the device and provide them the game instructions.

Fig. 6.
figure 6

Example of participants playing the proposed game.

During each test, information was collected from each player’s virtual environment, such as rewards earned and balloon speed. We use the SUS (System Usability Scale) questionnaire [4] model to check the usability of the game. It consists of the following 10 item questionnaire that each participant answered after the test:

  1. 1.

    Frequency of using the game.

  2. 2.

    Complexity in interacting with the environment.

  3. 3.

    Facility of use.

  4. 4.

    Constant support from someone to instruct in the use.

  5. 5.

    Well-integrated interaction device;

  6. 6.

    Inconsistency in the game.

  7. 7.

    Quick learning ability.

  8. 8.

    Complicated to use the interaction device.

  9. 9.

    Confidence when playing.

  10. 10.

    Need to learn several things before playing.

The SUS questionnaire uses the Likert scale [12] to verify the level of agreement in each statement. The scale used in this model is five levels: 5 (strongly agree), 4 (agree), 3 (neutral), 2 (disagree), and 1 (strongly disagree). Obtain the global SUS value, which ranges from 0 to 100, is necessary to calculate the questionnaire score. Thus, odd questions (1, 3, 5, 7, and 9) subtract one less from the value marked on the scale. In the even questions (2, 4, 6, 8, and 10), subtract by five. After calculating the value of each question, it is necessary to sum all the values and multiply by 2.5 to obtain the overall result. As stated by Cunha [8], scores below 60 represent dissatisfaction to the user, while above 80 have a high level of satisfaction.

The results are described according to both the adaptivity of the game and the user experience playing the game.

According to the adaptivity of the game, Fig. 7 (top) shows the velocity by episode for the players 1 and 6. Figure 7 (bottom) shows the cumulative reward for each player indicating the performance of reinforcement learning algorithm. We selected two players (1 and 6) to show the learning environment at two different times. For player 1, it maintains a lower speed, while player 6 at a faster velocity. We can see that the game learn to adjust the velocity that allows the player to accumulate increasing positive rewards, according to its performance in that particular difficulty.

Fig. 7.
figure 7

Result of the experiment with speed being adjusted as the player advanced in the game (top); Result of the performance of the algorithm per player throughout the experiment (bottom).

Fig. 8.
figure 8

SUS score per players.

Considering an average of SUS of 68 scores, our system presented an average of 74 scores, which results in a usable and satisfactory environment for the user, as shown in Fig. 8.

5 Conclusions

We presented a serious game with a self-adaptive difficulty level to the player’s profile. We implemented the self-adaptive process through a reinforcement learning approach maximizing the users’ success according to their skills. This self-adaptive process occurs in real-time when the player interacts with the game. Here, a central intelligent process modifies the player environment while observing the user actions and performance results. Experimental results shows the effectiveness of our solution for the self-adaptive process, increasing user satisfaction and game engagement. This feature brings direct benefits to users contributing to the eye-hand coordination training skills. As the next steps, we intend to explore interactive RL approaches, where an external adviser speed up and enhances the game self-adaptive learning process. We also intend to explore new deep RL techniques to develop more complex games, manipulating state variables with a higher degree of dimensionality.