Keywords

1 Introduction

1.1 Current Issues in the Assessment of Executive Functions

The term executive functions refers to a wide range of high-level cognitive processes in the prefrontal lobe [1] associated with reactive inhibition and the regulation of goal achievement behavior [2]. Inhibitory control (i.e., resisting one’s initial impulse or a strong pull to do one thing and instead act more wisely) [3], working memory (i.e., holding information in mind while performing one or more mental operations) [4], and cognitive flexibility (i.e., the ability to flexibly adjust to changing demands or priorities, to look at the same thing in different ways or from different perspectives, as required for set shifting or task switching) [5] have been identified as the three main components of executive functions [6]. Executive functions are essential for many everyday life activities across a wide range of contexts, such as the ability to work, function independently at home, or maintain social relationships [3, 7]. Furthermore, these cognitive skills have been found to be predictive of health, wealth, academic success, and quality of life throughout life [8].

Given the relevance of executive functions in daily life and the great complexity of cognitive impairments associated with them, an accurate and efficient assessment is of particular relevance [9]. The most widely used tools for the executive functions assessment are paper-and-pencil, and computer-based tests such as the Trial Making Test (TMT) [10], the Stroop test [11], and the Wisconsin Card Sorting Test (WCST) [12, 13]; however, these tests have several important limitations, including:

  • Low accessibility: Traditional neuropsychological instruments have been developed to be used in clinical settings; hence, they are costly and require the presence of a health professional (i.e., neurologist or neuropsychologist). Accordingly, these measures are difficult to access and are often applied only retrospectively, that is, after the cognitive impairment has been detected. This delay has a serious negative influence on the effectiveness of the intervention and the treatment options [14]. In addition, most of these tests have been designed and tested on a population with medium educational levels; therefore, they are not suitable for individuals with less education [15];

  • Low ecological validity: Most neuropsychological tests are unable to measure complex real-life activities [16] and fail to predict the way in which individuals manage these activities in their daily living [17]. For example, performance on traditional tests of executive functions, such as the WCST and the Stroop test, does not reflect activities of daily living and do not predict functional performance across a range of real-world situations [18]. In addition, although commonly used for clinical and research purposes, traditional neuropsychological tasks are thought to lack the stresses and distractions of real-life settings [19]

  • Low quality of assessment data: Neuropsychological tests are often viewed as repetitive, frustrating, and boring, leading to individuals’ disengagement, which may negatively affect the quality of assessment data [20].

In response to the limits of conventional assessments of executive functions, neuropsychologists are increasingly emphasizing the need for a new generation of “function-led” tests that would be able to asses real-world functioning and tap into a number of executive domains [21, 22]. A number of ecological tasks have already been developed for the evaluation of executive functions, such as the Multiple Errand Test (MET) [23], or the Behavioral Assessment of the Dysexecutive Syndrome (BADS) [24]. Even if this new type of tasks provided more accurate estimates of the individual functional performance in activities of daily living compared to traditional neuropsychological tests [18] they would have several limitations that restrict their adoption. Among the main one, ecological tasks are time-consuming and require the presence of a specialist [25]. In addition, data collected in the real-world environment are frequently confounded by numerous factors, which cannot be captured by these measures [26].

1.2 Information and Communication Technologies for the Neuropsychological Evaluation of Executive Skills

With the aim of overcoming the existing problem in neuropsychological assessment of executive functions, Information and Communication Technologies (ICT) are increasingly employed, particularly virtual reality and video games, which are described in detail in the following paragraphs.

Virtual Reality.

By definition, virtual reality allows users to navigate and interact with a three-dimensional computer-generated environment in real-time [27]. This technology has been successfully applied in the assessment of cognitive skills, including executive functions, for decades [28, 29], when it was still much less advanced from the point of view of both hardware and software. Several studies have reported that virtual reality was effective in offering a more realistic, ecologically valid alternative to traditional neuropsychological tasks assessing executive functions [22, 30], providing a balance between naturalistic observations and the need to control the key variables [31]. Virtual environments, in fact, thanks to the possibility to provide various complex conditions [32] and to record and analyze individual performance [33], combine the rigor and the control of the laboratory measures with simulations that reflect real-life situations [34].

Several ad hoc virtual contents have been developed for the assessment of executive functions. In particular, different virtual shopping environments have been tested over the last decade to evaluate executive functions deficits in people with frontal head injuries or stroke, such as the Virtual Multiple Errand Test (VMET) [35, 36], the Adapted Four-Item Shopping Task [37], and the Virtual Supermarket (VSM) [38]. Another virtual reality tool that has been developed to test EF is the Jansari Assessment of Executive Functions (JEF) [39], which requires individuals to complete different tasks in an office, the Virtual Library Task (VLT) [40], that required individuals to complete tasks as if they were running a library, and RehabCity [41], which assessed executive functions by asking the users to accomplish goals that require problem-solving. Furthermore, a recent study reported the efficacy of three immersive virtual reality tasks (i.e., a seating arrangement task, an item location task, and a virtual parking simulator) in assessing executive functions in a healthy population of younger and older adults [42].

Video Games.

Every day, about 2.6 billion people worldwide, 61% male and 39% female, with a mean age of 34 years old, play video games on their personal computers, mobile phones, or consoles [43]. One of the main reason for their success and diffusion is related to the fact that computer games are designed to be motivating and challenging [44], providing easy access to “fun” and to a sense of engagement and self-efficacy [45]. This ability to motivate the user [46], as well as to elicit positive emotions [47] and a short-term increase in subjective well-being [48], has in the last years begun to be exploited for purposes beyond entertainment, including the mental health panorama [49, 50]. In the neuropsychological field, in particular, video games appear relevant also because of their nature of requiring complex cognitive skills, including planning and strategizing, which place executive-functioning demands on players, such as selective attention and inhibition [51].

As early as in 1987, it was observed, for the first time, that famous commercial video games such as Donkey Kong (Nintendo) and Pac-Man (Atari) can have a positive effect on cognitive skills, improving the reaction times of older adults [52]. A few years later, in 1989, Space Fortress, the first of a series of non-commercial computer games designed by cognitive psychologists as a training and research tool [53] - often defined in literature as “serious games” as they use gaming features as the primary medium for serious purposes [54] - became so successful that it was added to the training program of the Israeli Air Force. From that moment on, efficacy has been demonstrated not only for non-commercial video games developed ad hoc, but also for commercial video games, in the training of several cognitive abilities [55], especially perceptual attentional skills [51], mental spatial rotation abilities [56], and executive functions, such as task-switching [57] and working memory [58].

Regarding the use of video games in the assessment of executive functions, especially in these last years, different serious games and game-based versions of neuropsychological tests have been tested. For example, the time to complete an ad hoc low-cost computer game, the Stroop Stepping Test (SST), developed using dance pads adapted from exercise-based video games, has been reported to correlate significantly with traditional measure of executive functions (i.e., TMT, Stroop task) among older people [59]. Another serious game has been developed to assess the cognitive status of the elderly through a touch-based device, showing a strong relationship between the video game performance and inhibitory ability as measured in the Stroop task [60]. In a subsequent study, the same authors tested the effectiveness of a serious game developed with a low-cost game engine to assess executive functions skills, reporting strong correlations between classic test (i.e., WCST, Stroop task, N-Back task) and the game performance [61]. Furthermore, a computational model based on game data obtained by playing an ad hoc video game has been showed to be effective in the estimation of TMT performance [62]. Finally, two recent studies have shown a correlation between performance on serious games developed for the assessment of executive functions (i.e., EXPANSE, Kitchen and Cooking) and scores on TMT [63, 64].

1.3 Aim of the Study

As described in the above sections, several previous studies have provided evidence on the efficacy of both virtual reality content as well as video games in the assessment of executive functions [22]. In particular, they have demonstrated the effectiveness of using virtual reality contents or video games created specifically for the measurement of these cognitive functions in the neuropsychological assessment. However, studies conducted thus far have some limitations, specifically:

  • They have focused only on the testing of non-commercial contents: These types of solutions are often expensive and time-consuming in their development [65], as well as focused on a single specific population, which makes it difficult to replicate the study and to standardize the data.

  • They have investigated separately virtual reality contents and video games: Thus far, only non-gamified virtual contents [35], or non-immersive video games (i.e., played on a desktop) [62], have been tested.

A new type of interactive content, in particular, virtual reality video games, could address such limitations. Immersive video games, which entered the market in 2016 thanks to the commercial diffusion of products, such as HTC Vive (Valve) and PlayStation VR (Sony Corp.), could be particularly interesting not only for entertainment purposes, but also in many other fields, including psychology and neuropsychology [66, 67]. Thanks to unique features of immersion (i.e., the perception of being physically present in a non-physical world) [68] and interactivity (i.e., the ability of the digital content to respond to the users’ actions and be modified by them), virtual reality video games lead in fact to incomparable opportunities, significantly different from those provided by non-immersive gaming, such as traditional console or PC games [67, 69] Interestingly, a previous study [70] has reported the efficacy of a virtual reality video game, in particular, an exercise-based dance game (i.e., DANCE), for the training of executive functions in older people. However, this study tested only a non-commercial virtual reality game, and no indications regarding the possible effectiveness of this type of content even for the assessment of executive functions were given.

Within the context described above, this study aimed to compare the effectiveness of a commercial virtual reality video game (i.e., a dance game) in the assessment of executive functions of healthy young adults with the effectiveness of traditional neuropsychological measures. Furthermore, the specific objective of this experiment was to implement a structural equation model to estimate the association of the traditional neuropsychological measures (as a latent variable) with the virtual reality game performance (see the conceptual model in Fig. 1). The main hypotheses explored by the research were:

Fig. 1.
figure 1

Conceptual model of relationship among variables.

Full size image

Hypothesis 1.

Performance on the traditional neuropsychological test will correlate significantly with the performance on the virtual reality video game.

Hypothesis 2.

The direct path between scores on the virtual reality video game and scores on the traditional neuropsychological test (i.e., time to complete the tasks) will be negative.

2 Materials and Methods

2.1 Participants

38 participants – 13 females (34.2%) and 25 males (65.8%) - were recruited among students and personnel of the University of Milano-Bicocca and of other universities in Milan. Participants’ average age was 25.8 ± 4.14 years (min–max = 18–35), average level of education 16 ± 3.05 years (min–max = 11–21) No credits (ECTS) or economic rewards were provided during the research. To be included in the study, individuals had to meet the following criteria: (1) age between 18 and 35 years old; (2) no major medical disorders (heart disease or high blood pressure, neurological disorders, epilepsy); (3) no left-handed; (4) no presence of pharmacotherapy (psychoactive drugs, anti-hypertensive, anti-depressants); (5) no significant visual impairment (all with normal or corrected-to-normal visual acuity); and (6) not having previous experience with the Audioshield game. Before participating, all participants were provided with written information about the study and were required to give written consent in order to be included. The Ethical Committee of the University of Milano-Bicocca approved the study. The research was conducted in accordance with the American Psychological Association ethical principles and code of conduct.

2.2 Measures

Psychological Assessment.

The following self-report questionnaires were administered to the participants at the start of the experimental session.

Demographics.

The participants were asked to indicate their gender (female/male), age (years old), and years of education.

Gaming Habits and Virtual Reality Knowledge.

Individuals were asked to indicate their gaming habits (mean hours spent gaming per week), their previous experience with games included in the study (yes/no), and their previous experience with virtual reality (yes/no); furthermore, they were asked to assess their knowledge of virtual reality on a 7-point Likert scale (from “not at all” to “very much”).

Cognitive Assessment.

To test the participant’s executive functions the following traditional neuropsychological tests was used:

Trial Making Test (TMT) [10]. The TMT is one of the most widely used instruments to measure executive functioning in neuropsychological assessment [71, 72]. In this study, we adopted the Italian version included in the ENB-2 battery [73]. The test consists of two parts (A and B) that clinically assess psychomotor speed (Part A) and mental flexibility (Part B) [74]. The individual performance on TMT is given in the form of a score on TMT-A and on TMT-B, which represents the time (in seconds) taken to complete the task.

2.3 Virtual Reality Video Game

In this study, we adopted Audioshield, a virtual reality dance game created by Dylan Fitterer for the HTC Vive (Valve) and launched in April 2016. We selected this particular game among other commercial virtual reality games for two main reasons. First, this genre of video games (i.e., dance game), which incorporates both cognitive engagement and physical activity, may have particular benefits for executive functions [75, 76]. Second, this title has some characteristics that could be closely related to executive functions, such as the inhibition of responses and working memory.

Audioshield, in particular, is a dance game in which orbs come flying towards the player, who needs to follow the beat of the music to successfully hit them. The player uses the HTC Vive’s handheld motion-sensing controls to operate two shields, blue and red. Red balls must be deflected with the red shield controlled by the right hand while blue balls must be deflected with the left hand. Purple orbs require a combination of both arms. During the game, the color and the direction of the orbs that the player has to hit changes continuously, for example, the blue balls can come from the right (or red from the left) and require the user to respond correctly very quickly.

As detailed later in the procedure, after a brief explanation of the video game by the experimenter, the participants were asked to practice for about 2 min using the song “Engage” (difficulty: normal; shield: gladiator; environment: horizon). Subsequently, the individuals were asked to complete the song “I drop gems” (difficult: normal; shield: gladiator; environment: horizon) while their performance was evaluated. It took about 5 min to complete the gameplay. The dependent variables used to evaluate the video game performance were: (a) the technical score (i.e., a measure of how many orbs players hit, with 10.00 being a perfect technical score with no misses and 0.00 being all missed), (b) the number of balls the player missed, and (c) the numbers of orbs the player hit.

Regarding the hardware, the virtual reality setting adopted in this study included the following units:

  • HTC Vive system (Valve), a consumer-grade virtual reality system designed for the use in video games, consists of an HMD, two controllers, and two infrared laser emitter units. The headset covers a nominal field of view of about 110 (approximately 90 per eye) through two 1080 × 1200 pixel displays that are updated at 90 Hz. Games played with the Vive allow physical movement within a play area that is limited to 4 × 4 m.

  • A portable computer (MSI GT73 VR, Intel® Core™ i7 processor, GeForce® GTX 1070 8 GB, 17.3” Full HD 1920 × 1080 pixel displays).

2.4 Procedure

After individuals gave written informed consent to participate, they completed the self-report questionnaire assessing demographics, gaming habits, and virtual reality knowledge. Subsequently, the participants underwent the neuropsychological evaluation. They were required to complete the paper-and-pencil version of the TMT (Part A and Part B). Once this phase was completed, the Vive was connected to the PC through a 5 m cable with an HMDI connection, a USB 2 connection, and power. Participants were asked to wear the HMD and were given the HTC Vive controllers. They were then provided with a training period of about 2 min on Audioshield using the song “Engage” to familiarize themselves with the video game tasks and controllers. In the next step, individuals completed the song “I drop gems” while their performance was evaluated. The audio level was set to 45 for all participants.

Subjectively, tracking appeared stable when using this configuration, and the video game was playable with no visible tracker artifacts. All measurements were taken in an 8 × 5 m room with a 3.2 m high ceiling lighted by fluorescent lighting, with no reflective surfaces and no exposure to natural lighting. In the center of this room, a 4 × 4 m grid (i.e., the play area) was drawn on the floor using string and chalk, with grid lines drawn 1 m apart. The entire procedure took about 40 min to complete.

2.5 Analytic Strategy and Data Modelling

This research adopted the structural equation modeling (SEM) approach to analyze the cumulative network of relationships among the variables under study. SEM is a widely used quantitative technique for testing models by allowing a simultaneous estimation of a system of regression equations involving multiple endogenous variables as well as their measurement errors [77]. The conceptual model (Fig. 1) was designed by considering game performance (as measured by two observed variables: Audioshield technical score and number of hits) and executive functioning [as measured by two observed variables: psychomotor speed (TMT - Part A) and mental flexibility (TMT - Part B)] as an endogenous latent variable.

The Maximum Likelihood method [78] was adopted to determine the parameters for the SEM analysis. The practical and statistical significance of the model was evaluated using the following goodness-of-fit indices: Root Mean Square Error of Approximation (RMSEA, RMSEA < 0.05) [79]; Standardized Root Mean Square Residual (SRMR, SRMR < 0.05) [80]; Normed fit Index (NFI, NFI > 0.95) [81]; Tucker-Lewis Index (TLI, TLI > 0.95) [81]; and Comparative Fit Index (CFI, CFI > 0.95) [81]. All models were tested using Amos software. In keeping with the current literature (e.g., [82]), we estimated confidence limits using both Monte Carlo simulation and bootstrapping methods with a set of random samples (k = 500) and 95% confidence intervals for unstandardized effects.

3 Results

Main descriptive statistics and zero-order correlations of considered variables are summarized in Table 1.

Table 1. Main descriptive statistics and zero-order correlations for TMT scores and Audioshield performance (N = 38). Note: * p < .05, ** p < .01 *** p < .001
Full size table

The results of the correlational analysis revealed that, in general, game performance was negatively associated with TMT scores (i.e., time to complete the TMT-A and TMT-B). In particular, statistically significant negative correlations were found between Audioshield scores and both psychomotor speed (TMT-part A, r = −.477, p < .001) and mental flexibility (TMT-part B, r = −.417, p < .001). Interestingly, the number of Audioshield total hits correlated negatively with mental flexibility (TMT-part B, r = −.375, p < .001) whereas it did not correlate with psychomotor speed (TMT-part A, r = −.241, p = n.s.). A more detailed picture of the set of relationships among variables under study can be obtained by estimating the structural equation model (See Fig. 2).

Fig. 2.
figure 2

Results of the structural model estimation. Standardized direct effects were reported, bootstrap estimation (k = 1,500) was conducted along with 95% confidence intervals

Full size image

The results of the structural equation model revealed an excellent general fit of the conceptual structure with the observed data. In fact, all goodness of fit indexes suggested that the data fit the measurement model well: χ2(1) = 1.41, p = .235, NC = 1.41, RMSEA = .050, pclose = .257, NFI = .968, NNFI = .988, and CFI = .988, supporting the robust effects among the variables under study both conceptually and statistically. In particular, performance on Audioshield was negatively associated with the time to complete the TMT part A and B (β = −.77, p < .01, 95% CI [−1.82; −.099]), with higher scores on the virtual reality game indicating less time to complete the tasks. The results support the H2, suggesting that scores on Audioshield can be used to reliably estimate the results of TMT.

4 Discussion

Starting from the first hypothesis of the study, the proposed significant correlation between performance on the virtual reality game and scores on the TMT was supported. A traditional paper-and-pencil task was used to assess various cognitive processes, including psychomotor speed and executive functions (i.e., task-switching, flexibility) [74, 83], especially the part B (TMT-B) [71, 84]. In particular, both the time to complete the TMT-A and the TMT-B correlated significantly with the performance on the virtual reality game (i.e., number of orbs hit by the players and technical score on Audioshield), suggesting that the virtual reality game may measure the same aspect of cognition as the traditional paper-and-pencil TMT.

Regarding the second hypothesis, as expected, the structural equation model revealed that performance on the virtual reality game can be used to estimate scores on the TMT. In particular, the results showed that performance on Audioshield was negatively associated with the time to complete the TMT part A and B, with higher scores on the virtual reality game indicating less the time to complete the tasks.

Our findings are consistent with previous results showing a correlation between the performance on some video games and the scores on the TMT [63, 64] as well as the possibility of predicting scores on this traditional test using individuals’ game performance data [62]. For example, it has been recently observed that individuals’ game performance data (i.e., latency times, correct answers) on a serious game called EXPANSE, which aims to assesses executive functions, correlated strongly with standard task aimed at assessing these functions, including the TMT-A and TMT-B, in a sample of 354 healthy adults [63]. Similarly, another study conducted with 12 patients with Alzheimer’s disease, focusing on 9 mild cognitive impairments, reported a correlation between the performance on a serious game board-game named Kitchen and Cooking and the TMT [64]. Furthermore, a computational model based on the performance on Scavenger Haunt, a computer game designed heavenly inspired by the TMT, was reported to be effective in predicting performance both at the TMT-A and the TMT-B [62].

However, it is important to underline that our study found important differences compared to the research carried out in the past. First, in this study, we tested a commercial video game instead of a serious game [63, 64] or a game-based version of the TMT [62]. In addition, we tested an immersive video game in this experiment while other studies used non-immersive video games (i.e., played on desktops) [62,63,64]. Compared to previous literature, in fact, the current research is the first to provide preliminary evidence on the effectiveness of immersive commercial video games, a type of video game that combines the advantages of commercial games with those offered by virtual reality [67].

Although the results of the present study could be interesting for their possible applications, this research has some important limitations that could affect the generalizability of the findings. First, it is important to underline that the obtained results refer to a specific virtual reality video game, in this case, Audioshield, a dance game launched in April 2016. In the future, it would be interesting to explore this particular genre of games more in-depth, as these games are deemed interesting with respect to the functioning of executive functions [70]. For example, it would be interesting to test Beat Saber (Beat Games), since it has very similar characteristics as the game tested in this study, and it was the most successful virtual reality game title of 2018 on the Steam platform (Vive) [85]. In addition, future studies should investigate the effectiveness of other genres of video games played in virtual reality, for example, puzzle and casual games appear to be particularly interesting, since they are genres with potentially very relevant features in relation to the functioning of executive functions [86, 87].

Second, the results emerging from this study refer to the specific virtual reality system in use [i.e., HTC Vive (Steam)]. Future research should investigate the adoption of other commercial virtual reality systems with different characteristics in terms of immersion and interactions with the game, including other off-the-shelf virtual reality systems, such as Oculus Go (Oculus) or Playstation VR (Sony corp.), for instance. These products are very appealing, since unlike the system tested in this study, they could be used very easily even by non-expert operators and are more budget-friendly.

Third, in this study, a specific test was used to measure executive functions (i.e., the TMT). It would be interesting to adopt also other traditional tests, such as the WCST or the Stroop test, that assess different aspects of executive functions. Finally, another limitation of this study is related to the small sample size and the specific sample included in the study comprising young adults who played often (more than 12 h per week) and had a low knowledge of virtual reality (68% of the sample did not try it before the experiment).

5 Conclusion

Our findings show the feasibility of using a commercial virtual reality game to assess executive functions and cognitive abilities, as measured by the TMT. In particular, user performance on Audioshield correlated significantly with time to complete the TMT-A as well as the TMT-B. Furthermore, the most important finding of this exploratory study was that the performance on the immersive video game predicted the TMT scores (i.e., time to complete the TMT-A and TMT-B).

Future studies should investigate the possibility of using commercial virtual reality games to assess executive functions as well as of other cognitive skills, since they represent a very promising tool for the neuropsychological assessment. Using off-the-shelves video games rather than video games created ad hoc could have several advantages for the assessment of executive functions as well as other cognitive processes, in particular:

  • Low-cost and ready-to-use: Commercial video games are low-cost and their production time is short, unlike serious games and game-based version of the neuropsychological test.

  • Advanced graphic quality and gameplay mechanics: Commercial video games are more costly and a greater number of people is involved in its development compared to ad hoc computer games, for example, Red Dead Redemption 2 and GTA V (Rockstar Games) costed about 944 million and 256 million dollars, respectively, involving each one a team of at least 1.000-person [88, 89] to ensure an astonishing quality of the game, which has a significant effect on the overall gaming experience. This would not be achievable with smaller budgets and staff.

  • Possibility to collect the data on very large populations: Commercial video games are played in different cultural and social contexts, which makes it easy to replicate studies in different contexts and to collect the data from large samples.

In addition, virtual reality games have also several advantages compared to video games played on desktops, including:

  • Require high physical and cognitive involvement: Immersive video games require a high involvement of the player at both the motor and cognitive level, with a greater engagement compared to non-immersive games [67, 69].

  • Elicit more intense positive emotions in the players: Video games played in virtual reality elicit a higher self-reported sense of happiness in users in comparison to non-immersive games [67];

  • Allow to gather multiple types of data of the user: Video games played in virtual reality allow the collection of a wide variety of data compared to non-immersive games (e.g., scores, time to complete the task, etc.), such as those about the player’s movement within the virtual environment.