1 Introduction

The video game industry is fast growing, and new technologies are being developed to provide realistic, immersive, and enjoyable experiences to users. Video games have been traditionally played using joysticks, keyboards, mouse, and gaming controllers, but this approach is not well suited for people with physical disabilities. Recently, there has been a shift in gaming devices to include more natural control modalities such as body gestures (i.e. Microsoft Kinect), but these controllers also present limitations for users with physical disabilities (i.e. upper limb differences or those on a wheelchair). A novel approach is to use Brain-Computer Interface (BCI) devices as a gaming controller, which can be used by both able-bodied participants and with disabilities [1]. As BCI technologies become more ubiquitous, lower cost, and its usage expands from research laboratories to user’s homes, BCI devices will be widely adapted for brain-controlled activities. Technology advancements in the past two decades allow the use of BCIs for decoding brain signals into control commands for both videogames [1] and drones [2]. In this paper, we present a brain-controlled drone racing game built with the Unity engine. In addition, we present the results of a user study performed to evaluate how the game affects participants affective state, participant’s performance, game usability, and to receive qualitative feedback towards the concept of brain-controlled games.

Drones are used for a wide range of applications, such as photography, power and pipeline inspection, search and rescue, and environmental monitoring. Furthermore, they are also widely used for racing and entertainment. As drone racing grows as a sport, we expect that drone racing videogames will also grow in popularity. Analogous to popular car racing games, a drone racing game can be used to overcome potential hassles of racing physical drones. For example, a game/simulation would allow players to practice at a lower cost, without the danger of crashes, and does not require special authorizations for flying what physical drones do (i.e. FAA regulations in USA).

The game allows users to control the movement of a virtual drone using an electroencephalography (EEG) headset while performing motor imagery, Fig. 1 shows a player wearing the BCI headset while playing the game. Motor imagery in this context is defined as the imagination of muscle movement, resulting in signals in the motor cortex area of the brain allowing the use of BCI devices to infer user’s intent [1]. The game allows selection of different drones to be used as the racing avatar, type of race (i.e. lap-based vs. drag), and customization of distractions (i.e. sound and visual). As the game requires the user to focus to perform motor imagery, customized distractions can be used to increase the difficulty and improve players ability. The current prototype of the gaming simulation is played using the Emotiv Insight BCI, a non-invasive 5-channel EEG headset that reads electrical signal from the scalp through semi-dry electrodes.

Fig. 1.
figure 1

Player controlling a virtual drone using a non-invasive brain-computer interface.

Full size image

This paper’s contributions include an analysis of the change in players affective state, analysis of player performance due to different backgrounds (i.e. gaming experience), evaluation of the system, and user’s perception towards brain-controlled drone racing games. The standard PANAS survey [3] was used to score the players positive and negative affect scores, the GEQ questionnaire [4] was used to evaluate the gameplay experience, furthermore, open-ended questions allowed participants to provide their insights about the experiment and the concept of controlling games with BCI devices.

The remaining of this paper is organized as the following: Sect. 2 presents previous work done in the fields of brain-controlled games and brain-controlled drones. Following, Sect. 3 describes the developed system, including technical details and gameplay characteristics. Section 4 presents the methodology of the user study and Sect. 5 contains the analysis and results. Concluding this paper, Sect. 6 summarizes our findings and discusses future work possibilities.

2 Related Work

Previous research has explored the use of BCI for gaming purposes, which can benefit both able-bodied and users with disabilities by creating a new communication channel between the player and the game [5]. Among different BCI technologies, non-invasive EEG headsets are the most suitable for gaming due to high temporal resolution, low cost (compared to other BCI’s), safety, and portability [1]. BCI have been used for action, simulation, puzzle, strategy and role-playing games [1]. For instance, a modified version of the popular World of Warcraft game changes the player’s avatar form accordingly to his/her affective state measured with a BCI [6]. Other examples include a brain-controlled Pacman and Pong game [7].

Previous research has also explored passive use of these devices for gaming. In the latter case, they can be used to measure the players cognitive activity and detect what they are experiencing during gameplay [8]. Such information could serve as an extra information channel and allow the game to adapt its track or even difficulty level to provide a better user experience. Previous work found that adapting the game to the user’s affective state allows adjustment of the information flow, providing an effective and pleasant experience [9].

The concept of controlling unmanned aerial vehicles using brain computer interfaces has existed for almost a decade, studies in this field date back to 2010 [10]. As drone-racing emerges as a sport, brain-drone racing has the ability to become a universal sport, allowing all participants to compete fairly independently of body type, gender or disabilities; the first brain-drone race was held at University of Florida in 2016 [11]. Although brain-control of physical drones already exists, there are many advantages for a videogame version, analogous to racing games that simulates real car racing. A videogame allows players to train controlling drones with BCI without the constraint of physical drones (i.e. cost, legislation, danger of accidents).

This paper presents the first brain-controlled drone racing simulation to be used as a game, including gamification characteristics such as score keeping and competition among players. Furthermore, this work differs from previous as it approaches the topic of brain-controlled videogames and drones from a human-computer interaction perspective. Lastly, this paper presents the results of an exploratory study containing both quantitative data about the participants affective state change during gameplay and qualitative data on the system.

3 System Description

The created prototype is a racing game where players can control a virtual drone around different racing tracks. The current iteration, the drone avatar is programmed to follow a pre-programmed path, while the user can control the speed of the drone using a BCI device. The player performs motor imagery to accelerate the drone and relax his/her mind to de-accelerate. The system was developed using the Unity Game Engine and integrated with the Emotiv Insight EEG headset.

The brain-controlled drone transverses through the environment by following a series of pre-determined waypoints. In order to ensure smooth turns through the track, a series of cubic Bezier curves with four control points were used to create a parametric curve. Since each turn would contain a copious number of points to transverse, a smooth turn is achieved as these points are always in close proximity to each other. An example can be seen in Fig. 2, which demonstrates the race track and the drone’s pre-programmed path in pink.

Fig. 2.
figure 2

Race track with drone pre-programmed race track demonstrated in pink. (Color figure online)

Full size image

Upon starting the game, the user is prompted to create or log in to their account. When the user logs in using appropriate credentials, they will have the option to view the leaderboard, play the game or access the game tutorial. If the user proceeds to play the game, they will be redirected to a drone selection menu (see Fig. 3) where they will be able to choose the drone that they wish to use. After selecting a drone, the player must select the desired track as shown in Fig. 4. An example of a race track is show in Fig. 5, on the top right corner the lap time is displayed allowing the player to keep track of his/her performance. At completion of each lap, the lap time is recorded and saved along with the user name and ID. The game data was stored into Firebase Realtime Database cloud-hosted database service.

Fig. 3.
figure 3

Drone selection menu.

Full size image
Fig. 4.
figure 4

Track selection menu.

Full size image
Fig. 5.
figure 5

Brain-controlled drone game stadium race track.

Full size image

4 Methodology

This study explored the use a BCI device as a control modality for videogames, more specifically drone racing games. A user study was performed to evaluate players performance accordingly to their different backgrounds, changes in their affective state while playing the game, and evaluate their experience of using a BCI device as a gaming controller.

4.1 Design

The experiment consisted of an exploratory user study where participants controlled a virtual racing drone around the track using a non-invasive 5-channel EEG headset. Study sessions were conducted individually with each user. Participants completed four laps around the track, and their performance (lap-time) was recorded for post-analysis. A pre/post questionnaire was used to elicit participants positive and negative affective state before and after the experiment through PANAS questionnaire to analyze how their affect state changed. A standard questionnaire was also administered after the experiment to assess the gameplay experience. Lastly, participants were asked to provide feedback about the experiment and concept of brain-controlled games through open-ended questions.

Two standardized questionnaires were used in this user study. The “Positive Affect and Negative Affect Scale” (PANAS) [3] was used as a pre/post questionnaire to calculate the affective state changes during gameplay. The “Game Experience Questionnaire” (GEQ) was used to receive feedback and evaluate the participants gameplay experience during the study [4].

PANAS.

This standard questionnaire is designed to calculate the user’s positive and negative affect state. In this study it was administered as a pre/post questionnaire, as participants answered the questions immediately before and after the gaming experience. This approach allows the comparison of pre/post PANAS scores, and analysis of how the game influenced the participants affect state. The PANAS consists of a 20 items scale, 10 items are related to positive affect (enthusiastic, interested, determined, excited, inspired, alert, active, strong, proud, and attentive) and 10 items for negative affect (scared, afraid, upset, distressed, jittery, nervous, ashamed, guilty, irritable, and hostile). Each item is scored on a 5-point Likert scale (1- not at all, a little, moderately, quite a bit, 5- extremely).

GEQ.

This standard questionnaire aims to assess game experience, scoring it in seven categories: competence, sensory & imaginative immersion, flow, tension/annoyance, challenge, negative affect and positive affect. The questionnaire was administered through a Qualtrics link immediately after participants ended the game. The core module of GEQ used in this study consists of 33 statements which the participant must score using a 1-5 Likert scale (1- not at all, slightly, moderately, fairly, and 5- extremely).

4.2 Participants

A total of 30 participants were recruited to participate in the experiment, all of them were students at the University of South Florida, Tampa, Florida, USA. Eighteen of them were male, 12 females; 24 participant’s age were between 18 and 24, and 6 participants were between 25 and 34 years old. Eighteen participants reported that they play videogames in a weekly basis.

4.3 Equipment

The experiment was performed using the Emotiv Insight headset, a non-invasive 5 channel EEG headset (Fig. 6). The headset is used to read the electrical activity on the participant’s scalp through the use of semi-dry electrodes. The Emotiv ControlPanel software was used to interface with the hardware device, reading and decoding the brain activity into either a neutral or action state. Furthermore, the Emotiv Emokey software is connected to the ControlPanel and it emulates keyboard strokes to control the game based on the current brain activity state (neutral vs action).

Fig. 6.
figure 6

Emotiv Insight, the BCI headset used to control the game.

Full size image

4.4 Procedure

Each participant attended one session that lasted approximately 30 min, and it was comprised of four phases: (1) introduction; (2) pre-survey; (3) game; and (4) post-survey. During the introduction, the procedures were explained, and informed consent was acquired in order to proceed with the experiment. During the pre-survey phase, participants were asked to answer a series of questions using a Qualtrics survey on a provided computer. Questions were designed to elicit demographic data, handedness, gaming background, coffee and energy drink consumption, and how many hours participants slept during the prior night. During this phase, participants also answered the PANAS standard survey.

During the third phase participants were instructed on how to perform motor imagery to control the game and were assisted in wearing the BCI headset, ensuring good electrode-skin contact and signal quality. A research team member aided each participant to create and train a profile using the Emotiv ControlPanel software. This training phase consisted of capturing and recording brain activity during a neutral phase (baseline) and then again during the execution of motor imagery, such data was used to train the Emotiv software algorithms and allow it to decode the brain activity into game commands. Following this, the participant controlled the drone to perform 4 laps around the race track, with sounds disabled and visual distractions (i.e. fireworks, flags) enabled, each lap completion time was recorded.

Lastly, the participant was asked to complete the post-ex-experiment Qualtrics. Feedback about the experience was acquired through the Game Experience Questionnaire (GEQ) [4]. The participant also completed the PANAS survey [3] once more, allowing for comparison of positive and negative affects prior and after the game. Finally, open ended questions were presented to acquire qualitative feedback about the experience, and the concept of using BCI devices for gaming.

5 Results and Discussion

5.1 PANAS: Affective State Change

The responses to the PANAS survey can be seen in Fig. 7, which demonstrates the individual items mean score across participants for before and after the experiment. The average positive and negative affect scores were calculated by adding its respective PANAS items and are displayed in Table 1. A paired T-test demonstrated a significant increase in the average positive affect from 34.7 to 37.43 with a p value of 0.0039, there was no significant change in the decrease of negative affect score (Table 1). Furthermore, as shown in Table 2 there was also significant increases in positive items (alert, inspired, attentive, proud) and decreases in negative items (distress, scared, afraid).

Fig. 7.
figure 7

PANAS scores prior and after experiment.

Full size image
Table 1. PANAS analysis results. Positive and negative score prior and after experiment.
Full size table
Table 2. Paired T-Test result in PANAS items. Items not displayed were not statistically significant. Positive affective items displayed in white, negative items displayed in red.
Full size table

Our analysis of participant’s affective state changes demonstrated interesting results. The statistically significant increase in the average positive score indicates a positive experience to participants. We believe that this result is related to participants enthusiasm to be able to control a video-game solely through their brain-waves, and due to the novelty of our system. Our data also demonstrates an increase in alertness and attention, suggesting that our simulation can potentially be used for ADHD therapy. Additionally, participants also benefited from a decrease in negative items; we believe that the game immersion lead to a decrease in distress. Moreover, a decrease in two fear-related items (scared, afraid) suggests that participants were apprehensive prior to trying the BCI device and got more comfortable during the experiment.

5.2 GEQ: Gameplay Experience

Results from the Game Experience Questionnaire can be seen in Table 3, which displays the mean, and the standard deviation across all participant responses. The GEQ score guidelines described in [4] were used to calculate category scores for competence, sensory & imaginative immersion, flow, tension & annoyance, challenge, negative affect, and positive affect; which are shown in Fig. 8. The highest scores (above 4.0) were all positive items: impressiveness, challenge, fun, and fully occupation with the game; while the lowest scores (below 2.0) were negative items: annoyance, boredom, and irritation.

Table 3. GEQ results (mean, std deviation and variance) for each questionnaire item
Full size table
Fig. 8.
figure 8

GEQ categories average score across participants. Scale: 1 = “not at all”, 2 = “slightly”, 3 = “moderately”, 4 = “fairly”, 5 = “extremely”.

Full size image

5.3 Performance

During the experiment, each participant completed four laps around the race track with the respective lap times being recorded. The average time for lap completion can be seen on Table 4. A paired T-test was executed to compare the average times of male and female, as well as participants with gaming experience versus participants without. Although, average results were different across populations, our results were not statistically significant (P > 0.05).

Table 4. Average time to complete lap around race track.
Full size table

5.4 Qualitative Feedback

In addition, participants also provided feedback about the experiment and the concept of using BCI for gaming. Majority of the feedback was positive, for instance, when asked if they had any extra comments about the experiment 12 out of 30 participants described the experience enthusiastically (i.e. “It was a lot of fun!”, “Very exciting!”, “The game was impressive”). Furthermore, participants stated that BCI for gaming is a very promising concept, and that they expect it to become popular in the future. Such feedback with our expectation that BCI devices can be used as a gaming controller for drone racing games, proving a positive experience to gamers. Additionally, three players described that the game allowed them to improve their focus, which is aligned with the results presented in Sect. 5.1. This data enforces our belief that this simulation can potentially be used as a therapy tool to help users diagnosed with ADHD improve their attention. The open-ended questions were also useful to acquire improvement suggestions for our system. Participants suggested the development of a “free flight” mode to allow controlling the drone in three dimensions, improvements of graphics, and improvements in the flight dynamics (i.e. drone acceleration could be more realistic). It was also suggested by three participants that BCIs should be integrated with virtual reality games.

6 Conclusion

In this study we presented a brain-controlled drone racing game and the results of a user study where participants had to control the virtual drone around the race track 4 times. Our results demonstrate a statistically significant increase in the positive affect of participants after the experiment, indicating that the game was a pleasant experience for its users. Furthermore, our analysis also discovered significant increases in pride, alertness, inspiration and attention scores for participants, while decreasing distress, scariness and fear. Our analysis did not show any statistical significance in performance based on the comparison of gender and previous gaming experience. Positive feedback was received from users, both through a standard game experience questionnaire and open-ended questions. Additionally, 29 out of 30 participants stated that they would purchase a brain-computer interface device for gaming purposes. These data suggest that BCI’s have potential as the next generation game controllers.

In future iterations of this project we plan to improve the game based on participants suggestions. Moreover, we propose further research to understand what factors influence players performance, as we did not find significant results in this matter.