1 Introduction

Cerebrovascular accidents (i.e., strokes) are a leading cause of adult long-term disability worldwide [1], with up to 74% of stroke survivors requiring assistance with daily life activities due to motor impairments (e.g., an inability to move the affected side) [2]. Rehabilitation for these individuals is difficult because most current training options require some volitional movement to train the affected side. However, research has shown that individuals with severe stroke may have modest benefits from action observation, virtual reality (VR), and brain-computer interfaces (BCIs). First, in healthy subjects, action observation of motor actions has been shown to facilitate the formation of motor memories and effects of physical training. Moreover, it has been shown that action observation in association with physical training can enhance the effects of motor training after stroke [3], eliciting motor-related brain activity on the lesioned hemisphere which leads to modest motor improvements after severe stroke [4, 5].

Second, action observation in VR through a head-mounted display (HMD) has been shown to increase motor activity in both the healthy and the post-stroke brain [6, 7]. Furthermore, gamification mechanisms, which capitalize on motivational factors that are essential for recovery and adherence to the treatment, can be built into head-mounted VR rehabilitation environments [8]. VR rehabilitation environments can also be tailored to allow for the personalization of training, self-monitoring, and monitoring by therapists. Additionally, they can enable patients to play a more active role in their rehabilitation by taking part in the development process through participatory design approaches [9].

Finally, another treatment for individuals with severe stroke is the use of neurofeedback through BCIs, which also does not require active motor control. BCIs are communication systems capable of establishing a pathway between the user’s brain activity and a computer system [10]. The most common brain signal acquisition technology in stroke BCIs is non-invasive electroencephalography (EEG) [10], as it is the most cost-effective solution for brain-computer interfacing [11]. EEG signals are distinguished by different wave patterns in the frequency domain called EEG bands or rhythms. These EEG rhythms are divided into different ranges including Delta (1–4 Hz), Alpha (8–13 Hz), Beta (13–30 Hz), Theta (4–8 Hz), and Gamma (25–90 Hz), and each rhythm or combination of rhythmic activity has been previously related with sensorimotor and/or cognitive states [12]. For example, during a motor attempt, the temporal pattern of the Alpha rhythm desynchronizes, forming a special shape which, when inverted in polarity (negativity is up), is reminiscent of the Greek letter μ (mu). This rhythm is also named Rolandic mu or the sensorimotor rhythm (SMR) because of its localization over the sensorimotor cortices of the brain. Mu-rhythms are considered indirect indications of functioning of the mirror neuron system [13] and general sensorimotor activity, and are often detected together with Beta rhythm changes in the form of an Event-Related Desynchronization (ERD) when a motor action is executed [14]. These EEG signatures are primarily detected during task-based EEG (e.g., when the participant is actively moving or imagining movement). Moreover, specific signatures from resting-state EEG activity—that is, EEG activity in the absence of a task—have been also utilized as a biomarker in research for motor deficits [15]. When combined with neural injury information, resting EEG measures, such as of frontoparietal activity, can be used to predict the efficacy of stroke therapy [16].

In EEG-based BCIs for rehabilitation, motor-related brain signals generated by the patient are reinforced by rewarding feedback, even if the patient cannot move [10]. In this way, BCI feedback can be used to strengthen key motor pathways thought to support motor recovery after stroke. Such feedback has shown modest success in motor rehabilitation for severe stroke patients [17]. The fusion of BCI and VR feedback allows for a wide range of experiences where participants can feel immersed in various aspects of their environment - either in an explicit or implicit manner-and which they can control using only their brain activity [18, 19]. This direct brain-to-VR communication can induce a sensorimotor contingency between one’s internal intentions and the environments responsive actions, increasing the sense of embodiment of the virtual avatar [20].

In this study, we combined the principles of action observation, virtual reality, and BCI for stroke rehabilitation to try to elicit optimal rehabilitation gains. We developed a platform called REINVENT which takes post-stroke brain signals indicating an attempt to move and drives the movement of a virtual avatar arm, providing patient-driven action observation in head-mounted VR [21].

The purpose of this study is twofold: (1) to describe the new modular REINVENT architecture that provides increased accessibility to the system, and (2) to test whether REINVENT is feasible to use across repeated sessions to strengthen motor-related brain signals in an individual after stroke.

2 Methods

2.1 Participant

For this study, a 69-year-old male stroke survivor was recruited. The participant suffered a right hemisphere middle cerebral artery stroke 9 years prior resulting in severe left hemiparesis in his upper arm. Upon inclusion in the study, the participant was unable to actively extend his wrist or fingers greater than 5°. The experimental protocol was approved by the University of Southern California Health Sciences Campus Institutional Review Board (IRB) and performed in accordance with the 1964 Declaration of Helsinki. Informed consent was obtained from the participant upon recruitment.

2.2 Experimental Setup

The experimental setup was composed of a desktop computer (OS: Windows 10, CPU: Intel® Core™ i7-6700 at 4.00 GHz, RAM: 16 GB DDR3 1600 MHz, Graphics: NVIDIA GeForce GTX 1080), running the VR task, and the EEG and EMG data acquisition. For EEG acquisition, a Starstim 8 (Neuroelectrics, Barcelona, Spain) system was used. Starstim is a wearable, wireless sensor with 8 EEG channels and a triaxial accelerometer, allowing for the recording and visualization of 24-bit EEG data at 500 Hz (Fig. 1a). The spatial distribution of the electrodes followed the 10–20 system configuration [22] with the following electrodes over the somatosensory and motor areas bilaterally: Frontal-Central (FC3, FC4), Central (C3, C4, C5, C6), and Central-Parietal (CP3, CP4) (see Fig. 1 for set-up). The EEG system was connected via Bluetooth to the dedicated desktop computer for raw signal acquisition and processing. For the EMG data acquisition, a Delsys Trigno Wireless System (Delsys, MA, USA) was used, incorporating 4 differential Ag active electrodes with 16-bit A/D converter at 2000 Hz and 3-axes acceleration data at 150 Hz and 8-bit ADC resolution. The EMG sensors were placed on Extensor Digitorum Comunis (EDC), Flexor Carpi Ulnaris (FCU), Biceps Brachii (BB) and Triceps Brachii (TB) muscles of the paretic arm (Fig. 1c). The raw data was acquired from Delsys through the Lab Streaming Layer protocol [23] and processed through a custom script in Matlab (MathWorks, MA, USA). For delivering the VR feedback to the user, an Oculus Rift CV1 HMD was used (Oculus VR, Menlo Park, California, USA). The HMD has two OLED displays, 1080 × 1200 resolution per eye, at 90 Hz refresh rate, and 110° field of view. The HMD also features 6-DoF tracking (3-axis rotational tracking and 3-axis positional tracking), and integrated headphones with 3D spatial audio (Fig. 1b). Finally, the VR task was designed in Unity game engine (Unity Technologies, San Francisco, CA, USA) and rendered in the HMD using the Oculus SDK (Fig. 1d).

Fig. 1.
figure 1

System architecture: (a) EEG system with 8 electrodes over the motor and somatosensory areas, (b) Oculus Rift HMD delivering the VR feedback, (c) 4 EMG sensors over target muscles of the affected arm, (d) the dedicated desktop computer running the VR task and data acquisition.

Full size image

2.3 REINVENT Architecture

The goal of the software architecture design was to be able to tailor REINVENT to each participant’s specific rehabilitation needs and current level of impairment. A secondary goal was to develop a flexible architecture that could integrate new hardware easily to keep up with the rapid pace of technological improvements in VR and wearable sensing devices. To do this, we upgraded the previous REINVENT system [21] with a distributed architecture, making it hardware independent. REINVENT allows for neurofeedback from a variety of interfacing devices that require different degrees of freedom from patients, including those with (1) no active movement through EEG in a direct brain-to-VR interfacing, (2) weak muscle activation through EMG in a muscle-to-VR interfacing, and (3) substantial active movement through hand tracking. The new architecture is built in an open and modular design, with the data acquisition and processing modules independent from the VR task, communicating bidirectionally over a network layer (Fig. 2). In the current study, the EMG and kinematics were only used for logging and not interaction due to the participant’s ability level.

Fig. 2.
figure 2

REINVENT distributed architecture for a closed neurofeedback loop.

Full size image

2.4 REINVENT Training Task

The VR feedback involved the rendering of two virtual hands performing a flexion/extension training task from a first-person perspective (Fig. 3). The manipulation of the virtual hand was triggered through the user’s brain activity. The experimental protocol was designed for a 3-week intervention, resulting in 8 training sessions. Due to the severity of the participant’s motor impairment, and positive response after the first 8 sessions, the patient participated in a second set of 8 sessions, for a total of 16 sessions with a one week break between the sessions. The duration of each session was 1.5 h during which resting state data were acquired at the beginning and end (4 min), with 4 blocks of 20 trials each of the training task in VR (80 trials total) in the middle (Fig. 4). Each task trial included a baseline rest period of 10 s, followed by a motor attempt of hand extension towards a target in VR, which lasted up to 20 s. The patient’s virtual arm moved towards the target if their sensorimotor brain activity during motor attempt increased relative to baseline (e.g., increased desynchronization between 8–24 Hz under motor electrodes C3 or C4). At the end of each block of trials, a total score was calculated as a percentage of successful hand movement in VR.

Fig. 3.
figure 3

VR feedback of the training task: (a) idle state during baseline measurement, (b) target onset with a ball appearing on the table, (c) motor action of wrist extension towards the target.

Full size image
Fig. 4.
figure 4

Training protocol per session. In the beginning of each session, resting state data for 4 min were acquired before and after the 4 training blocks (5 min for each training block). Within each block, 20 trials of motor attempt training were performed.

Full size image

2.5 Behavioral and Clinical Assessment

A set of clinical outcome measures were acquired from the patient at the beginning and the end of the intervention by a trained occupational therapist. The clinical scales included: (a) the Fugl-Meyer Assessment (FMA) for motor impairment, with 66 as the maximum score for upper limb [24] and (b) the Stroke Impact Scale (SIS), a subjective scale of the perceived stroke impact and recovery as reported by the patient with a maximum score of 100 [25].

In addition, a series of questionnaires about the VR experience were collected at three time-points: baseline (session number 1); mid (session number 8); final (session number 16). Those included the (a) Simulator Sickness questionnaire [26], (b) the Embodiment Questionnaire [27] and (c) the Presence Questionnaire [28].

The Simulator Sickness questionnaire (revised by the UQO Cyberpsychology Lab, 2013) included 16 questions on a 0 to 3 Likert scale resulting in two sub-scales: Nausea (9 questions for a maximum of 27 points) and Oculo-Motor (7 questions for a maximum of 21 points) [29]. The Presence Questionnaire was adapted from Witmer and Singer (1998) and asked participants a series of questions related to their sense of presence in VR. Responses were reported on a 1 to 7 Likert scale divided in five sub-domains: Realism, Possibility to Act, Quality of Interface, Possibility to Examine, and Self-Evaluation of Performance. The Embodiment Questionnaire included a series of questions to gauge their sense of embodiment. Responses were reported on a 1 to 10 Likert scale related to either Self Embodiment or Spatial Embodiment sub-domains.

2.6 Data Processing

EEG signals were processed in Matlab with the EEGLAB toolbox [30]. A bandpass filter was applied between 1–50 Hz, following bad channel removal. All EEG channels were re-referenced with an average reference and divided into motor-execution epochs for every trial. Finally, Independent Component Analysis (ICA) was used for removing all major artefacts related with power-line noise, eye blinking, ECG and EMG activity.

For acquiring the Power Spectral Density (PSD), the power spectrum was extracted for the following frequency bands: Alpha (8–12 Hz), Beta (12–30 Hz), Theta (4–7 Hz), and Gamma (25–90 Hz). For the current analysis, and because we were only measuring from sensorimotor areas, an Event-Related Spectral Perturbation (ERSP) analysis was performed. The ERSP is a time-frequency representation of the spectrograms of the post-stimulus EEG, divided by its pre-stimulus baseline, and then averaged across all trials (or epochs) [31]. It serves as a generalization of the Event-related synchronization/desynchronization (ERS/ERD) [32]. In our analysis, we converted all ERSP values into ERS/ERD percentages for the Mu (8–12 Hz) and Beta (12–30 Hz) bands over the C3 and C4 electrode locations in order to capture motor related activation. Finally, for assessing differences in activation from both hemispheres we extracted a hemispheric asymmetry index. Here, we defined hemispheric asymmetry as the relative power values detected at C3 and C4 for both the ERD values and Alpha power. To estimate the asymmetry, the power at the electrode contralateral to the movement side was subtracted from the ipsilateral (e.g., for left hand actions, it would be defined as C4-C3).

2.7 Statistical Analysis

Normality of the distribution of all data was assessed using the Shapiro-Wilk (S-W) normality test. A one-sample t-test was used to determine whether there was a significant difference between the patient’s ERD values versus the mean ERD values of healthy population. Moreover, for comparing the means between two related score-groups of the patient on the same continuous, dependent variable (score in %), a paired-samples t-test was used. Finally, a non-parametric Spearman’s rho test was used for assessing significant correlations between ERD values, EEG resting state and training scores. For all statistical comparisons the significance level was set to 5% (p  <  0.05) and all statistical analysis were completed using IBM SPSS 20 (SPSS Inc., Chicago, IL, USA).

3 Results

3.1 Brain-to-VR Interaction

One of the major goals of the REINVENT architecture was to evoke motor-related activation from stroke participants’ brains that mirrored healthy motor brain activity. Our current results show that when using REINVENT, we are able to evoke distinct patterns of sensorimotor activation during motor attempt over the lesioned area. This was achieved by extracting the level of ERD/ERS over the motor and somatosensory areas through C3 and C4 electrodes during the virtual motor training task (Fig. 5).

Fig. 5.
figure 5

Percentage of the Event-related synchronization (ERS) and desynchronization (ERD) over the lesioned area (electrode C4) between mu and Beta frequency ranges including head maps pre-post the stimulus. The blue trace 500 ms post stimulus at 12–24 Hz is the signature of the motor intention of the user over the right hemisphere as quantified by REINVENT. (Color figure online)

Full size image

Although the motor intention signatures are clearly illustrated on the extracted EEG data, we also compared the patient’s data with healthy population data in order to quantify how closely the patient’s ERD values matched those of healthy individuals. To achieve this, we included data from two well established studies in EEG research. The first study (Study 1) from Pfurtscheller and Aranibar. included data from 10 healthy subjects performing voluntary self-paced movement [33], while the second study (Study 2) from Pfurtscheller et al. included 9 participants performing motor-imagery tasks [34]. In this way we compared the activation during stroke motor attempt to both actual motor execution and motor imagery. For consistency, we included data from the same electrode location (C4) in order to compare it with the lesioned side of the patient.

A one-sample t-test was used to determine whether there was a significant difference between the patient’s ERD values versus the mean ERD values of each of the two studies. Our results show that the mean ERD values of the patient data (M = −10.8, SD = 21.3) were significantly higher than those from Study 1 (M = −70.8, SD = 59.7), t (15) = 10.918, p < 0.05) and Study 2 (M = −86.7, SD = 8.9), t (15) = 13.813, p < 0.05) (Fig. 6). This suggests that although we are able to evoke motor related activation from the lesioned side, it is still far from optimal activation compared with healthy data. This distance between healthy ERD and stroke constitutes an important feature since it can be used to better quantify a stroke ERD goal amplitude during BCI-VR training.

Fig. 6.
figure 6

ERD values of the patient with healthy population values during voluntary movement (Study 1) and motor imagery (Study 2) [33, 34]. * indicates significance of p < 0.05.

Full size image

Regardless of the low ERD from the lesioned side, the VR training task scores revealed that the patient was able to voluntarily control the virtual limb by using brain signals from the lesioned side with up to 95% success (M = 77.2, SD = 10.5) (Table 1). Since the patient had two sets of training (8 sessions each), we divided the score into two sessions. Session 1 (M = 74.2, SD = 5.6) included sessions 1 to 8, and Session 2 (M = 80.1, SD = 5.2) included sessions 9 to 16 (Fig. 7). A paired-samples t-test revealed marginally significant differences between the two sessions (t(7) = −2.2584, p = 0.058).

Table 1. Score table for all trials per session in %.
Full size table
Fig. 7.
figure 7

Scores of Group 1 for sessions 1 and 8, and Group 2 for sessions 9 to 16.

Full size image

Finally, in terms of EEG rhythmic activity during resting state, non-parametric Spearman’s rho tests revealed significant correlations between resting state Alpha and performance in VR training. Specifically, Alpha rhythm during resting state each day was moderately correlated with the training score in VR that day (r = 0.603, p = 0.013). Similarly, ERD in both Mu (r = −0.500, p = 0.03) and Beta (r = −0.541, p = 0.043) during training showed significant negative correlations with resting state Alpha. That is, a higher resting state Alpha correlated with more sensorimotor activity (measured as increased negative ERD amplitude) and subsequently higher training scores. In addition, the ERD hemispheric asymmetry (that is the difference between C4-C3) of the Beta rhythm was significantly negatively correlated with the session number (r = −0.726, p = 0.001). That is, motor-related beta activity increased over the healthy hemisphere over time.

3.2 Behavioral Data

Regarding reported presence in VR, in looking at all sub-domains extracted from the Presence Questionnaire, there was an increasing trend starting from the first session, moving to the mid-session (8th) up to the last session (16th) across four sub-domains: Realism (First: 27, Mid: 30, Last: 33), Possibility to act (First: 16, Mid: 17, Last: 20), Quality of Interface (First: 4, Mid: 5, Last: 6) and Possibility to examine (First: 12, Mid: 6, Last: 18). The Self-evaluation of performance remained stable over time (First: 14, Mid: 14, Last: 14) (Fig. 8). In terms of simulator sickness, the patient did not report any increases in either Nausea (First: Pre = 1, Post = 0, Mid: Pre = 1, Post = 0, Last: Pre = Not collected, Post = 0) or Oculo-motor sickness (First: Pre = 3, Post = 1, Mid: Pre = 3, Post = 3, Last: Pre = Not collected, Post = 1) at any of the timepoints. This suggests that the repeated VR intervention is feasible, and this was consistent over time with no increases in simulator sickness.

Fig. 8.
figure 8

Presence questionnaire subscales

Full size image

Finally, in terms of embodiment, an increasing trend over time was observed concerning the participant feeling that the virtual arm was his own (real) arm (Q3), feeling that the virtual arm was him (Q4), and feeling that he was surrounded by the virtual environment (Q7). Only the question about feeling that the virtual environment seemed like the real world decreased in the last session (Q9), while the rest of the questions maintained relatively stable across all sessions (Fig. 9).

Fig. 9.
figure 9

Embodiment questions. Q1: To what extent was the virtual arm an extension of yourself?; Q2: To what extent did you feel if something happened to the virtual arm it felt like it was happening to you?; Q3: To what extent did you feel that the virtual arm was your own (real) arm?; Q4: To what extent did you feel that the virtual arm was yours?; Q5: How much did the virtual arm’s actions correspond with your commands?; Q6: To what extent did you feel like you were really at the virtual environment?; Q7: To what extent did you feel surrounded by the virtual environment?; Q8: To what extent did you feel like you really visited the virtual environment?; Q9: To what extent did you feel that the virtual environment seemed like the real world?; Q10: To what extent did you feel like you could reach out and touch the objects in the virtual room?

Full size image

3.3 Clinical Outcome

In terms of motor impairment as assessed by the Fugl-Meyer scale, the patient showed a very modest improvement in the total score after the end of the intervention (Pre: 13, Post: 14). Moreover, the patient reported an increase in stroke-related quality of life, particularly in the physical domain, as reported on the Stroke Impact Scale (Pre: 45, Post: 75).

4 Conclusions

First, the current findings illustrate that the new REINVENT architecture can be successfully used by an individual with stroke, increasing the levels of presence over time and the sense of embodiment of the virtual arm. Second, as anticipated, the stroke survivor showed sensorimotor brain activity during motor attempt from the lesioned side which resembled that of motor activity in healthy individuals, although not at the same level. This suggests that REINVENT can activate targeted motor pathways with a virtual representation of the affected arm in VR, despite lack of volitional movement. In addition, the patient was able to voluntarily control the virtual limb corresponding to the affected side with up to 95% accuracy during the VR training task and showed generally improved control over time through the increased score, suggesting that the patient was learning to modulate his own motor brain activity. Correspondingly, to examine differences in activation over time, we conducted a correlation analysis between the lesioned hemisphere and session number, showing decrease of the hemispheric asymmetry index over time. That is, motor-related beta activity increased over the healthy hemisphere. This finding is inline with stroke research showing that action observation is lateralized to the dominant, rather than ipsilesional, hemisphere [35].

Moreover, we found a positive correlation between resting state Alpha with training in VR, which was measured as sensorimotor desynchronization in Mu and Beta bands, suggesting sustained increases in motor-related brain activity. Since it is known that Mu has been associated with somatosensory information and Beta with actual motor processing [36], it is likely that the current relationship with resting state Alpha could be used as a potential predictor of training performance in VR and a potential biomarker for rehabilitation efficacy using EEG-based BCI-VR training paradigms [37, 38].

Finally, the participant reported improvements in functional measures, as captured through the Stroke Impact Scale and Fugl-Meyer following REINVENT training.

Overall, in this study, we present a new, flexible architecture for a brain-computer interface for stroke that provides neurofeedback in HMD-VR and show that it is feasible to use for individuals with chronic severe stroke across 16 sessions. Future studies may examine the use of REINVENT as a personalized training system for post-stroke individuals across varying levels of ability and should explore the impact of REINVENT training on functional activities and across additional settings (e.g., lab, clinic, home).