1 Introduction

A Brain Computer Interface (BCI) is an alternative, augmentative communication channel [1] which aims at providing the user with an interaction path based on the sole interpretation of her/his brain activity. In this sense, BCI can be viewed as a particular class of HCI devices, with different interaction requirements with respect to “conventional” HCI approaches. For the ease of discussion, but without loss of generality, let us focus on a possible use of BCI as a HCI, namely in Ambient Assisted Living (AAL [2]) system control [3, 4].

In order to be accepted and effective, such BCI-enabled HCI channel needs to be perceived as natural as possible; in other terms, BCI operation should be continuous and self-paced [5, 11], i.e. the device must be able to discern user’s intentional control periods from nonintentional ones, providing reliable command decoding in the former case (since the user’s interactions are quite sporadic, a major concern is being able to minimize false positive classifications). Furthermore, a “Plug&Play” approach is highly desirable in such application scenario, since complex and time-consuming “ad personam” calibration procedures, could be perceived as an excessive burden. In addition, the device’s behavior should be uniform across different users, and fine performance tuning should be just limited to a few high-level parameters.

Previously, the need of false positive minimization was mentioned as capital for effective BCI-enabled control. This can be accomplished by exploiting multiple and different input channels, such as the (possibly minimal) residual motor ability: information on muscular activation could be picked up and monitored by means of ElectrMyoGraphy (EMG). In this case EMG signals could be integrated into a hybrid BCI (hBCI) framework and be used to switch on and off the SSVEP visual stimulation unit when not needed. This can improve user’s comfort (less eye fatigue on the long run), as well as further reduce the BCI false positives (long inactive periods with SSVEP stimuli are excluded).

In the following the implementation of the BCI algorithms internals are discussed, focused on enabling effective AAL system control.

2 BCI Operating Paradigm and Infrastructure

Various paradigms are commonly exploited in BCI literature, including:

  • Slow Cortical Potentials (SCP), i.e., potential shifts in the EEG waves voluntarily induced by user, who can learn to control them through biofeedback-like approaches [6].

  • Event Related Desynchronization (ERD) and ER Synchronization (ERS) [7]: this paradigm exploits the brain response arising when preparing (or just imagining) to start a movement. In such conditions, neurons tend to de-synchronize from their idling state, to be allocated to motor processing, this reflecting in a decrease of spectral energy in the μ and β bands (8-12 Hz and around 20 Hz, respectively). After ERD, a pattern consisting in increase in the energy band after the completion of the motor task can also be observed (ERS).

  • P300 [8, 9]: when a rare target stimulus is presented to the user during a sequence of repetitive, non-target stimuli, a characteristic pattern can be observed in the EEG signals, approximately after 300 ms from the target stimulus appearance.

  • Steady State Visual Evoked Potential (SSVEP) [10, 13]: it is a periodic brain response elicited by a visual stimulus, flickering at a constant frequency; a peak in the brain power spectrum, synchronous with such frequency, can be produced just by looking at the visual stimulus.

Among the presented paradigms, SSVEP was chosen for core BCI operation. SSVEP has recently received much attention, especially in communication or control applications where fast, reliable interaction is needed and multiple simultaneous choices are presented to the user. SSVEP are regarded as robust features for BCI, given their inherently higher SNR (Signal to Noise Ratio) with respect to other paradigms (e.g., with respect to motor imagery, as discussed in [10]). Moreover, since it exploits involuntary response, SSVEP do not require, in principle, any specific user skill and thus involve no user training. In addition, the steady-state, repetitive nature of such potentials makes it possible to design calibration-free classification methods.

From the hardware point of view, our BCI solution is built on top of a custom, dedicated EEG module: it features 16 input channels in a small, 100 × 130 mm form factor, and can be powered by means of 4 AA alkaline batteries. Production costs are also contained, with respect to current, commercial EEG devices: in medium scale, device manufacturing amounts to, approximately, 300 €. The module communicates via a full-speed USB 2.0 link (12 Mbps), and can be controlled and set-up directly by a host computer.

Finally, the module was validated and compared against a reference, commercial EEG device (as discussed in [4, 13]), showing good performance and proving its suitability for EEG studies.

3 BCI Signal Processing

3.1 Classifying SSVEP in a Self-paced Scenario

Many algorithms exist in literature for SSVEP classification; among the most popular ones are: MEC [14] (Minimum Energy Combination), AMCC [15] (Average Maximum Contrast Combination) and CCA [16] (Canonical Correlation Analysis). A review of such methods goes beyond the scope of this article; the interested reader could refer, for example, to [12, 17]. Our implementation choice stems from a CCA-based approach, and proposes an extension in order to improve the system immunity against false positives.

CCA is a statistical method, generally used for finding the correlations between two sets of multi-dimensional variables. It seeks a pair of linear combinations (canonical variables, characterized by weight vectors w x , w y ) for the two sets, such that the correlation between the two linear combinations x L  = w T x X and y L  = w T y Y is maximized:

$$ \mathop {\hbox{max} }\limits_{{w_{x} ,w_{y} }} \rho = \frac{{E\left[ {x_{L} y_{L}^{T} } \right]}}{{\sqrt {E\left[ {x_{L} x_{L}^{T} } \right]E\left[ {y_{L} y_{L}^{T} } \right]} }} = \frac{{w_{x} XY^{T} w_{y} }}{{\sqrt {w_{x}^{T} XX^{T} w_{x} w_{y}^{T} YY^{T} w_{y} } }}\;\;. $$
(1)

Here X, Y are the input and the SSVEP reference matrix, respectively. Y is composed by N h (sin, cos) couples representing a steady state sinusoidal response, with N h representing the number of considered harmonics:

$$ X = \left[ {\begin{array}{*{20}c} {\sin \;2\pi ft_{1} \;\;\cos \;2\pi ft_{1} \;} & \cdots & {\sin \;2\pi N_{h} ft_{1} \;\;\cos \;2\pi N_{h} ft_{1} } \\ \vdots & \cdots & \vdots \\ {\sin \;2\pi ft_{Nt} \;\;\cos \;2\pi ft_{Nt} } & \cdots & {\sin \;2\pi N_{h} ft_{Nt} \;\;\cos \;2\pi N_{h} ft_{Nt} } \\ \end{array} } \right]\;, $$
(2)

Given the presence of a SSVEP in the observed EEG window, a classifier could pick the target frequency f class ∈{flickering stimuli} which yields the largest correlation coefficient ρ i :

$$ f_{class} = \arg \hbox{max} \rho_{i} \;. $$
(3)

However, in practical scenarios, the assumption of a SSVEP presence within the observed window does not hold true for any window: in other words, such simple classifier is not suitable for self-paced operation, where a “no SSVEP” class needs to be contemplated. A common solution to this problem is to smooth the classifier output, validating the classifier output only if n previous samples agree with the current one. In the following we adopt a different methodology, identifying a feature which could be used to assess the level of confidence in the prediction and that can be exploited to discern between user control and rest periods.

First, let us consider an offline 4-class SSVEP problem (i.e., each epoch contains a SSVEP in the {16, 18, 20, 22} Hz set) and introduce the notion of a confidence indicator for improving baseline CCA accuracy. We define such indicator as the absolute difference between the largest correlation coefficient and the second largest one as such indicator, from here on referred to as parameter d.

$$ d = \mathop {\hbox{max} }\limits_{f \in Fstim} \;\left| {\rho_{f} } \right| - \mathop {\hbox{max} }\limits_{{f \in Fstim\backslash \{ fmax\} }} \;\left| {\rho_{f} } \right|\;, $$
(4)

where ρ f is the correlation coefficient yielded by CCA as described in Eq. (1), Fstim is the set of possible stimuli frequencies and fmax the frequency associated to the largest ρ f .

Figure 1 allows to assess the usefulness of the introduced confidence indicator by reporting the distribution of correctly and wrongly classified epochs as a function of parameter d. Ideal behavior should associate all errors (black bars) to low values of d, with correct classification (light grey bars) associated to largest values instead. An optimal threshold, d*, could then be easily determined and a rejection criterion set up to discard all epochs associated with low d values (d ≤ d*). However, since actual data show overlaps between the correct and wrong classification distributions, a tradeoff between prediction accuracy and data yield (i.e. the fraction of non-neutral epochs) is needed.

Fig. 1.
figure 1

Distribution of classifiers’ hits and errors (light grey and black, respectively; each is normalized to the sample size) as a function of the parameter d. The distributions are also plotted for different EEG window lengths.

Full size image

Figure 2 better explains such a tradeoff. In order to derive it, a d* is fixed and classification performed using such value as decision threshold for the aforementioned rejection mechanism. Epochs not meeting the rejection criterion (d ≤ d*) are discarded, and accuracy is computed over the remaining ones; such computed accuracy, together with the corresponding fraction of rejected (neutralized) epochs, identifies a point in a 2D plane, and this procedure is repeated for different values of d*. Figure 2 graphically summarizes this tradeoff with a scatter plot: in order to appreciate the improvement introduced by neutralization strategy, the performance achieved with no neutralization is also reported (grey solid line). Consistent improvement is achieved over the reference case, even at lower neutralization rates, i.e., without implying too relevant data loss. It is important to notice that the proposed confidence indicator method was evaluated over the whole subject population (four in this offline experiment), instead of relying on a per-user basis analysis: this is in line with our view of subject-independent approach.

Fig. 2.
figure 2

Accuracy as a function of neutralization rate, at different values of threshold d*. The solid grey line represents the original, “raw” accuracy level, without neutralization (i.e., d* = 0). The graphs are plotted for different EEG window lengths.

Full size image

The quality metric introduced above can also be exploited to effectively adapt the length of the observed EEG window. In particular, the SSVEP classification algorithm could start with a very short observation window, in order to maximize the system’s responsiveness; in case the gathered waveforms do not support a reliable classification (the aforementioned rejection criterion is not passed), the observed EEG window length is increased, looking for further evidence. This process continues until either a classification is attempted or a maximum pre-determined window length is reached. Such an adaptive mechanism gives more flexibility with respect to a fixed-window approach, where responsiveness and accuracy are more tightly coupled.

3.2 Real-Time, Self-paced Operation: Results

A self-paced, online BCI, exploiting a 4-class SSVEP paradigm was implemented and tested. Experimental setup is as follows: 4 visual stimuli (LED, organized in a rectangular pattern over a box) are shown simultaneously, with blinking frequencies equal to {16, 18, 20, 22} Hz; the subject is seated approximately 1 m away from the visual stimuli. Only 6 passive Ag/AgCl electrodes are used to acquire signals form scalp locations Pz, P3, P4, POz, O1, O2. The protocol associates a particular home automation task (namely on/off switching of a light and opening of a motorized shutter) to each stimulus, and the user is asked to perform several control actions, at his own pace and will. Moreover, in order to assess the immunity to false positive events, long idle periods are introduced on purpose, during which the subject does not make any intentional choice and is allowed to talk and, partially, move. A total of 10 healthy volunteers (age 24-61, 4 females) participated in this study, none of them with any prior BCI-control experience, nor was involved in any calibration/training phase.

The self-paced BCI has an update rate of 5 Hz, i.e. a classification is attempted every 200 ms. The optimal observed EEG window length is determined according to the adaptive criterion presented above. In this case, the minimum window length is set to 2 s, and it is allowed to grow up to 4 s in steps of 500 ms. In addition, in order to further improve immunity to false positives, a post-smoother is optionally added, which averages the last 5 classification outputs for each class (the 4 targets plus the neutral state): if the average for a class exceeds a given threshold, the classification is validated, otherwise a null output is assumed (i.e. no SSVEP detected).

Table 1 reports the online experiment results (mean and standard deviation), in terms of true positive, false negative and false positive rates. A very good performance is achieved, both in terms of true positive and false positive rates. In particular, false positives are kept to a very small amount (≈ 0.16 min−1 on average, i.e. approximately a false positive every 6’15’’), improving over literature data [18, 19]. It is important to remark that such results were achieved without any subject-specific parameter tuning: in other words, all user share the exact same setup, this being in line with our subject-independent BCI approach.

Table 1. Online performance of the self-paced, 4-class experiment (10 subjects)
Full size table

Finally, the entire system was put to test in a relatively harsher environment, in the context of the Handimatica 2014 exhibition. Here, high background luminosity, noise, electromagnetic interference, are not controlled as lab environments, and may potentially hinder the effectiveness of the solution. Furthermore, the subject was relatively free to move and speech, in order to interact with people. Overall, 6 live demos were performed, for the approximate duration of 30 min each. Although non-conclusive from a statistical point of view, promising results were achieved: the subject was able to successfully operate the BCI (controlling the on/off switching of a light and the opening of a mechanical shutter), and the false positive rate was as low as 0.14 min−1. This encourages the transition of such technology also outside of lab environments.

4 An Auxiliary EMG-Based Input Channel

In the introduction, it was stated that a hybrid approach is sought for, looking to exploit different input channels other than pure EEG. A solution could be to sense the (possibly weak) residual motor ability via EMG. In this case, for instance, the EMG channel could be used as a binary switch for enabling the EEG signal analysis; when not enables, the visual stimuli unit could be turned off in order to improve user’s comfort (long exposure to light flashing periods induces eye fatigue). Moreover, turning LED stimuli off when not needed could mean improving BCI robustness against false positives, since EEG peaks at the specific target frequencies are less likely to occur. Nonetheless, it is still important for the EEG-based BCI section to be able to make such a distinction on its own, since false activations could be triggered by the EMG part.

Two experiments are performed, exploiting different muscular activations: jaw clench and eye movements (vertical or horizontal). In the former experiment, EMG is acquired from the masseter muscle via a single, differential channel, whereas in the latter from the frontalis and orbicularis muscles. Sampling rate, in both cases, is set to 1000 SPS to pick-up relevant signal features.

After collection, the signal undergoes basic pre-processing. For the jaw clench case, a basic band-pass filtering is performed ([100-350] Hz bandwidth, optimized to extract the more significant signal features), followed by a squaring. Before extracting the necessary features, the observed signal window is inspected for potentially interesting peaks based on a percentile criterion. Based on this, two features are extracted from the isolated signals: the integral and the mean. Those features are then passed to a linear kernel SVM, which takes care of the classification between epoch with or without muscular activation. The ocular-based experiment, follows a similar path, with a basic low-pass (f cut  = 200 Hz) and notch (50 Hz) filtering, followed by a Savitzky–Golay smoothing stage (in order to preserve the signal shape). Temporal features are then extracted, based on mean and standard deviation and fed to a SVM classifier.

The training phase consisted of a series of 100 activations performed by a subject. Online, real-time performance was then assessed on two subjects, and no further subject-specific training was performed in both cases. In the first experiment, a real-time test session consisted of 50 attempts to achieve control by a slight jaw clench, performed in a self-paced fashion, with at least 6 s between consecutive activations. In this scenario, a false positive event represents a detected activation while the subject was not trying to achieve control, whereas a false negative is a missed activation attempt. In the second experiment, a subject follows the same protocol, but attempts to achieve control via marked horizontal/vertical eye movements.

Results are still preliminary but encouraging: in the jaw clench experiment, one subject (the one which performed the training phase) was able to achieve perfect control over the whole real-time test session; the other subject did achieve a very good performance too, with just one false positive (≈ 0.067 min−1) and two false negatives (6 %). In the second experiment, false positives are kept within 0.1 min−1, whereas false negatives were, on average less than 10 %. These findings are promising and encourage in moving towards a hybrid BCI architecture, fusing EMG and EEG information to achieve a more robust device. It is expected that the fusion of multiple input sources will further improve the performance in terms of false positives.

5 Conclusions

In this paper, a complete implementation of a SSVEP-based BCI was presented, and a proof of concept for a possible extension exploiting EMG as auxiliary input signal was discussed. This aims at a future hybrid BCI implementation, with possible positive impacts on system performance indicators such a s accuracy or false positives immunity.

First, the EEG-based BCI section was discussed, based on the SSVEP paradigm. A methodology for achieving online, self-paced operation was presented in detail, and the notion of a confidence indicator introduced to improve the BCI performance, both in terms of accuracy and false positives immunity. It is worth remarking that, given the specific application target (namely, BCI-enabled AAL system control), false positive immunity and robustness are primary concerns with respect to, for example, system responsiveness (data throughput). In fact, user’s interactions are limited to a very small amount and sparse in time. Also, for this reason, undergoing long or periodical system calibration phases could be perceived as an excessive burden by the user (this spoiling acceptance and usability chances). Therefore, a calibration and training-free approach was pursued. Subject-independent operation was demonstrated, at the same time achieving remarkably good performance. The results achieved are very good and improve over literature in terms of false positives rejection (0.16 min−1 on average). Moreover, the entire setup was also replicated outside lab-controlled conditions, with very promising results, encouraging the adoption of such technology in more realistic contexts.

Also, a possible hybrid BCI architecture was discussed, exploiting EMG as an auxiliary input channel. A proof of concept of EMG as a binary switch was presented with two experiments, aiming at detecting jaw clenches or ocular movements. The use of EMG as a binary switch, turning off the EEG-based BCI section when not needed, can have two major implications: the first is an improved user’s comfort (less exposure to flashing visual stimuli, which could otherwise lead to eye fatigue), the second is a better false positives rejection (EEG peaks at the specific target frequencies are less likely to occur with the stimuli unit turned off). Preliminary results show the feasibility of such an approach, encouraging the development of a hybrid BCI architecture.