SSVEP signal classification and recognition based on individual signal mixing template multivariate synchronization index algorithm

https://doi.org/10.1016/j.bspc.2020.102304Get rights and content

Highlights

  • Individual signal mixing template MSI (IST-MSI) algorithm was proposed in this paper.

  • IST-MSI algorithm incorporates individual SSVEP training data into the reference signal

  • IST-MSI algorithm takes the personal harmonic sensitivity into the SSVEP identification process.

  • CCA method is used to reduce the redundant information in individual templates.

  • The proposed R3 reference signal achieves about 5.7% accuracy improvement.

Abstract

With the development of automation technology, Brain Computer Interface (BCI) has been increasingly integrated into people's daily life, among which Steady State Visual Evoked Potential (SSVEP) has attracted much attention due to its high signal-to-noise ratio (SNR) and wide application scenarios. To improve the classification accuracy of SSVEP signals, a novel individual signal mixing template multivariate synchronization index algorithm (IST-MSI) was proposed in this paper, which incorporated individual training template and individual harmonic sensitivity coefficient into the standard MSI algorithm. Specifically, the proposed method first enlarged the frequency-domain power spectrum of the fundamental frequency and its harmonics to reduce the redundant information in the individual training template. The synchronization index values at non-target frequency identified by MSI algorithm are significantly reduced through unequal ratio scaling of harmonic sensitivity coefficient, thereby improving the SSVEP recognition. The experimental results showed that under the signal length of 1.2 s, the average classification accuracy of IST-MSI algorithm reached 84.3 % in six target frequencies, which was 5.8 % higher than that of standard MSI algorithm. This study confirmed the efficacy of the proposed IST-MSI algorithm for SSVEP recognition, demonstrating its promise in developing an improved BCI system.

Introduction

Brain Computer Interface (BCI) is an advanced technology for human-computer interaction [1], which aims to construct a direct information channel between human brain and external hardware devices. The user's electroencephalography (EEG) is decoded and converted into the instructions of the computer or control system to realize the brain's direct control for external hardware devices [2].BCI technology provides a new communication and control method for patients with physical inconvenience to realize wheelchair manipulation [3], prosthetic limb control [4], stimulation of nerve recovery [5] and other functions by using brain ideas. In addition, this technology has also been applied in virtual reality games [6], military communication control [7], operator fatigue state detection [8,9] and other medical fields.

BCI systems can induce different EEG signals according to different experimental paradigms. The most common experimental paradigms include motor imagination(MI), steady-state visual evoked potential (SSVEP), event-related potential(ERP) [[10], [11], [12], [13], [14]] and hybrid BCI stimulation potential [[15], [16], [17], [18]]. Among them, SSVEP signal has a higher signal-to-noise ratio (SNR) and a shorter response time [19]. And the SSVEP-based BCI showed its unique characteristics of high information transformation rate and low training demand for subjects [20]. Therefore, SSVEP signal has received extensive attention in the related researches of BCI systems [21] in recent studies. The EEG signal collected by electrodes on the scalp surface has the advantages of high temporal resolution, non-invasive and low cost, becoming the most widely used acquisition method in BCI systems [22].

The traditional SSVEP identification method used Fast Fourier Transform (FFT) to perform power spectral density analysis (PSDA) of EEG signals. Target frequency was identified by detecting the peak frequency [23,24]. Or PSDA method was used as a feature extraction method, followed by supervised machine learning algorithm training [25,26] to identify target frequencies. In recent years, multichannel-based correlation analysis methods have been widely used in the recognition of SSVEP signals [27,28]. Among them, canonical correlation analysis (CCA) algorithm and multivariable synchronization index (MSI) algorithm are the main frequency identification methods adopted by SSVEP-based BCI systems [[29], [30], [31], [32], [33]]. The CCA algorithm was proposed by Gao et al. [34] of Tsinghua University, which showed significant performance over the traditional algorithms in the recognition process of SSVEP signals; the MSI algorithm was proposed by Zhang et al. [35] for SSVEP recognition and achieved 86 % recognition accuracy in an online system.

However, both methods are limited by the fact that the sine-cosine reference signals cannot fully reflect the characteristics of the subjects and the characteristics of the harmonic components of the EEG signals [36,37]. In the study of SSVEP brain-computer interface, Wang et al. [36] incorporated training EEG signal into reference signal and proposed a typical correlation algorithm based on individual template data (ITCCA), which significantly increased the correlation coefficient difference between target frequency and non-target frequency, and improved the recognition accuracy of CCA algorithm by 6.8 % in 10 subjects. Chen et al. [37] proposed a typical correlation algorithm based on the filter bank(Filter Bank CCA, FBCCA) which effectively utilized SSVEP fundamental frequency components and harmonic components and improved the recognition accuracy of CCA algorithm by 12.67 % in 10 subjects [38].

In the previous research works of this paper, it was found that in the process of SSVEP signal recognition, the results of single sine-cosine harmonic calculation can get better recognition than MSI algorithm through different coefficient combinations. The combined coefficients are called harmonic sensitivity coefficients in this study. In order to improve the robustness of the algorithm, the calculation results of coefficient combinations are mixed with those of standard MSI algorithm in different proportions. But simply using a single sine-cosine wave cannot get a stable improvement effect. Inspired by ITCCA in [36], the individual template of the subject is added to the single sine-cosine wave reference signal. In order to reduce the redundant information in individual templates, the frequency domain enhancement mapping of individual templates is carried out by CCA method. And the Individual signal mixing template MSI (IST-MSI) algorithm was proposed. On the basis of standard MSI algorithm, the individual template enhanced by frequency domain energy is added into the single corresponding sine-cosine harmonic reference signal. After the unequal ratio scaling of harmonic sensitivity coefficient, it is mixed and added with the calculation result of the standard MSI algorithm to make the final calculation results. Finally, the synchronization index at non-target frequencies identified by standard MSI algorithms are significantly reduced, resulting in a sharp peak in target frequencies, which improves the recognition accuracy of MSI algorithm. The IST-MSI algorithm is applied to the SSVEP classification of 6 target frequency stimuli in 6-bit subjects, and compared with the standard MSI algorithm. The reduction of training set and the optimal structure of individual template are analyzed and discussed in the framework of IST-MSI algorithm.

The following chapters are arranged as follows: Section 2 introduced the signal acquisition and data preprocessing, Section 3 introduced the IST-MSI algorithm flow proposed in this paper, Section 4 discussed and analyzed the results of IST-MSI algorithm, and Section 5 discussed and summarized the results.

Section snippets

Data acquisition experiment

The offline and online SSVEP experiment was designed by using a 6-bit BCI number-speller based on SSVEP. Six healthy subjects (4 males and 2 females, aged 22–26 years, mean age: 24 years) with normal or corrected vision participated in the experiments. Before the experiment, each subject was informed of the content and process of the experiment and signed an informed consent.

During the experiment, EEG signals were recorded by a 16-channel wireless physiological signal acquisition system

Individual signal mixing template MSI algorithm

The flowchart of IST-MSI algorithm is shown in Fig. 2. Through the offline experiment of Phase I, the real EEG response (raw individual signal template) of the subjects to SSVEP stimulation can be obtained. Then the CCA method was used to get the individual signal template. The training dataset of IST-MSI algorithm was provided by the offline experiment phase II. Through the training dataset, the optimized individual parameters (Wi, SA, SB) of IST-MSI algorithm could be obtained in the

Bandpass filtering range of IST-MSI algorithm

In order to determine the filtering range of the IST-MSI algorithm, this study analyzed the amplitude spectrum and SNR of the SSVEP signal. The amplitude spectrum of the SSVEP signals at each stimulation frequency was calculated by averaging the corresponding channels from the offline experimental data of Subject Phase II. Fig. 4(a) shows the amplitude spectrum of SSVEP at Oz channel with a stimulation frequency of 15 Hz. The highest amplitude (0.816 u V) was found at the 15 Hz fundamental

Minimizing the training sets

For a BCI system based on SSVEP signals, offline experiment time of 30 blocks is too much. Therefore, this paper studied the application effect of different numbers of offline training data blocks (TDB) in the simulated online experiment. The results are shown in Fig. 10. When the number of TDB is small, the application effect is unstable, and even the recognition accuracy of subjects 1 and 3 is lower than that of MSI algorithm. However, as the amount of TDB increases, the effect of online

CRediT authorship contribution statement

Ke Qin: Formal analysis, Investigation, Methodology, Visualization, Writing - review & editing. Raofen Wang: Conceptualization, Funding acquisition, Methodology, Supervision, Writing - review & editing.

Declaration of Competing Interest

The authors report no declarations of interest.

Acknowledgments

This work was supported in part by the Natural Science Foundation of China (61803255) and the Natural Science Foundation of Shanghai (18ZR1416700).

References (43)

  • L. Angrisani et al.

    A wearable brain-computer interface instrument for augmented reality-based inspection in industry 4.0

    IEEE Trans. Instrum. Meas.

    (2020)
  • L.W. Ko et al.

    Development of a smart helmet for strategical BCI applications

    Sensors

    (2019)
  • R.F. Wang et al.

    An adaptive neural network approach for operator functional state prediction using psychophysiological data

    Integr. Comput. Eng.

    (2016)
  • B.Z. Allison et al.

    Brain-computer interface systems: progress and prospects

    Expert Rev. Med. Devices

    (2007)
  • M. Li et al.

    An event-related potential-based adaptive model for telepresence control of humanoid robot motion in an environment cluttered with obstacles

    IEEE Trans. Ind. Electron.

    (2017)
  • M.H. Choi et al.

    Perceptual threshold level for the tactile stimulation and response features of ERD/ERS-Based specific indices upon changes in high-frequency vibrations

    Front. Hum. Neurosci.

    (2017)
  • D. Zhang et al.

    Exploring the temporal dynamics of sustained and transient spatial attention using steady-state visual evoked potentials

    Exp. Brain Res.

    (2017)
  • Yan Zheng et al.

    SSVEP brain-computer interface based on dual-frequency stimulation of left and right fields of vision

    J. Tsinghua Univ.: Natl. Sci. Ed.

    (2009)
  • A. Katyal et al.

    A novel hybrid paradigm based on steady state visually evoked potential & P300 to enhance information transfer rate

    Biomed. Signal Process. Control

    (2020)
  • J.D.R. Millan et al.

    Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges

    Front. Neurosci.

    (2010)
  • Y. Li et al.

    A hybrid BCI system combining P300 and SSVEP and its application to wheelchair control

    IEEE Trans. Biomed. Eng.

    (2013)
  • Cited by (6)

    • Spatially-coded SSVEP BCI without pre-training based on FBCCA

      2023, Biomedical Signal Processing and Control
    • Spatio-Spectral CCA (SS-CCA): A novel approach for frequency recognition in SSVEP-based BCI

      2022, Journal of Neuroscience Methods
      Citation Excerpt :

      They considered phase shifting reference signals and accommodated time lag of subjective and device response in their method. Other studies (Chevallier et al., 2021; Li et al., 2020; Ojha and Mukul, 2021; Wang et al., 2020; Wu et al., 2020; Hsu et al., 2020; Ojha and Mukul, 2021; Qin and Wang, 2021) have addressed the recognition of the SSVEP signal, in which they are not based on CCA algorithm. The CCA algorithm contain three main limitations: First, the harmonics' information are not effectively incorporated.

    • Decoding SSVEP patterns from EEG via multivariate variational mode decomposition-informed canonical correlation analysis

      2022, Biomedical Signal Processing and Control
      Citation Excerpt :

      Its advantage over the black and white encoding flashing method is that it is not limited by the refresh rate, and the stimulation frequencies as low as half of the screen refresh rate can be achieved. Because SSVEP can induce higher quality EEG in the medium and low-frequency ranges [41], this article chooses [8 9 10 11 12 13 14 15] Hz as the visual stimulus frequency. Fig. 2 depicts the visual stimulation interface.

    • P300 Evoked Potential Detection Method Using the MSI Algorithm in the Problem of Human-Computer Interaction

      2022, 2022 24th International Conference on Digital Signal Processing and its Applications, DSPA 2022
    1

    Address: Room 7504, School of Electrical and Electronic Engineering, 333 Rd. Longteng, Shanghai 201620, China.

    View full text