Skip to main content
SpringerLink
Log in

Effect of 3D paradigm synchronous motion for SSVEP-based hybrid BCI-VR system

  • Original Article
  • Published:
Medical & Biological Engineering & Computing Aims and scope Submit manuscript

Abstract

A brain-computer interface (BCI) system and virtual reality (VR) are integrated as a more interactive hybrid system (BCI-VR) that allows the user to manipulate the car. A virtual scene in the VR system that is the same as the physical environment is built, and the object’s movement can be observed in the VR scene. The four-class three-dimensional (3D) paradigm is designed and moves synchronously in virtual reality. The dynamic paradigm may affect their attention according to the experimenters’ feedback. Fifteen subjects in our experiment steered the car according to a specified motion trajectory. According to our online experimental result, different motion trajectories of the paradigm have various effects on the system’s performance, and training can mitigate this adverse effect. Moreover, the hybrid system using frequencies between 5 and 10 Hz indicates better performance than those using lower or higher stimulation frequencies. The experiment results show a maximum average accuracy of 0.956 and a maximum information transfer rate (ITR) of 41.033 bits/min. It suggests that a hybrid system provides a high-performance way of brain-computer interaction. This research could encourage more interesting applications involving BCI and VR technologies.

Graphical Abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Guger C, Schlogl A, Neuper C, Walterspacher D, Strein T, Pfurtscheller G (2001) Rapid prototyping of an EEG-based brain-computer interface (BCI). IEEE Trans Neural Syst Rehabil Eng 9(1):49–58. https://doi.org/10.1109/7333.918276

    Article  CAS  PubMed  Google Scholar 

  2. Wolpaw JR et al (2000) Brain-computer interface technology: a review of the first international meeting. IEEE Trans Rehabil Eng 8(2):164–173. https://doi.org/10.1109/tre.2000.847807

    Article  CAS  PubMed  Google Scholar 

  3. Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM (2002) Brain-computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791. https://doi.org/10.1016/s1388-2457(02)00057-3

    Article  PubMed  Google Scholar 

  4. Chen X, Wang Y, Nakanishi M, Gao X, Jung TP, Gao S (2015) High-speed spelling with a noninvasive brain-computer interface. Proc Natl Acad Sci U S A 112(44):E6058–E6067. https://doi.org/10.1073/pnas.1508080112

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Wang K et al (2017) A brain-computer interface driven by imagining different force loads on a single hand: an online feasibility study. J Neuroeng Rehabil 14(1):93. https://doi.org/10.1186/s12984-017-0307-1

    Article  PubMed  PubMed Central  Google Scholar 

  6. Wu Y, Li M, Wang J (2016) Toward a hybrid brain-computer interface based on repetitive visual stimuli with missing events. J Neuroeng Rehabil 13(1):66. https://doi.org/10.1186/s12984-016-0179-9

    Article  PubMed  PubMed Central  Google Scholar 

  7. Xu L, Xu M, Jung TP, Ming D (2021) Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface. Cogn Neurodyn 15(4):569–584. https://doi.org/10.1007/s11571-021-09676-z

    Article  PubMed  PubMed Central  Google Scholar 

  8. Mussa-Ivaldi S (2000) Real brains for real robots. Nature 408(6810):305–306. https://doi.org/10.1038/35042670

    Article  CAS  PubMed  Google Scholar 

  9. Horki P, Solis-Escalante T, Neuper C, Muller-Putz G (2011) Combined motor imagery and SSVEP based BCI control of a 2 DoF artificial upper limb. Med Biol Eng Comput 49(5):567–577. https://doi.org/10.1007/s11517-011-0750-2

    Article  PubMed  Google Scholar 

  10. Dong E, Zhang H, Zhu L, Du S, Tong J (2022) A multi-modal brain-computer interface based on threshold discrimination and its application in wheelchair control. Cogn Neurodyn 16(5):1123–1133. https://doi.org/10.1007/s11571-021-09779-7

    Article  PubMed  Google Scholar 

  11. Ron-Angevin R, Velasco-Alvarez F, Fernandez-Rodriguez A, Diaz-Estrella A, Blanca-Mena MJ, Vizcaino-Martin FJ (2017) Brain-computer interface application: auditory serial interface to control a two-class motor-imagery-based wheelchair. J Neuroeng Rehabil 14(1):49. https://doi.org/10.1186/s12984-017-0261-y

    Article  PubMed  PubMed Central  Google Scholar 

  12. Lo CC, Tsai SH, Lin BS (2017) Novel non-contact control system of electric bed for medical healthcare. Med Biol Eng Comput 55(3):517–526. https://doi.org/10.1007/s11517-016-1533-6

    Article  PubMed  Google Scholar 

  13. Vlahovic S, Suznjevic M, Skorin-Kapov L (2022) “A survey of challenges and methods for quality of experience assessment of interactive VR applications, ” (in English). J Multimodal User Interfaces 16(3):257–291. https://doi.org/10.1007/s12193-022-00388-0

    Article  Google Scholar 

  14. Liarokapis F, von Mammen S, Vourvopoulos A (2021) “Advanced multimodal interaction techniques and user interfaces for serious games and virtual environments, ” (in English). J Multimodal User Interfaces 15(3):255–256. https://doi.org/10.1007/s12193-021-00380-0

    Article  Google Scholar 

  15. Cattan GH, Andreev A, Mendoza C, Congedo M (2021) A comparison of mobile VR display running on an ordinary smartphone with standard pc display for P300-BCI stimulus presentation. IEEE Transactions on Games 13(1):68–77. https://doi.org/10.1109/tg.2019.2957963

    Article  Google Scholar 

  16. Shin JH, Ryu H, Jang SH (2014) A task-specific interactive game-based virtual reality rehabilitation system for patients with stroke: a usability test and two clinical experiments. J Neuroeng Rehabil 11:32. https://doi.org/10.1186/1743-0003-11-32

    Article  PubMed  PubMed Central  Google Scholar 

  17. Bonkon K, Hwan-Gon L, Yunjun N, Seungjin C (2015) Immersive BCI with SSVEP in VR head-mounted display. Annu Int Conf IEEE Eng Med Biol Soc 2015:1103–1106. https://doi.org/10.1109/EMBC.2015.7318558

    Article  Google Scholar 

  18. Stawicki P et al (2018) SSVEP-based BCI in virtual reality - control of a vacuum cleaner robot, (in English), 2018 Ieee Int Conf Syst, Man, Cybern (Smc) 534–537 https://doi.org/10.1109/Smc.2018.00749.

  19. Abbasi-Asl R, Keshavarzi M, Chan DY (2019) Brain-computer interface in virtual reality, ” (in English). 2019 9th Int Ieee/Embs Confer Neural Eng (Ner) 1220–1224 https://doi.org/10.1109/NER.2019.8717158

  20. Ke Y, Liu P, An X, Song X, Ming D (2020) An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng 17(1):016066. https://doi.org/10.1088/1741-2552/ab4dc6

    Article  PubMed  Google Scholar 

  21. Na R et al. (2021) An embedded lightweight SSVEP-BCI electric wheelchair with hybrid stimulator, (in English). Digit Sig Process 116:103101. ARTN 10310110.1016/j.dsp.2021.103101.

  22. Cattan G, Andreev A, Visinoni E (2020) Recommendations for integrating a P300-based brain–computer interface in virtual reality environments for gaming: an update. Computers 9(4):92. https://doi.org/10.3390/computers9040092

    Article  Google Scholar 

  23. P. Gergondet, D. Petit, A. Kheddar (2012) Steering a robot with a brain-computer interface: impact of video feedback on BCI performance. RO-MAN, 2012 Int Symp Robot Human Interact Commun 271–276 https://doi.org/10.1109/ROMAN.2012.6343765.

  24. Bell CJ, Shenoy P, Chalodhorn R, Rao RP (2008) Control of a humanoid robot by a noninvasive brain-computer interface in humans. J Neural Eng 5(2):214–220. https://doi.org/10.1088/1741-2560/5/2/012

    Article  PubMed  Google Scholar 

  25. Stawicki P, Gembler F, Volosyak I (2016) Driving a semiautonomous mobile robotic car controlled by an SSVEP-based BCI. Comput Intell Neurosci 2016:4909685. https://doi.org/10.1155/2016/4909685

    Article  PubMed  PubMed Central  Google Scholar 

  26. Gergondet P, Druon S, Kheddar A, Hintermüller C, Guger C, Slater M (2011) Using brain-computer interface to steer a humanoid robot. 2011 IEEE Int Conf Robot Biomimetics 192–197 https://doi.org/10.1109/ROBIO.2011.6181284

  27. Nakanishi M, Wang Y, Wang YT, Mitsukura Y, Jung TP (2014) A high-speed brain speller using steady-state visual evoked potentials. Int J Neural Syst 24(6):1450019. https://doi.org/10.1142/S0129065714500191

    Article  PubMed  Google Scholar 

  28. Fouad IA, Hadidi T (2015) Classifying brain-computer interface features based on statistics and density of power spectrum. Int J Biomed Eng Technol 18(1):1–13. https://doi.org/10.1504/ijbet.2015.069849

    Article  Google Scholar 

  29. Kalunga E, Djouani K, Hamam Y, Chevallier S, Monacelli E (2013) “SSVEP enhancement based on canonical correlation analysis to improve BCI performances, ” (in English). Africon 2013:620–624. https://doi.org/10.1109/AFRCON.2013.6757776

    Article  Google Scholar 

  30. Lee J-Y, Lee Y-R, Kim H-N (2015) Frequency recognition in SSVEP-based BCI systems with a combination of CCA and PSDA. J Inst Electron Inf Eng 52(10):139–147. https://doi.org/10.5573/ieie.2015.52.10.139

    Article  CAS  Google Scholar 

  31. Zhang Y, Zhou G, Jin J, Wang X, Cichocki A (2014) Frequency recognition in SSVEP-based BCI using multiset canonical correlation analysis. Int J Neural Syst 24(4):1450013. https://doi.org/10.1142/S0129065714500130

    Article  PubMed  Google Scholar 

  32. Chen YJ, Chen SC, Zaeni IAE, Wu CM (2016) Fuzzy tracking and control algorithm for an SSVEP-based BCI system, (in English). Appl Sci-Basel 6(10):270 ARTN 27010.3390/app6100270

  33. Jain PK, Gupta S, Bhavsar A, Nigam A, Sharma N (2020) Localization of common carotid artery transverse section in B-mode ultrasound images using faster RCNN: a deep learning approach. Med Biol Eng Comput 58(3):471–482. https://doi.org/10.1007/s11517-019-02099-3

    Article  PubMed  Google Scholar 

  34. Li J et al (2017) A unified canonical correlation analysis-based framework for removing gradient artifact in concurrent EEG/fMRI recording and motion artifact in walking recording from EEG signal. Med Biol Eng Comput 55(9):1669–1681. https://doi.org/10.1007/s11517-017-1620-3

    Article  PubMed  Google Scholar 

  35. Zhang Y, Zhou G, Jin J, Wang X, Cichocki A (2015) SSVEP recognition using common feature analysis in brain-computer interface. J Neurosci Methods 244:8–15. https://doi.org/10.1016/j.jneumeth.2014.03.012

    Article  PubMed  Google Scholar 

  36. Nwachukwu SE et al. (2019) An SSVEP recognition method by combining individual template with CCA, (in English). 3rd Int Conf Innov Artif Intell (Iciai 2019) 6–10 https://doi.org/10.1145/3319921.3319925

  37. Tello RMG, Muller SMT, Bastos T, Ferreira A (2014) A comparison of techniques and technologies for SSVEP classification, (in English). 5th Issnip-Ieee Biosignals Biorobotics Conf (2014): Biosignals Robotics Better Safer Living 7–12 https://doi.org/10.1109/BRC.2014.6880956

  38. Tong C, Wang H, Wang Y (2022) Relation of canonical correlation analysis and multivariate synchronization index in SSVEP detection. Biomed Signal Process Control 73:103345. https://doi.org/10.1016/j.bspc.2021.103345

    Article  Google Scholar 

  39. Lalor EC et al (2005) “Steady-state VEP-based brain-computer interface control in an immersive 3D gaming environment, ” (in English). Eurasip J Appl Signal Process 2005(19):3156–3164. https://doi.org/10.1155/Asp.2005.3156

    Article  Google Scholar 

Download references

Funding

This work was supported in part by the National Key R&D Program of China, grant no. 2021YFC0122700; the National Natural Science Foundation of China, grant no. 61904038 and no. U1913216; the Shanghai Sailing Program, grant no. 19YF1403600; the Shanghai Municipal Science and Technology Commission, grant no. 19441907600, no. 19441908200, and no. 19511132000; and the Yiwu Research Institute of Fudan University, grant no. 20–1-16. Opening Project of Zhejiang Lab, grant no. 2021MC0AB01; Fudan University-CIOMP Joint Fund, grant no. FC2019-002; Ji Hua Laboratory, grant no. X190021TB190 and no. X190021TB193; Shanghai Municipal Science and Technology Major Project, grant no. 2021SHZDZX0103 and no. 2018SHZDZX01.

Author information

Authors and Affiliations

  1. Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China

    Lan Niu, Jianxiong Bin, Lihua Zhang, Zhongxue Gan & Xiaoyang Kang

  2. Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China

    Lan Niu, Jianxiong Bin, Jun Kong Shuai Wang, Gege Zhan, Lihua Zhang, Zhongxue Gan & Xiaoyang Kang

  3. Department of Rehabilitation Medicine, Huashan Hospital, Fudan University, Shanghai, 200040, China

    Jie Jia

  4. Yiwu Research Institute of Fudan University, Chengbei Road, Yiwu City, 322000, Zhejiang, China

    Xiaoyang Kang

  5. Research Center for Intelligent Sensing, Zhejiang Lab, Hangzhou, 311100, China

    Xiaoyang Kang

Authors
  1. Lan Niu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianxiong Bin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jun Kong Shuai Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gege Zhan
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jie Jia
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lihua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Zhongxue Gan
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiaoyang Kang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaoyang Kang.

Ethics declarations

Conflict of interest

The authors declare no competing financial interest.

Additional information

Publisher's note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Niu, L., Bin, J., Wang, J.K.S. et al. Effect of 3D paradigm synchronous motion for SSVEP-based hybrid BCI-VR system. Med Biol Eng Comput 61, 2481–2495 (2023). https://doi.org/10.1007/s11517-023-02845-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11517-023-02845-8

Keywords

Search

Navigation