Skip to main content
SpringerLink
Log in

A hybrid brain-computer interface control strategy in a virtual environment

  • Published:
Journal of Zhejiang University SCIENCE C Aims and scope Submit manuscript

Abstract

This paper presents a hybrid brain-computer interface (BCI) control strategy, the goal of which is to expand control functions of a conventional motor imagery or a P300 potential based BCI in a virtual environment. The hybrid control strategy utilizes P300 potential to control virtual devices and motor imagery related sensorimotor rhythms to navigate in the virtual world. The two electroencephalography (EEG) patterns serve as source signals for different control functions in their corresponding system states, and state switch is achieved in a sequential manner. In the current system, imagination of left/right hand movement was translated into turning left/right in the virtual apartment continuously, while P300 potentials were mapped to discrete virtual device control commands using a five-oddball paradigm. The combination of motor imagery and P300 patterns in one BCI system for virtual environment control was tested and the results were compared with those of a single motor imagery or P300-based BCI. Subjects obtained similar performances in the hybrid and single control tasks, which indicates the hybrid control strategy works well in the virtual environment.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  • Allison, B.Z., Brunner, C., Kaiser, V., Müller-Putz, G.R., Neuper, C., Pfurtscheller, G., 2010. Toward a hybrid brain-computer interface based on imagined movement and visual attention. J. Neural Eng., 7(2):026007. [doi:10.1088/1741-2560/7/2/026007]

    Article  Google Scholar 

  • Bayliss, J.D., 2003. Use of the evoked potential P3 component for control in a virtual apartment. IEEE Trans. Neural Syst. Rehabil. Eng., 11(2):113–116. [doi:10.1109/TNSRE.2003.814438]

    Article  MathSciNet  Google Scholar 

  • Bayliss, J.D., Ballard, D.H., 2000. A virtual reality testbed for brain-computer interface research. IEEE Trans. Rehabil. Eng., 8(2):188–190. [doi:10.1109/86.847811]

    Article  Google Scholar 

  • Bishop, C.M., 2006. Pattern Recognition and Machine Learning. Springer, USA, p.179–192.

    MATH  Google Scholar 

  • Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., Müller, K.R., 2008. Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process. Mag., 25(1): 41–56. [doi:10.1109/MSP.2008.4408441]

    Article  Google Scholar 

  • Brunner, C., Allison, B.Z., Krusienski, D.J., Kaiser, V., Müller-Putz, G.R., Pfurtscheller, G., Neuper, C., 2010. Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface. J. Neurosci. Meth., 188(1):165–173. [doi:10.1016/j.jneumeth.2010.02.002]

    Article  Google Scholar 

  • Chen, W.D., Zhang, J.H., Zhang, J.C., Li, Y., Qi, Y., Su, Y., Wu, B., Zhang, S.M., Dai, J.H., Zheng, X.X., et al., 2010. A P300 based online brain-computer interface system for virtual hand control. J. Zhejiang Univ.-Sci. C (Comput. & Electron.), 11(8):587–597. [doi: 10.1631/jzus.C0910530]

    Article  Google Scholar 

  • Cristianini, N., Shawe-Taylor, J., 2000. An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods. Cambridge University Press, Cambridge, UK, p.103–131.

    Google Scholar 

  • Croft, R.J., Barry, R.J., 2000. Removal of ocular artifact from the EEG: a review. Clin. Neurophysiol., 30(1):5–19. [doi:10.1016/S0987-7053(00)00055-1]

    Article  Google Scholar 

  • Donchin, E., Spencer, K.M., Wijesinghe, R., 2000. The mental prosthesis: assessing the speed of a P300-based braincomputer interface. IEEE Trans. Rehabil. Eng., 8(2):174–179. [doi:10.1109/86.847808]

    Article  Google Scholar 

  • Edlinger, G., Krausz, G., Groenegress, C., Holzner, C., Guger, C., Slater, M., 2008. Brain-Computer Interfaces for Virtual Environment Control. Proc. 13th Int. Conf. on Biomedical Engineering, p.366–369. [doi:10.1007/978-3-540-92841-6_90]

  • Farwell, L.A., Donchin, E., 1988. Taking off the top of your head-toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol., 70(6):510–523.

    Article  Google Scholar 

  • Kronegg, J., Chanel, G., Voloshynovskiy, S., Pun, T., 2007. EEG-based synchronized brain-computer interfaces: a model for optimizing the number of mental tasks. IEEE Trans. Neural Syst. Rehabil. Eng., 15(1):50–58. [doi:10.1109/tnsre.2007.891389]

    Article  Google Scholar 

  • Leeb, R., Pfurtscheller, G., 2004. Walking Through a Virtual City by Thought. Proc. 26th Annual Int. Conf. of the IEEE Engineering in Medicine and Biology Society, p.4503–4506. [doi:10.1109/IEMBS.2004.1404251]

  • Leeb, R., Lee, F., Keinrath, C., Scherer, R., Bischof, H., Pfurtscheller, G., 2007a. Brain-computer communication: motivation, aim, and impact of exploring a virtual apartment. IEEE Trans. Neural Syst. Rehabil. Eng., 15(4): 473–482. [doi:10.1109/tnsre.2007.906956]

    Article  Google Scholar 

  • Leeb, R., Friedman, D., Müller-Putz, G.R., Scherer, R., Slater, M., Pfurtscheller, G., 2007b. Self-paced (asynchronous) BCI control of a wheelchair in virtual environments: a case study with a tetraplegic. Comput. Intell. Neurosci., 2007:79642. [doi:10.1155/2007/79642]

    Google Scholar 

  • Middendorf, M., McMillan, G., Calhoun, G., Jones, K.S., 2000. Brain-computer interfaces based on the steady-state visual-evoked response. IEEE Trans. Rehabil. Eng., 8(2): 211–214. [doi:10.1109/86.847819]

    Article  Google Scholar 

  • Obermaier, B., Neuper, C., Guger, C., Pfurtscheller, G., 2001. Information transfer rate in a five-classes brain-computer interface. IEEE Trans. Neural Syst. Rehabil. Eng., 9(3): 283–288. [doi:10.1109/7333.948456]

    Article  Google Scholar 

  • Pfurtscheller, G., Neuper, C., 2001. Motor imagery and direct brain-computer communication. Proc. IEEE, 89(7):1123–1134. [doi:10.1109/5.939829]

    Article  Google Scholar 

  • Pfurtscheller, G., Allison, B.Z., Brunner, C., Bauernfeind, G., Solis-Escalante, T., Scherer, R., Zander, T.O., Müller-Putz, G.R., Neuper, C., Birbaumer, N., 2010a. The hybrid BCI. Front. Neurosci., 4:30. [doi:10.3389/fnpro.2010.00003]

    Google Scholar 

  • Pfurtscheller, G., Solis-Escalante, T., Ortner, R., Linortner, P., Müller-Putz, G.R., 2010b. Self-paced operation of an SSVEP-based orthosis with and without an imagerybased “brain switch”: a feasibility study towards a hybrid BCI. IEEE Trans. Neural Syst. Rehabil. Eng., 18(4): 409–414. [doi:10.1109/TNSRE.2010.2040837]

    Article  Google Scholar 

  • Piccione, F., Priftis, K., Tonin, P., Vidale, D., Furlan, R., Cavinato, M., Merico, A., Piron, L., 2008. Task and stimulation paradigm effects in a P300 brain computer interface exploitable in a virtual environment: a pilot study. PsychNol. J., 6(1):99–108.

    Google Scholar 

  • Ramoser, H., Müller-Gerking, J., Pfurtscheller, G., 2000. Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans. Rehabil. Eng., 8(4): 441–446. [doi:10.1109/86.895946]

    Article  Google Scholar 

  • Ron-Angevin, R., Diaz-Estrella, A., 2009. Brain-computer interface: changes in performance using virtual reality techniques. Neurosci. Lett., 449(2):123–127. [doi:10.1016/j.neulet.2008.10.099]

    Google Scholar 

  • Schalk, G., McFarland, D.J., Hinterberger, T., Birbaumer, N., Wolpaw, J.R., 2004. BCI2000: a general-purpose, braincomputer interface (BCI) system. IEEE Trans. Biomed. Eng., 51(6):1034–1043. [doi:10.1109/tbme.2004.827072]

    Article  Google Scholar 

  • Scherer, R., Lee, F., Schloegl, A., Leeb, R., Bischof, H., Pfurtscheller, G., 2008. Toward self-paced braincomputer communication: navigation through virtual worlds. IEEE Trans. Biomed. Eng., 55(2):675–682. [doi: 10.1109/tbme.2007.903709]

    Article  Google Scholar 

  • Su, Y., Wu, B., Chen, W., Zhang, J., Jiang, J., Zhuang, Y., Zheng, X., 2008. P300-based brain computer interface: prototype of a Chinese speller. J. Comput. Inform. Syst., 4(4):1515–1522.

    Google Scholar 

  • Velasco-Álvarez, F., Ron-Angevin, R., 2009. Asynchronous brain-computer interface to navigate in virtual environments using one motor imagery. LNCS, 5517:698–705. [doi:10.1007/978-3-642-02478-8_87]

    Google Scholar 

  • Wolpaw, J.R., Birbaumer, N., Heetderks, W.J., McFarland, D.J., Peckham, P.H., Schalk, G., Donchin, E., Quatrano, L.A., Robinson, C.J., Vaughan, T.M., 2000. Braincomputer interface technology: a review of the first international meeting. IEEE Trans. Rehabil. Eng., 8(2): 164–173. [doi:10.1109/TRE.2000.847807]

    Article  Google Scholar 

  • Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M., 2002. Brain-computer interfaces for communication and control. Clin. Neurophysiol., 113(6):767–791. [doi:10.1016/S1388-2457(02)00057-3]

    Article  Google Scholar 

  • Zhao, Q.B., Zhang, L.Q., Cichockis, A., 2009. EEG-based asynchronous BCI control of a car in 3D virtual reality environments. Chin. Sci. Bull., 54(1):78–87. [doi:10.1007/s11434-008-0547-3]

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, 310027, China

    Yu Su, Yu Qi, Jian-xun Luo, Bian Wu, Fan Yang, Yi Li, Yue-ting Zhuang, Xiao-xiang Zheng & Wei-dong Chen

  2. School of Computer Science and Technology, Zhejiang University, Hangzhou, 310027, China

    Yu Su, Yu Qi, Jian-xun Luo, Fan Yang, Yi Li, Yue-ting Zhuang & Wei-dong Chen

  3. Department of Biomedical Engineering, Zhejiang University, Hangzhou, 310027, China

    Bian Wu & Xiao-xiang Zheng

  4. Key Laboratory of Biomedical Engineering of the Ministry of Education, Hangzhou, 310027, China

    Bian Wu & Xiao-xiang Zheng

Authors
  1. Yu Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yu Qi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jian-xun Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bian Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Fan Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Yi Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yue-ting Zhuang
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Xiao-xiang Zheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Wei-dong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Wei-dong Chen.

Additional information

Project supported by the National Natural Science Foundation of China (Nos. 30800287, 60703038, 60873125, 61001172, and 61031002), the Zhejiang Provincial Natural Science Foundation of China (No. Y2090707), and the Fundamental Research Funds for the Central Universities of China

Rights and permissions

Reprints and permissions

About this article

Cite this article

Su, Y., Qi, Y., Luo, Jx. et al. A hybrid brain-computer interface control strategy in a virtual environment. J. Zhejiang Univ. - Sci. C 12, 351–361 (2011). https://doi.org/10.1631/jzus.C1000208

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1631/jzus.C1000208

Key words

CLC number

Search

Navigation