Skip to main content

A Virtual Mouse Based on Parallel Cooperation of Eye Tracker and Motor Imagery

  • Conference paper
  • First Online:
Image and Graphics (ICIG 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12890))

Included in the following conference series:

  • 2562 Accesses

Abstract

It has become a reality for paralyzed patients to use independent BMI-based assistive application to facilitate their lives. However, it was still a problem to switch among applications, if applications like typing and wheelchair-control were integrated in a device. In order to solve this problem, this paper realized a virtual mouse by combining eye tracker and MI, hoping to help patients to switch BMI applications independently and conveniently. The improved VT filter proposed in this paper solved the stable validity problem of the cursor. During the 2000 ms for MI, the time when the mouse stabilized in the valid area of the file can reach 94%. Asynchronous MI was realized in this paper through sliding window to enhance the flexibility of the system. The classification accuracy of left hand and right hand was 92.36%, that of left hand and idle state was 90.28%, and that of right hand and idle state was 90.63%. The final results were obtained by voting, which reached 91.33%. The results proved that the decoding of MI data would not be impacted during the control of eye tracker, which meant that the parallel control mechanism toward multi-cognitive modality proposed in this paper was feasible. This multi-cognitive modality based parallel control mechanism is in line with cognitive habits of human beings, and promising in improving the performance of human-machine fusion systems, in the future.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Berger, H.I.: Über das Elektrenkephalogramm des Menschen. Eur. Arch. Psychiatry Clin. Neurosci. 87, 527–570 (1928)

    Google Scholar 

  2. Liu, Y., Habibnezhad, M., Jebelli, H.: Brain-computer interface for hands-free teleoperation of construction robots. Autom. Constr. 123 (2021)

    Google Scholar 

  3. B., et al.: Noninvasive neuroimaging enhances continuous neural tracking for robotic device control. Sci. Robot. 4(31), eaaw6844 (2019)

    Google Scholar 

  4. Han, X., et al.: A novel system of SSVEP-based human-robot coordination. J. Neural Eng. 16(1), 016006 (2019)

    Google Scholar 

  5. Kuhner, D., et al.: A service assistant combining autonomous robotics, flexible goal formulation, and deep-learning-based brain–computer interfacing. Robot. Auton. Syst. 116, 98–113 (2019)

    Article  Google Scholar 

  6. Xu, Y., et al.: Shared control of a robotic arm using non-invasive brain–computer interface and computer vision guidance. Robot. Auton. Syst. 115, 121–129 (2019)

    Article  Google Scholar 

  7. Hochberg, L.R., et al.: Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398), 372–375 (2012)

    Article  Google Scholar 

  8. Wessberg, J., et al.: Real-time prediction of hand trajectory by ensembles of cortical neurons in primates. Nature 408(6810), 361–365 (2000)

    Article  Google Scholar 

  9. Farwell, L.A., Donchin, E.: Talking off the top of your head: toward a mental prosthesis utilizing event-related brain potentials. Electroencephalogr. Clin. Neurophysiol. 70(6) (1988)

    Google Scholar 

  10. Serruya, M.D., et al.: Instant neural control of a movement signal. Nature (2002)

    Google Scholar 

  11. Kim, S.P., et al.: Neural control of computer cursor velocity by decoding motor cortical spiking activity in humans with tetraplegia. J. Neural Eng. 5(4), 455–476 (2008)

    Article  Google Scholar 

  12. Simeral, J.D., et al.: Neural control of cursor trajectory and click by a human with tetraplegia 1000 days after implant of an intracortical microelectrode array. J. Neural Eng. 8(2), 025027 (2011)

    Google Scholar 

  13. Degenhart, A.D., et al.: Stabilization of a brain–computer interface via the alignment of low-dimensional spaces of neural activity. Nature Biomed. Eng. 4(7), 1–14 (2020)

    Article  Google Scholar 

  14. Egovnik, T., et al.: An analysis of the suitability of a low-cost eye tracker for assessing the cognitive load of drivers. Appl. Ergon. 68, 1–11 (2018)

    Article  Google Scholar 

  15. Fabio, R.A., et al.: Longitudinal cognitive rehabilitation applied with eye-tracker for patients with Rett Syndrome. Res. Dev. Disabil. 111(1), 103891 (2021)

    Google Scholar 

  16. Daniela, V., et al.: Visual exploration patterns of human figures in action: an eye tracker study with art paintings. Front. Psychol. 6, 1636 (2015)

    Google Scholar 

  17. Chen, Y., Tsai, M.J.: Eye-hand coordination strategies during active video game playing: an eye-tracking study. Comput. Hum. Behav. 51(OCT.), 8–14 (2015)

    Google Scholar 

  18. Tchalenko, J.: Eye-hand coordination in portrait drawing (2001)

    Google Scholar 

  19. Gonzalez, D.A., Niechwiej-Szwedo, E.: The effects of monocular viewing on hand-eye coordination during sequential grasping and placing movements. Vis. Res. 128, 30–38 (2016)

    Article  Google Scholar 

  20. Wang, H., et al.: Hybrid gaze/EEG brain computer interface for robot arm control on a pick and place task. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (2015)

    Google Scholar 

  21. Dong, X., et al.: Hybrid Brain Computer Interface via Bayesian integration of EEG and eye gaze. In: 2015 7th International IEEE/EMBS Conference on Neural Engineering (NER) (2015)

    Google Scholar 

  22. Doherty, D.O., et al.: Exploring gaze-motor imagery hybrid brain-computer interface design. In: The IEEE International Conference on Bioinformatics and Biomedicine (BIBM) (2015)

    Google Scholar 

  23. Olsen, A.: The Tobii I-VT fixation filter: algorithm description (2012)

    Google Scholar 

  24. Delorme, A., Makeig, S.: EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134(1), 9–21 (2004)

    Article  Google Scholar 

  25. Wu, Z., Huang, N.E.: Ensemble empirical mode decomposition: a noise-assisted data analysis method. Adv. Adapt. Data Anal. 1(1) (2009)

    Google Scholar 

  26. Huang, N.E., et al.: The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proc. Roy. Soc. Lond. Ser. A: Math. Phys. Eng. Sci. 454(1971), 903–995 (1998)

    Google Scholar 

  27. Müller-Gerking, J., Pfurtscheller, G., Flyvbjerg, H.: Designing optimal spatial filters for single-trial EEG classification in a movement task. Clin. Neurophysiol. 110(5), 787–798 (1999)

    Article  Google Scholar 

  28. Ramoser, H., Muller-Gerking, J.: Optimal spatial filtering of single trial EEG during imagined hand movement. IEEE Trans. Rehabil. Eng. 8, 441–446 (2000)

    Article  Google Scholar 

  29. Johnson, G.D., Krusienski, D.J.: Ensemble SWLDA classifiers for the P300 speller. In: Jacko, J.A. (ed.) HCI 2009. LNCS, vol. 5611, pp. 551–557. Springer, Heidelberg (2009). https://doi.org/10.1007/978-3-642-02577-8_60

    Chapter  Google Scholar 

  30. Krusienski, D.J., et al.: Toward enhanced P300 speller performance. J. Neurosci. Methods 167(1), 15–21 (2008)

    Article  Google Scholar 

  31. Zeyl, T., et al.: Adding real-time Bayesian ranks to error-related potential scores improves error detection and auto-correction in a P300 speller. IEEE Trans. Neural Syst. Rehabil. Eng. 24(1), 46–56 (2016)

    Article  Google Scholar 

  32. Behroozmand, R., Korzyukov, O., Larson, C.R.: Effects of voice harmonic complexity on ERP responses to pitch-shifted auditory feedback. Clin. Neurophysiol. 122(12), 2408–2417 (2011)

    Article  Google Scholar 

  33. Kim, K., et al.: Odor habituation can modulate very early olfactory event-related potential. Sci. Rep. 10(1), 18117 (2020)

    Article  Google Scholar 

Download references

Acknowledgement

This work was supported by the National Natural Science Foundation of China (62006239).

Author information

Authors and Affiliations

Authors

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ye, Z., Liu, Y., Yu, Y., Zeng, L., Zhou, Z., Xie, F. (2021). A Virtual Mouse Based on Parallel Cooperation of Eye Tracker and Motor Imagery. In: Peng, Y., Hu, SM., Gabbouj, M., Zhou, K., Elad, M., Xu, K. (eds) Image and Graphics. ICIG 2021. Lecture Notes in Computer Science(), vol 12890. Springer, Cham. https://doi.org/10.1007/978-3-030-87361-5_53

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-87361-5_53

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-87360-8

  • Online ISBN: 978-3-030-87361-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics