Skip to main content
SpringerLink
Log in

A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI

  • Original Article
  • Published:
Neuroinformatics Aims and scope Submit manuscript

Abstract

Decoding neural responses from multimodal information sources, including electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS), has the transformative potential to advance hybrid brain-computer interfaces (hBCIs). However, existing modest performance improvement of hBCIs might be attributed to the lack of computational frameworks that exploit complementary synergistic properties in multimodal features. This study proposes a multimodal data fusion framework to represent and decode synergistic multimodal motor imagery (MI) neural responses. We hypothesize that exploiting EEG nonlinear dynamics adds a new informative dimension to the commonly combined EEG-fNIRS features and will ultimately increase the synergy between EEG and fNIRS features toward an enhanced hBCI. The EEG nonlinear dynamics were quantified by extracting graph-based recurrence quantification analysis (RQA) features to complement the commonly used spectral features for an enhanced multimodal configuration when combined with fNIRS. The high-dimensional multimodal features were further given to a feature selection algorithm relying on the least absolute shrinkage and selection operator (LASSO) for fused feature selection. Linear support vector machine (SVM) was then used to evaluate the framework. The mean hybrid classification performance improved by up to 15% and 4% compared to the unimodal EEG and fNIRS, respectively. The proposed graph-based framework substantially increased the contribution of EEG features for hBCI classification from 28.16% up to 52.9% when introduced the nonlinear dynamics and improved the performance by approximately 2%. These findings suggest that graph-based nonlinear dynamics can increase the synergy between EEG and fNIRS features for an enhanced MI response representation that is not dominated by a single modality.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  • Acharya, U. R., Vinitha Sree, S., et al. (2011a). Application of Recurrence Quantification Analysis for the Automated Identification of Epileptic EEG Signals. International Journal of Neural Systems, 21(3), 199–211.

    Article  PubMed  Google Scholar 

  • Acharya, U. R., Chua, E. -P., et al. (2011b). Automated Detection of Sleep Apnea from Electrocardiogram Signals Using Nonlinear Parameters. Physiological Measurement, 32(3), 287–303.

    Article  PubMed  Google Scholar 

  • Ahn, S., & Jun, S. C. (2017). Multi-Modal Integration of EEG-FNIRS for Brain-Computer Interfaces – Current Limitations and Future Directions. Frontiers in Human Neuroscience, 11, 503.

    Article  PubMed  PubMed Central  Google Scholar 

  • Al-Shargie, F., et al. (2016). Mental Stress Assessment Using Simultaneous Measurement of EEG and FNIRS. Biomedical Optics Express, 8, 2583–2598.

    Article  Google Scholar 

  • Al-Shargie, F., Tang, T. B., & Kiguchi, M. (2017). Assessment of Mental Stress Effects on Prefrontal Cortical Activities Using Canonical Correlation Analysis: An FNIRS-EEG Study. Biomedical Optics Express, 8(5), 2583–2598.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ayaz, H., et al. (2013). Continuous Monitoring of Brain Dynamics with Functional near Infrared Spectroscopy as a Tool for Neuroergonomic Research: Empirical Examples and a Technological Development. Frontiers in Human Neuroscience, 7, 871.

    Article  PubMed  PubMed Central  Google Scholar 

  • Baghdadi, G., Amiri, M., Falotico, E., & Laschi, C. (2021). Recurrence Quantification Analysis of EEG Signals for Tactile Roughness Discrimination. International Journal of Machine Learning and Cybernetics, 12(4), 1115–1136.

    Article  Google Scholar 

  • Bauer, C. M., et al. (2017). The Effect of Muscle Fatigue and Low Back Pain on Lumbar Movement Variability and Complexity. Journal of Electromyography and Kinesiology : Official Journal of the International Society of Electrophysiological Kinesiology, 33, 94–102.

    Article  CAS  Google Scholar 

  • Brunner, C., Delorme, A., & Makeig, S. (2013). Eeglab – an Open Source Matlab Toolbox for Electrophysiological Research. Biomedical Engineering.

  • Buccino, A. P., Keles, H. O., & Omurtag, A. (2016). Hybrid EEG-FNIRS Asynchronous Brain-Computer Interface for Multiple Motor Tasks. PLoS ONE, 11(1), 1–16.

    Article  Google Scholar 

  • Chiarelli, A. M., Croce, P., Merla, A., & Zappasodi, F. (2018). Deep Learning for Hybrid EEG-FNIRS Brain-Computer Interface: Application to Motor Imagery Classification. Journal of Neural Engineering, 15(3), 36028.

    Article  Google Scholar 

  • Cui, X., Bray, S., & Reiss, A. L. (2010). Functional near Infrared Spectroscopy (NIRS) Signal Improvement Based on Negative Correlation between Oxygenated and Deoxygenated Hemoglobin Dynamics. NeuroImage, 49, 3039–3046.

    Article  CAS  PubMed  Google Scholar 

  • Deligani, R. J., Borgheai, S. B., McLinden, J., & Shahriari, Y. (2021). Multimodal Fusion of EEG-FNIRS: A Mutual Information-Based Hybrid Classification Framework. Biomedical Optics Express, 12(3), 1635–1650.

    Article  PubMed  PubMed Central  Google Scholar 

  • Donner, R. V., et al. (2010). Recurrence Networks-a Novel Paradigm for Nonlinear Time Series Analysis. New Journal of Physics, 12, 033025.

    Article  Google Scholar 

  • Donner, R. V., Small, M., Donges, J. F., Marwan, N., Zou, Y., Xiang, R., & Kurths, J. (2011). Recurrence-Based Time Series Analysis by Means of Complex Network Methods. International Journal of Bifurcation and Chaos, 21, 1019–1046.

    Article  Google Scholar 

  • Eckmann, J. P., Oliffson Kamphorst, O., & Ruelle, D. (1987). Recurrence Plots of Dynamical Systems. World Scientific Series on Nonlinear Science Series A, 16, 441–446.

    Article  Google Scholar 

  • Fazli, S., et al. (2012). Enhanced Performance by a Hybrid NIRS-EEG Brain Computer Interface. NeuroImage, 59(1), 519–529.

    Article  PubMed  Google Scholar 

  • Feldhoff, J. H., et al. (2013). Geometric Signature of Complex Synchronisation Scenarios. EPL, 102, 30007.

    Article  CAS  Google Scholar 

  • Gao, J. B. (1999). Recurrence Time Statistics for Chaotic Systems and Their Applications. Physical Review Letters, 83(16), 3178.

    Article  CAS  Google Scholar 

  • Holper, L., Shalóm, D. E., Wolf, M., & Sigman, M. (2011). Understanding Inverse Oxygenation Responses during Motor Imagery: A Functional near-Infrared Spectroscopy Study. European Journal of Neuroscience, 33, 2318–2328.

    Article  PubMed  Google Scholar 

  • Hong, K. S., Jawad Khan, M., & Hong, M. J. (2018). Feature Extraction and Classification Methods for Hybrid FNIRS-EEG Brain-Computer Interfaces. Frontiers in Human Neuroscience, 12, 246.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hong, K. S., Raheel Bhutta, M., Liu, X., & Shin, Y. I. (2017). Classification of Somatosensory Cortex Activities Using FNIRS. Behavioural Brain Research, 333, 225–234.

    Article  PubMed  Google Scholar 

  • Hosni, S. M., Borgheai, S. B., McLinden, J., & Shahriari, Y. (2020). An FNIRS-Based Motor Imagery BCI for ALS: A Subject-Specific Data-Driven Approach. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 28(12), 3063–3073.

    Article  CAS  PubMed  Google Scholar 

  • Hosni, S. M., et al. (2019). An Exploration of Neural Dynamics of Motor Imagery for People with Amyotrophic Lateral Sclerosis. Journal of Neural Engineering, 17, 16005.

    Article  Google Scholar 

  • Hu, X. S., Hong, K. S., Ge, S. S., & Jeong, M. Y. (2010). Kalman Estimator- and General Linear Model-Based on-Line Brain Activation Mapping by near-Infrared Spectroscopy. BioMedical Engineering Online, 9, 1–15.

    Article  Google Scholar 

  • Ikegawa, S., et al. (2000). Nonlinear Time-Course of Lumbar Muscle Fatigue Using Recurrence Quantifications. Biological Cybernetics, 82, 373–382.

    Article  CAS  PubMed  Google Scholar 

  • Ismail Hosni, S., et al. (2021). Graph-Based Recurrence Quantification Analysis of EEG Spectral Dynamics for Motor Imagery-Based BCIs. In 43rd Annual International Conference of the IEEE Engineering in Medicine and Biology Society (accepted).

  • Javorka, M., et al. (2009). The Effect of Orthostasis on Recurrence Quantification Analysis of Heart Rate and Blood Pressure Dynamics. Physiological Measurement, 30, 29.

    Article  CAS  PubMed  Google Scholar 

  • Jiang, J., et al. (2020). Temporal Combination Pattern Optimization Based on Feature Selection Method for Motor Imagery BCIs. Frontiers in Human Neuroscience, 14, 231.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kasahara, T., et al. (2012). The Correlation between Motor Impairments and Event-Related Desynchronization during Motor Imagery in ALS Patients. BMC Neuroscience, 13, 1–10.

    Article  Google Scholar 

  • Khan, M. J., Hong, M. J., & Hong, K. -S. (2014). Decoding of Four Movement Directions Using Hybrid NIRS-EEG Brain-Computer Interface. Frontiers in Human Neuroscience, 8, 244.

    Article  Google Scholar 

  • Kübler, A., et al. (2005). Patients with ALS Can Use Sensorimotor Rhythms to Operate a Brain-Computer Interface. Neurology, 64, 1775–1777.

    Article  PubMed  Google Scholar 

  • Li, R., Potter, T., Huang, W., & Zhang, Y. (2017). Enhancing Performance of a Hybrid EEG-FNIRS System Using Channel Selection and Early Temporal Features. Frontiers in Human Neuroscience, 11, 462.

    Article  PubMed  PubMed Central  Google Scholar 

  • Lotte, F., et al. (2007). A Review of Classification Algorithms for EEG-Based Brain-Computer Interfaces. Journal of Neural Engineering, 4(2), R1-13.

    Article  CAS  PubMed  Google Scholar 

  • Marwan, N. (2013). Cross Recurrence Plot Toolbox for MATLAB®. http://tocsy.pik-potsdam.de/CRPtoolbox/. Accessed 28 Jul 2020.

  • Marwan, N., et al. (2002). Recurrence-Plot-Based Measures of Complexity and Their Application to Heart-Rate-Variability Data. Physical Review E - Statistical Physics, Plasmas, Fluids, and Related Interdisciplinary Topics, 66, 026702.

    Article  Google Scholar 

  • Marwan, N., Donges, J. F., Zou, Y., Donner, R. V., & Kurths, J. (2009). Complex Network Approach for Recurrence Analysis of Time Series. Physics Letters, Section A: General, Atomic and Solid State Physics, 373, 4246–4254.

    Article  CAS  Google Scholar 

  • Marwan, N., Carmen Romano, M., Thiel, M., & Kurths, J. (2007). Recurrence Plots for the Analysis of Complex Systems. Physics Reports, 438, 237–329.

    Article  Google Scholar 

  • Marwan, N., & Meinke, A. (2004). Extended Recurrence Plot Analysis and Its Application to ERP Data. International Journal of Bifurcation and Chaos in Applied Sciences and Engineering, 14, 761–771.

    Google Scholar 

  • McFarland, D. J., McCane, L. M., David, S. V., & Wolpaw, J. R. (1997). Spatial Filter Selection for EEG-Based Communication. Electroencephalography and Clinical Neurophysiology, 103, 386–394.

    Article  CAS  PubMed  Google Scholar 

  • McKenna, T. M., McMullen, T. A., & Shlesinger, M. F. (1994). The Brain as a Dynamic Physical System. Neuroscience, 60(3), 587–605.

    Article  CAS  PubMed  Google Scholar 

  • Naseer, N., & Hong, K. S. (2013). Classification of Functional Near-Infrared Spectroscopy Signals Corresponding to the Right- and Left-Wrist Motor Imagery for Development of a Brain-Computer Interface. Neuroscience Letters, 553, 84–89.

    Article  CAS  PubMed  Google Scholar 

  • Naseer, N., & Hong, K. -S. (2015). FNIRS-Based Brain-Computer Interfaces: A Review. Frontiers in Human Neuroscience, 9, 3.

    PubMed  PubMed Central  Google Scholar 

  • Naseer, N., Noori, F. M., Qureshi, N. K., & Hong, K. S. (2016). Determining Optimal Feature-Combination for LDA Classification of Functional near-Infrared Spectroscopy Signals in Brain-Computer Interface Application. Frontiers in Human Neuroscience, 10, 237.

    Article  PubMed  PubMed Central  Google Scholar 

  • Ngamga, E. J., et al. (2016). Evaluation of Selected Recurrence Measures in Discriminating Pre-Ictal and Inter-Ictal Periods from Epileptic EEG Data. Physics Letters, Section a: General, Atomic and Solid State Physics, 380, 1419–1425.

    Article  CAS  Google Scholar 

  • Nguyen, T., et al. (2017). Utilization of a Combined EEG/NIRS System to Predict Driver Drowsiness. Scientific Reports, 7(1), 43933.

    Article  PubMed  PubMed Central  Google Scholar 

  • Pfurtscheller, G., & Lopes Da Silva, F. H. (1999). Event-Related EEG/MEG Synchronization and Desynchronization: Basic Principles. Clinical Neurophysiology, 110(11), 1842–1857.

    Article  CAS  PubMed  Google Scholar 

  • Pitsik, E., et al. (2020). Motor Execution Reduces EEG Signals Complexity: Recurrence Quantification Analysis Study. Chaos, 30, 023111.

    Article  PubMed  Google Scholar 

  • Qureshi, N. K., et al. (2017). Enhancing Classification Performance of Functional Near-Infrared Spectroscopy-Brain-Computer Interface Using Adaptive Estimation of General Linear Model Coefficients. Frontiers in Neurorobotics, 11, 33.

    Article  PubMed  PubMed Central  Google Scholar 

  • Saadati, M., Nelson, J., & Ayaz, H. (2020a). Convolutional Neural Network for Hybrid FNIRS-EEG Mental Workload Classification. In A. Hasan (Ed.), International Conference on Applied Human Factors and Ergonomics (pp. 221–232). Cham: Springer International Publishing.

    Google Scholar 

  • Saadati, M., Nelson, J., & Ayaz, H. (2020b). Multimodal FNIRS-EEG Classification Using Deep Learning Algorithms for Brain-Computer Interfaces Purposes. In H. Ayaz (Ed.), Advances in Neuroergonomics and Cognitive Engineering (pp. 209–220). Springer International Publishing.

    Chapter  Google Scholar 

  • Santosa, H., Hong, M. J., & Hong, K. S. (2014). Lateralization of Music Processing with Noises in the Auditory Cortex: An FNIRS Study. Frontiers in Behavioral Neuroscience, 8, 418.

    Article  PubMed  PubMed Central  Google Scholar 

  • Sassaroli, A., & Fantini, S. (2004). Comment on the Modified Beer-Lambert Law for Scattering Media. Physics in Medicine and Biology, 49, N255.

    Article  PubMed  Google Scholar 

  • Schalk, G., et al. (2004). BCI2000: A General-Purpose Brain-Computer Interface (BCI) System. IEEE Transactions on Biomedical Engineering, 51, 1034–1043.

    Article  PubMed  Google Scholar 

  • Shin, J., et al. (2018). Simultaneous Acquisition of EEG and NIRS during Cognitive Tasks for an Open Access Dataset. Scientific Data, 5(1), 180003.

    Article  PubMed  PubMed Central  Google Scholar 

  • Takens, F. (1981). Detecting Strange Attractors in Turbulence. In D. Rand & L. S. Young (Eds.), Dynamical Systems and Turbulence, Warwick 1980. Lecture Notes in Mathematics 898. Berlin, Heidelberg: Springer.

    Google Scholar 

  • Venugopalan, J., Tong, Li., Hassanzadeh, H. R., & Wang, M. D. (2021). Multimodal Deep Learning Models for Early Detection of Alzheimer’s Disease Stage. Scientific Reports, 11(1), 1–13.

    Article  Google Scholar 

  • von Lühmann, A., Ortega-Martinez, A., Boas, D. A., & Yücel, M. A. (2020). Using the General Linear Model to Improve Performance in FNIRS Single Trial Analysis and Classification: A Perspective. Frontiers in Human Neuroscience, 14, 30.

    Article  Google Scholar 

  • Webber, C. L., Jr., & Marwan, N. (2015). Recurrence Quantification Analysis – Theory and Best Practices. Understanding Complex Systems. Springer International Publishing, Cham Switzerland.

    Book  Google Scholar 

  • Wu, C. W., et al. (2019). Indication of Dynamic Neurovascular Coupling from Inconsistency between EEG and FMRI Indices across Sleep-Wake States. Sleep and Biological Rhythms, 17(4), 423–431.

    Article  Google Scholar 

  • Yin, X., et al. (2015). A Hybrid BCI Based on EEG and FNIRS Signals Improves the Performance of Decoding Motor Imagery of Both Force and Speed of Hand Clenching. Journal of Neural Engineering, 12(3), 36004.

    Article  Google Scholar 

  • Zbilut, J. P., Thomasson, N., & Webber, C. L. (2002). Recurrence Quantification Analysis as a Tool for Nonlinear Exploration of Nonstationary Cardiac Signals. Medical Engineering & Physics, 24(1), 53–60.

    Article  Google Scholar 

Download references

Funding

This study was supported by the National Science Foundation (NSF-1913492, NSF-2006012) and the Institutional Development Award (IDeA) Network for Biomedical Research Excellence (P20GM103430).

Author information

Authors and Affiliations

  1. Department of Electrical, Computer & Biomedical Engineering, University of Rhode Island (URI), Kingston, RI, 02881, USA

    Sarah M. I. Hosni, Seyyed. B. Borgheai, John McLinden & Yalda Shahriari

  2. Department of Electrical and Computer Engineering, Northeastern University, Boston, MA, 02115, USA

    Shaotong Zhu, Xiaofei Huang & Sarah Ostadabbas

Authors
  1. Sarah M. I. Hosni
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Seyyed. B. Borgheai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. John McLinden
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Shaotong Zhu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaofei Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Sarah Ostadabbas
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yalda Shahriari
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yalda Shahriari.

Ethics declarations

Ethics Approval

The data recording was performed in the NeuralPC Lab, University of Rhode Island (URI) with Institutional Review Board (IRB) approval. Consent forms were obtained from all the subjects participating in the study.

Financial Interests

The authors declare they have no financial interests.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 1247 KB)

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hosni, S.M.I., Borgheai, S.B., McLinden, J. et al. A Graph-Based Nonlinear Dynamic Characterization of Motor Imagery Toward an Enhanced Hybrid BCI. Neuroinform 20, 1169–1189 (2022). https://doi.org/10.1007/s12021-022-09595-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12021-022-09595-2

Keywords

Search

Navigation