Abstract
Mental workload assessment is a core element of designing complex high-precision human machine interfaces with industrial and medical applications, from aviation to robotic neurosurgery. High accuracy and continuous mental state decoding play an essential role for keeping the operator’s mental workload on a moderate level to prevent cognitive tunneling and improve the safety and performance of complex machine use. Monitoring brain activity using wearable and increasingly portable functional near-infrared spectroscopy (fNIRS) sensors enable measurement in realistic and real-world conditions. While a variety of machine learning techniques have been evaluated for this application, Recurrent Convolutional Neural Networks (R-CNN) have received only minimal attention. A significant advantage of R-CNN compared to other classification methods is that it can capture temporal and spatial patterns of brain activity simultaneously without requiring prior feature selection or computationally demanding preprocessing or denoising. This study represents an investigation on designing a hybrid deep learning architecture combining CNN and RNN (Long Short Term Memory-LSTM). The proposed architecture is evaluated for mental workload memory (n-Back) tasks from an open-source dataset. Proposed architecture demonstrates higher performance than both Fully Connected Deep Neural Network and Support Vector Machine methods, showing a high capacity for simultaneous spatio-temporal pattern recognition. We obtained improvements of 15% and 11% in average subject accuracy with deoxy-hemoglobin (deoxy-Hb) in R-CNN compared to SVM and DNN methods, respectively.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Aghajani, H., Garbey, M., Omurtag, A.: Measuring mental workload with EEG+fNIRS. Front. Hum. Neurosci. 11, 359 (2017)
Bauernfeind, G., Steyrl, D., Brunner, C., Müller-Putz, G.R.: Single trial classification of fNIRS-based brain-computer interface mental arithmetic data: a comparison between different classifiers. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2004–2007. IEEE (2014)
Wang, L., Curtin, A., Ayaz, H.: Comparison of machine learning approaches for motor imagery based optical brain computer interface. In: International Conference on Applied Human Factors and Ergonomics, pp. 124–134. Springer (2018)
Naseer, N., Hong, K.-S.: Classification of functional near-infrared spectroscopy signals corresponding to the right-and left-wrist motor imagery for development of a brain–computer interface. Neurosci. Lett. 553, 84–89 (2013)
Trakoolwilaiwan, T., Behboodi, B., Lee, J., Kim, K., Choi, J.-W.: Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution. Neurophotonics 5, 011008 (2017)
Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning representations from EEG with deep recurrent-convolutional neural networks (2015). arXiv preprint arXiv:1511.06448
Schirrmeister, R.T., et al.: Deep learning with convolutional neural networks for EEG decoding and visualization. Hum. Brain Mapp. 38(11), 5391–5420 (2017)
Saadati, M., Nelson, J., Ayaz, H.: Multimodal fNIRS-EEG classification using deep learning algorithms for brain-computer interfaces purposes. In: Ayaz, H. (ed.) Advances in Neuroergonomics and Cognitive Engineering, pp. 209–220. Springer International Publishing, Cham (2020)
Saadati, M., Nelson, J., Ayaz, H.: Mental workload classification from spatial representation of fNIRS recordings using convolutional neural networks. In: 2019 IEEE International Workshop on Machine Learning for Signal Processing (2019)
Chiarelli, A.M., Zappasodi, F., Di Pompeo, F., Merla, A.: Simultaneous functional near-infrared spectroscopy and electroencephalography for monitoring of human brain activity and oxygenation: a review. Neurophotonics 4(4), 041411 (2017)
Saadati, M., Nelson, J., Ayaz, H.: Convolutional neural network for hybrid fNIRS-EEG mental workload classification. In: Ayaz, H. (ed.) Advances in Neuroergonomics and Cognitive Engineering, pp. 221–232. Springer International Publishing, Cham (2020)
Shin, J., Von Lühmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., Müller, K.-R.: Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Generic Res. Data (2018)
Shin, J., Von Lühmann, A., Kim, D.-W., Mehnert, J., Hwang, H.-J., Müller, K.-R.: Simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Sci. Data 5, 180003 (2018)
Asgher, U., et al.: Enhanced accuracy for multiclass mental workload detection using long short-term memory for brain–computer interface. Front. Neurosci. 14, 584 (2020)
Aram, S., Kornev, D., Sadeghian, R., Sardari, S.E., Venu, S.K., Dashtestani, H., Gandjbakhche, A.: Machine learning approaches and neuroimaging in cognitive functions of the human brain: a review. In: International Conference on Applied Human Factors and Ergonomics, pp. 23–29. Springer, Cham (2020)
Heidari, M., Jones, J.H.: Using BERT to extract topic-independent sentiment features for social media bot detection. In: 2020 11th IEEE Annual Ubiquitous Computing, Electronics & Mobile Communication Conference (UEMCON) New York, NY, USA, pp. 0542–0547 (2020). https://doi.org/10.1109/UEMCON51285.2020.9298158
Heidari, M., Rafatirad, S.: Using transfer learning approach to implement convolutional neural network model to recommend airline tickets by using online reviews. In: 2020 15th International Workshop on Semantic and Social Media Adaptation and Personalization (SMA Zakynthos), Greece, pp. 1–6 (2020). https://doi.org/10.1109/SMAP49528.2020.9248443
Pourshafie, A., Mortazavi, S.S., Saniei, M., Saadati, M., Saidian, A.: Optimal reactive power compensation in a deregulated distribution network. In: 2009 44th International Universities Power Engineering Conference (UPEC) Glasgow, UK, pp. 1–6 (2009)
Saadati, M., Nelson, J.K.: Multiple transmitter localization using clustering by likelihood of transmitter proximity. In: 2017 51st Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, pp. 1769–1773 (2017). https://doi.org/10.1109/ACSSC.2017.8335665
Chakladar, D.D., Dey, S., Roy, P.P., Dogra, D.P.: EEG-based mental workload estimation using deep BLSTM-LSTM network and evolutionary algorithm. Biomed. Sig. Process. Control 60, 101989 (2020)
Pan, C., Shi, C., Mu, H., Li, J., Gao, X.: EEG-based emotion recognition using logistic regression with Gaussian kernel and Laplacian Prior and investigation of critical frequency bands. Appl. Sci. 10(5), 1619 (2020)
Khan, M.U., Hasan, M.A.: Hybrid EEG-fNIRS BCI fusion using multi-resolution singular value decomposition (MSVD). Front. Hum. Neurosci. 14 (2020)
Neshatpour, K., Homayoun, H., Sasan, A.: ICNN: The iterative convolutional neural network. ACM Trans. Embed. Comput. Syst. (TECS) 18(6), 1–27 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Saadati, M., Nelson, J., Curtin, A., Wang, L., Ayaz, H. (2021). Application of Recurrent Convolutional Neural Networks for Mental Workload Assessment Using Functional Near-Infrared Spectroscopy. In: Ayaz, H., Asgher, U., Paletta, L. (eds) Advances in Neuroergonomics and Cognitive Engineering. AHFE 2021. Lecture Notes in Networks and Systems, vol 259. Springer, Cham. https://doi.org/10.1007/978-3-030-80285-1_13
Download citation
DOI: https://doi.org/10.1007/978-3-030-80285-1_13
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-80284-4
Online ISBN: 978-3-030-80285-1
eBook Packages: EngineeringEngineering (R0)