Skip to main content
SpringerLink
Log in

On Time Series Cross-Validation for Deep Learning Classification Model of Mental Workload Levels Based on EEG Signals

  • Conference paper
  • First Online:
Machine Learning, Optimization, and Data Science (LOD 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13811))

Abstract

The determination of a subject’s mental workload (MWL) from an electroencephalogram (EEG) is a well-studied area in the brain-computer interface (BCI) field. A high MWL level can significantly contribute to mental fatigue, decreased performance, and long-term health problems. Inspired by the success of machine learning in various areas, researchers have investigated the use of deep learning models to classify subjects’ MWL levels. A common approach that is used to evaluate such classification models is the cross-validation (CV) technique. However, the CV technique used for such models does not take into account the time series nature of EEG signals. Therefore, in this paper we propose a modification of CV techniques, i.e. a blocked form of CV with rolling window and expanding window strategies, which are more suitable for EEG signals. Then, we investigate the effectiveness of the two strategies and also explore the effects of different block sizes for each strategy. We then apply these models to several state-of-the-art deep learning models used for MWL classification from EEG signals using a publicly available dataset, STEW. There were two classification tasks: Task 1- resting vs testing state, and Task 2- low vs moderate vs high MWL. Our results show that the model evaluated by the expanding window strategy, when it was trained using the 90% of data, provided a better performance than the rolling window strategy and that the BGRU-GRU model outperformed the other models for both tasks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Ahmadi, A., Bazregarzadeh, H., Kazemi, K.: Automated detection of driver fatigue from electroencephalography through wavelet-based connectivity. Biocybernetics Biomed. Eng. 41(1), 316–332 (2021)

    Article  Google Scholar 

  2. Bergmeir, C., Benítez, J.M.: On the use of cross-validation for time series predictor evaluation. Inf. Sci. 191, 192–213 (2012)

    Article  Google Scholar 

  3. Bergmeir, C., Costantini, M., Benítez, J.M.: On the usefulness of cross-validation for directional forecast evaluation. Comput. Stat. Data Anal. 76, 132–143 (2014)

    Article  MathSciNet  MATH  Google Scholar 

  4. Buscher, G., Dengel, A., Biedert, R., Elst, L.V.: Attentive documents: eye tracking as implicit feedback for information retrieval and beyond. ACM Trans. Interact. Intell. Syst. (TiiS) 1(2), 1–30 (2012)

    Article  Google Scholar 

  5. Cao, Z., Yin, Z., Zhang, J.: Recognition of cognitive load with a stacking network ensemble of denoising autoencoders and abstracted neurophysiological features. Cogn. Neurodyn. 15(3), 425–437 (2021)

    Article  Google Scholar 

  6. Cerqueira, V., Torgo, L., Mozetič, I.: Evaluating time series forecasting models: an empirical study on performance estimation methods. Mach. Learn. 109(11), 1997–2028 (2020)

    Article  MathSciNet  MATH  Google Scholar 

  7. Dorffner, G.: Neural networks for time series processing. In: Neural network world. Citeseer (1996)

    Google Scholar 

  8. Hernández, L.G., Mozos, O.M., Ferrández, J.M., Antelis, J.M.: EEG-based detection of braking intention under different car driving conditions. Front. Neuroinform. 12, 29 (2018)

    Article  Google Scholar 

  9. Hoadley, B.: Asymptotic properties of maximum likelihood estimators for the independent not identically distributed case. Annals Math. Stat. 1977–1991 (1971)

    Google Scholar 

  10. Islam, M.K., Rastegarnia, A., Yang, Z.: Methods for artifact detection and removal from scalp EEG: a review. Neurophysiol. Clin. Clin. Neurophysiol. 46(4–5), 287–305 (2016)

    Article  Google Scholar 

  11. Islam, M.R., Barua, S., Ahmed, M.U., Begum, S., Di Flumeri, G.: Deep learning for automatic EEG feature extraction: an application in drivers’ mental workload classification. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2019. CCIS, vol. 1107, pp. 121–135. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32423-0_8

    Chapter  Google Scholar 

  12. Jeong, J.H., Yu, B.W., Lee, D.H., Lee, S.W.: Classification of drowsiness levels based on a deep spatio-temporal convolutional bidirectional LSTM network using electroencephalography signals. Brain Sci. 9(12), 348 (2019)

    Article  Google Scholar 

  13. Kingphai, K., Moshfeghi, Y.: On EEG preprocessing role in deep learning effectiveness for mental workload classification. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2021. CCIS, vol. 1493, pp. 81–98. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-91408-0_6

    Chapter  Google Scholar 

  14. Kingphai, K., Moshfeghi, Y.: On time series cross-validation for mental workload classification from EEG signals. In: Neuroergonomics Conference (2021)

    Google Scholar 

  15. Lee, D.H., Jeong, J.H., Kim, K., Yu, B.W., Lee, S.W.: Continuous EEG decoding of pilots’ mental states using multiple feature block-based convolutional neural network. IEEE Access 8, 121929–121941 (2020). https://doi.org/10.1109/ACCESS.2020.3006907

    Article  Google Scholar 

  16. Lim, J., Wu, W.C., Wang, J., Detre, J.A., Dinges, D.F., Rao, H.: Imaging brain fatigue from sustained mental workload: an ASL perfusion study of the time-on-task effect. Neuroimage 49(4), 3426–3435 (2010)

    Article  Google Scholar 

  17. Lim, W., Sourina, O., Wang, L.: Stew: simultaneous task EEG workload data set. IEEE Trans. Neural Syst. Rehabil. Eng. 26(11), 2106–2114 (2018)

    Article  Google Scholar 

  18. Mognon, A., Jovicich, J., Bruzzone, L., Buiatti, M.: Adjust: an automatic EEG artifact detector based on the joint use of spatial and temporal features. Psychophysiology 48(2), 229–240 (2011)

    Article  Google Scholar 

  19. Murakami, H., Kumar, B.V.: Efficient calculation of primary images from a set of images. IEEE Trans. Pattern Anal. Mach. Intell. 5, 511–515 (1982)

    Article  Google Scholar 

  20. Nagabushanam, P., Thomas George, S., Radha, S.: EEG signal classification using LSTM and improved neural network algorithms. Soft Comput. 24(13), 9981–10003 (2019). https://doi.org/10.1007/s00500-019-04515-0

    Article  Google Scholar 

  21. Nussbaumer, H.J.: The fast fourier transform. In: Fast Fourier Transform and Convolution Algorithms, pp. 80–111. Springer, Berlin (1981). https://doi.org/10.1007/978-3-662-00551-4_4

  22. Qayyum, A., Khan, M.A., Mazher, M., Suresh, M.: Classification of EEG learning and resting states using 1d-convolutional neural network for cognitive load assesment. In: 2018 IEEE Student Conference on Research and Development (SCOReD), pp. 1–5. IEEE (2018)

    Google Scholar 

  23. Qu, X., Liu, P., Li, Z., Hickey, T.: Multi-class time continuity voting for EEG classification. In: Frasson, C., Bamidis, P., Vlamos, P. (eds.) BFAL 2020. LNCS (LNAI), vol. 12462, pp. 24–33. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60735-7_3

    Chapter  Google Scholar 

  24. Qu, X., Sun, Y., Sekuler, R., Hickey, T.: EEG markers of stem learning. In: 2018 IEEE Frontiers in Education Conference (FIE), pp. 1–9 (2018). https://doi.org/10.1109/FIE.2018.8659031

  25. Saha, S., Baumert, M.: Intra-and inter-subject variability in EEG-based sensorimotor brain computer interface: a review. Front. Comput. Neurosci. 13, 87 (2020)

    Article  Google Scholar 

  26. Schaffer, C.: Selecting a classification method by cross-validation. Mach. Learn. 13(1), 135–143 (1993)

    Article  MathSciNet  Google Scholar 

  27. Stone, M.: Cross-validatory choice and assessment of statistical predictions. J. Roy. Stat. Soc.: Ser. B (Methodol.) 36(2), 111–133 (1974)

    MathSciNet  MATH  Google Scholar 

  28. Szalma, J.L., et al.: Effects of sensory modality and task duration on performance, workload, and stress in sustained attention. Hum. Factors 46(2), 219–233 (2004)

    Article  Google Scholar 

  29. Tashman, L.J.: Out-of-sample tests of forecasting accuracy: an analysis and review. Int. J. Forecast. 16(4), 437–450 (2000)

    Article  Google Scholar 

  30. Taylor, L.P.: Chapter 20 - independent assessor audit guide. In: Taylor, L.P. (ed.) FISMA Compliance Handbook, pp. 239–273. Syngress, Boston (2013). https://doi.org/10.1016/B978-0-12-405871-2.00020-8https://www.sciencedirect.com/science/article/pii/B9780124058712000208

  31. Teplan, M., et al.: Fundamentals of EEG measurement. Meas. Sci. Rev. 2(2), 1–11 (2002)

    Google Scholar 

  32. Urigüen, J.A., Garcia-Zapirain, B.: EEG artifact removal—state-of-the-art and guidelines. J. Neural Eng. 12(3), 031001 (2015). https://doi.org/10.1088/1741-2560/12/3/031001,https://doi.org/10.1088%2F1741-2560%2F12%2F3%2F031001

  33. Wang, S., Gwizdka, J., Chaovalitwongse, W.A.: Using wireless EEG signals to assess memory workload in the \( n \)-back task. IEEE Trans. Human-Mach. Syst. 46(3), 424–435 (2015)

    Article  Google Scholar 

  34. Wylie, C., Shultz, T., Miller, J., Mitler, M., Mackie, R., et al.: Commercial motor vehicle driver fatigue and alertness study: Technical summary (1996)

    Google Scholar 

  35. Yang, S., Yin, Z., Wang, Y., Zhang, W., Wang, Y., Zhang, J.: Assessing cognitive mental workload via EEG signals and an ensemble deep learning classifier based on denoising autoencoders. Comput. Biol. Med. 109, 159–170 (2019)

    Article  Google Scholar 

  36. Zeng, H., et al.: A lightgbm-based EEG analysis method for driver mental states classification. Comput. Intell. Neurosci. 2019 (2019)

    Google Scholar 

  37. Zhang, D., Cao, D., Chen, H.: Deep learning decoding of mental state in non-invasive brain computer interface. In: Proceedings of the International Conference on Artificial Intelligence, Information Processing and Cloud Computing, pp. 1–5 (2019)

    Google Scholar 

  38. Zhang, J., Li, S.: A deep learning scheme for mental workload classification based on restricted boltzmann machines. Cognit. Technol. Work 19(4), 607–631 (2017)

    Article  Google Scholar 

  39. Zhang, Q., Yuan, Z., Chen, H., Li, X.: Identifying mental workload using EEG and deep learning. In: 2019 Chinese Automation Congress (CAC), pp. 1138–1142. IEEE (2019)

    Google Scholar 

  40. Zhang, Y., Liu, B., Ji, X., Huang, D.: Classification of EEG signals based on autoregressive model and wavelet packet decomposition. Neural Process. Lett. 45(2), 365–378 (2017)

    Article  Google Scholar 

  41. Zhou, Y., Xu, T., Li, S., Shi, R.: Beyond engagement: an EEG-based methodology for assessing user’s confusion in an educational game. Univ. Access Inf. Soc. 18(3), 551–563 (2019)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. NeuraSearch Laboratory, Department of Computer and Information Sciences, University of Strathclyde, Glasgow, Scotland

    Kunjira Kingphai & Yashar Moshfeghi

Authors
  1. Kunjira Kingphai
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yashar Moshfeghi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yashar Moshfeghi .

Editor information

Editors and Affiliations

  1. University of Catania, Catania, Italy

    Giuseppe Nicosia

  2. University of Reading, Reading, UK

    Varun Ojha

  3. University of Oxford, Oxford, UK

    Emanuele La Malfa

  4. University of Cambridge, Cambridge, UK

    Gabriele La Malfa

  5. University of Florida, Gainesville, FL, USA

    Panos Pardalos

  6. Free University of Bozen-Bolzano, Bolzano, Italy

    Giuseppe Di Fatta

  7. University of Catania, Catania, Italy

    Giovanni Giuffrida

  8. Dana-Farber Cancer Institute, Boston, MA, USA

    Renato Umeton

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kingphai, K., Moshfeghi, Y. (2023). On Time Series Cross-Validation for Deep Learning Classification Model of Mental Workload Levels Based on EEG Signals. In: Nicosia, G., et al. Machine Learning, Optimization, and Data Science. LOD 2022. Lecture Notes in Computer Science, vol 13811. Springer, Cham. https://doi.org/10.1007/978-3-031-25891-6_30

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-25891-6_30

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-25890-9

  • Online ISBN: 978-3-031-25891-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation