Skip to main content
SpringerLink
Log in

A deep learning scheme for mental workload classification based on restricted Boltzmann machines

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

The mental workload (MWL) classification is a critical problem for quantitative assessment and analysis of operator functional state in many safety-critical situations with indispensable human–machine cooperation. The MWL can be measured by psychophysiological signals. In this work, we propose a novel restricted Boltzmann machine (RBM) architecture for MWL classification. In relation to this architecture, we examine two main issues: the optimal structure of RBM and selection of the most important EEG channels (electrodes) for MWL classification. The trial-and-error and entropy-based pruning methods are compared for the RBM structure identification. The degree of importance of EEG channels is calculated from the weights in a well-trained network in order to select the most relevant channels for classification task. Extensive comparative results showed that the selected EEG channels lead to accurate MWL classification across subjects.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22

Similar content being viewed by others

Abbreviations

MWL:

Mental workload

OFS:

Operator functional state

RBM:

Restricted Boltzmann machine

DBN:

Deep belief network

EEG:

Electroencephalogram

EOG:

Electrooculogram

ECG:

Electrocardiogram

MAD:

Mean absolute difference

aCAMS:

Automation-enhanced Cabin Air Management System

MPF:

Mean power–frequency

SE:

Sample entropy

References

  • An B, Ning Y, Jiang Z, Feng H, Zhou H (2006) Classifying ECoG/EEG-based motor imagery tasks. In: Proceedings of the 28th annual international conference of the IEEE Engineering in Medicine and Biology Society, New York, pp 6339–6342

  • Anthimopoulos M, Christodoulidis S, Ebner L, Christe A, Mougiakakou S (2016) Lung pattern classification for interstitial lung diseases using a deep convolutional neural network. IEEE Trans Med Imaging 35(5):1207–1216

    Article  Google Scholar 

  • Bell P, Swietojanski P, Renals S (2017) Multitask learning of context-dependent targets in deep neural network acoustic models. IEEE/ACM Trans Audio Speech Lang Process 25(2):238–247

    Article  Google Scholar 

  • Borghini G, Astolfi L, Vecchiato G, Mattia D, Babiloni F (2014) Measuring neurophysiological signals in aircraft pilots and car drivers for the assessment of mental workload, fatigue and drowsiness. Neurosci Biobehav Rev 44:58–75

    Article  Google Scholar 

  • Cecotti H, Graser A (2011) Convolutional neural networks for P300 detection with application to brain–computer interfaces. IEEE Trans Pattern Anal Mach Intell 33(3):433–445

    Article  Google Scholar 

  • Choi B, Lee JH, Kim DH (2008) Solving local minima problem with large number of hidden nodes on two-layered feed-forward artificial neural networks. Neurocomputing 71(16):3640–3643

    Article  Google Scholar 

  • Degani A, Goldman CV, Deutsch O, Tsimhoni O (2017) On human–machine relations. Cogn Technol Work. doi:10.1007/s10111-017-0417-3

    Google Scholar 

  • DiDomenico A, Nussbaum MA (2008) Interactive effects of physical and mental workload on subjective workload assessment. Int J Ind Ergon 38(11–12):977–983

    Article  Google Scholar 

  • Faure V, Lobjois R, Benguigui N (2016) The effects of driving environment complexity and dual tasking on drivers’ mental workload and eye blink behavior. Transp Res Part F Traffic Psychol Behav 40:78–90

    Article  Google Scholar 

  • Fischer A, Igel C (2014) Training restricted Boltzmann machines: an introduction. Pattern Recogn 47(1):25–39

    Article  MATH  Google Scholar 

  • Friedman N, Geiger D, Goldszmidt M (1997) Bayesian network classifier. Mach Learn 29(2–3):131–163

    Article  MATH  Google Scholar 

  • García-Gimeno RM, Hervás-Martínez C, De Siloniz MI (2002) Improving artificial neural networks with a pruning methodology and genetic algorithms for their application in microbial growth prediction in food. Int J Food Microbiol 72(1–2):19–30

    Article  Google Scholar 

  • Ghaemi A, Rashedi E, Pourrahimi AM, Kamandar M, Rahdari F (2017) Automatic channel selection in eeg signals for classification of left or right hand movement in brain computer interfaces using improved binary gravitation search algorithm. Biomed Signal Process Control 33:109–118

    Article  Google Scholar 

  • Goljahani A, D’Avanzo C, Schiff S, Amodio P, Bisiacchi P, Sparacino G (2011) A novel method for the determination of the EEG individual alpha frequency. Neuroimage 60(1):774–786

    Article  Google Scholar 

  • He S, Qu T, Huang X, Huang X (1998) Survey of architecture for multilayer feedforward neural networks. Control Theory Appl 15(3):313–319

    Google Scholar 

  • He T, Fan Y, Qian Y, Tan T, Yu K (2014) Reshaping deep neural network for fast decoding by node-pruning. In: Proceedings of the IEEE international conference on acoustics, speech and signal processing, Florence, pp 245–249

  • Hinton GE, Osindero S, Teh YW (2006) A fast learning algorithm for deep belief nets. Neural Comput 18(7):1527–1554

    Article  MathSciNet  MATH  Google Scholar 

  • Holden RJ (2011) Cognitive performance-altering effects of electronic medical records: an application of the human factors paradigm for patient safety. Cogn Technol Work 13(1):11–29

    Article  Google Scholar 

  • Horat SK, Herrmann FR, Favre G, Terzis J, Debatisse D, Merlo MC et al (2016) Assessment of mental workload: a new electrophysiological method based on intra-block averaging of ERP amplitudes. Neuropsychologia 82:11–17

    Article  Google Scholar 

  • Jenkins ND, Housh TJ, Bergstrom HC, Cochrane KC, Hill EC, Smith CM et al (2016) Basic reporting and interpretation of surface EMG amplitude and mean power frequency: a reply to Vitgotsky, Ogborn, and Phillips. Eur J Appl Physiol 116(3):659–661

    Article  Google Scholar 

  • Kontogiannis T (1999) Training effective human performance in the management of stressful emergencies. Cogn Technol Work 1(1):7–24

    Article  Google Scholar 

  • Kontogiannis T, Malakis S (2013) Strategies in controlling, coordinating and adapting performance in air traffic control: modelling ‘loss of control’ events. Cogn Technol Work 15(2):153–169

    Article  Google Scholar 

  • Krucoff MO, Rahimpour S, Slutzky MW, Edgerton VR, Turner DA (2016) Enhancing nervous system recovery through neurobiologics, neural interface training, and neurorehabilitation. Front Neurosci 10:584–585

    Article  Google Scholar 

  • Lal TN, Schroder M, Hinterberger T, Weston J (2004) Support vector channel selection in BCI. IEEE Trans Biomed Eng 51(6):1003–1010

    Article  Google Scholar 

  • Langford J, Li L, Zhang T (2009) Sparse online learning via truncated gradient. J Mach Learn Res 10(2):777–801

    MathSciNet  MATH  Google Scholar 

  • Li X, Chen Z, Liang Q, Yang Y (2014) Analysis of mental stress recognition and rating based on Hidden Markov Model. J Comput Inf Syst 10(18):7911–7919

    Google Scholar 

  • Liang NY, Saratchandran P, Huang GB, Sundararajan N (2006) Classification of mental tasks from EEG signals using extreme learning machine. Int J Neural Syst 16(1):29–38

    Article  Google Scholar 

  • Mahajan R, Morshed BI (2015) Unsupervised eye blink artifact denoising of EEG data with modified multiscale sample entropy, kurtosis, and Wavelet-ICA. IEEE J Biomed Health Inf 19(1):158–165

    Article  Google Scholar 

  • Narasimha PL, Delashmit WH, Manry MT, Li J, Maldonado F (2008) An integrated growing-pruning method for feedforward network training. Neurocomputing 71(13–15):2831–2847

    Article  Google Scholar 

  • Penaranda BN, Baldwin CL (2012) Temporal factors of EEG and artificial neural network classifiers of mental workload. Int J Comput Commun Control 56(1):188–192

    Google Scholar 

  • Qiao JF, Zhang Y, Han HG (2008) Fast unit pruning algorithm for feedforward neural network design. Appl Math Comput 205(2):622–627

    MathSciNet  MATH  Google Scholar 

  • Sanguansat P, Asdornwised W, Jitapunkul S, Marukatat S (2006) Two-dimensional linear discriminant analysis of principle component vectors for face recognition. IEICE Trans Inf Syst 2(7):2164–2170

    Article  Google Scholar 

  • Sarikaya R, Hinton GE, Deoras A (2014) Application of deep belief networks for natural language understanding. IEEE/ACM Trans Audio Speech Lang Process 22(4):778–784

    Article  Google Scholar 

  • Sellberg C, Susi T (2014) Technostress in the office: a distributed cognition perspective on human–technology interaction. Cogn Technol Work 16(2):187–201

    Article  Google Scholar 

  • Smith-Jackson TL, Klein KW (2009) Open-plan offices: task performance and mental workload. J. Environ Psychol 29(2):279–289

    Article  Google Scholar 

  • Sneddon A, Mearns K, Flin R (2006) Situation awareness and safety in offshore drill crews. Cogn Technol Work 8(4):255–267

    Article  Google Scholar 

  • Srivastava N, Salakhutdinov RR (2012) Multimodal learning with deep Boltzmann machines. J Mach Learn Res 15(8):1967–2006

    MathSciNet  MATH  Google Scholar 

  • Tolstrup N (1995) Pruning of a large network by optimal brain damage and surgeon: an example from biological sequence analysis. Int J Neural Syst 6(1):31–42

    Article  Google Scholar 

  • Wang Y, Gao S, Gao X (2006) Common spatial pattern method for channel selection in motor imagery based brain–computer interface. In: Proceedings of the 27th annual international conference of the IEEE Engineering in Medicine and Biology Society, 2005, Shanghai, pp 5392–5395

  • Wang S, Gwizdka J, Chaovalitwongse WA (2016) Using wireless EEG signals to assess memory workload in the n-back task. IEEE Trans Hum Mach Syst 46(3):424–435

    Article  Google Scholar 

  • Wong KFK, Galka A, Yamashita O, Ozaki T (2006) Modelling non-stationary variance in EEG time series by state space GARCH model. Comput Biol Med 36(12):1327–1335

    Article  Google Scholar 

  • Xu P, Liu T, Zhang R, Zhang Y, Yao D (2014) Using particle swarm to select frequency band and time interval for feature extraction of EEG based BCI. Biomed Signal Process Control 10(2):289–295

    Article  Google Scholar 

  • Yin Z, Zhang J (2014) Identification of temporal variations in mental workload using locally-linear-embedding-based EEG feature reduction and support-vector-machine-based clustering and classification techniques. Comput Methods Programs Biomed 115(3):119–134

    Article  Google Scholar 

  • Yin Z, Zhang J (2017) Cross-session classification of mental workload levels using EEG and an adaptive deep learning model. Biomed Signal Process Control 33:30–47

    Article  Google Scholar 

  • Zhang J, Yin Z, Wang R (2015) Recognition of mental workload levels under complex human–machine collaboration by using physiological features and adaptive support vector machines. IEEE Trans Hum Mach Syst 45(2):200–214

    Article  Google Scholar 

  • Zhang Q, Xiao Y, Dai W, Suo J, Wang C, Shi J et al (2016) Deep learning based classification of breast tumors with shear-wave elastography. Ultrasonics 72:150–157

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant No. 61075070 and Key Grant No. 11232005. The authors would like to thank the developers of the aCAMS software used in our experiments.

Author information

Authors and Affiliations

  1. School of Information Science and Engineering, East China University of Science and Technology, Shanghai, 200237, China

    Jianhua Zhang & Sunan Li

Authors
  1. Jianhua Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sunan Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jianhua Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Li, S. A deep learning scheme for mental workload classification based on restricted Boltzmann machines. Cogn Tech Work 19, 607–631 (2017). https://doi.org/10.1007/s10111-017-0430-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-017-0430-6

Keywords

Search

Navigation