Skip to main content
SpringerLink
Log in

Hybrid Models of Performance Using Mental Workload and Usability Features via Supervised Machine Learning

  • Conference paper
  • First Online:
Book cover Human Mental Workload: Models and Applications (H-WORKLOAD 2019)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1107))

Abstract

Mental Workload (MWL) represents a key concept in human performance. It is a complex construct that can be viewed from multiple perspectives and affected by various factors that are quantified by different collection of methods. In this direction, several approaches exist that aggregate these factors towards building a unique workload index that best acts as a proxy to human performance. Such an index can be used to detect cases of mental overload and underload in human interaction with a system. Unfortunately, limited work has been done to automatically classify such conditions using data mining techniques. The aim of this paper is to explore and evaluate several data mining techniques for classifying mental overload and underload by combining factors from three subjective measurement instruments: System Usability Scale (SUS), Nasa Task Load Index (NASATLX) and Workload Profile (WP). The analysis focused around nine supervised machine learning classification algorithms aimed at inducing model of performance from data. These models underwent through rigorous phases of evaluation such as: classifier accuracy (CA), receiver operating characteristics (ROC) and predictive power using cost/benefit analysis. The findings suggest that Bayesian and tree-based models are the most suitable for classifying mental overload/underload even with unbalanced data.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Reid, G.B., Nygren, T.E.: The subjective workload assessment technique: a scaling procedure for measuring mental workload. J. Adv. Psychol. 52, 158–218 (1988)

    Google Scholar 

  2. Stassen, H.G., Johannsen, G., Moray, N.: Internal representation, internal model, human performance model and mental workload. J. Autom. 26(4), 811–820 (1990)

    Article  Google Scholar 

  3. Wickens, C.D.: Mental workload: assessment, prediction and consequences. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 18–29. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_2

    Chapter  Google Scholar 

  4. Longo, L.: A defeasible reasoning framework for human mental workload representation and assessment. Behav. Inf. Technol. 34(8), 758–786 (2015)

    Article  Google Scholar 

  5. Longo, L.: Human-computer interaction and human mental workload: assessing cognitive engagement in the world wide web. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011. LNCS, vol. 6949, pp. 402–405. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-23768-3_43

    Chapter  Google Scholar 

  6. Longo, L.: Formalising human mental workload as non-monotonic concept for adaptive and personalised web-design. In: Masthoff, J., Mobasher, B., Desmarais, Michel C., Nkambou, R. (eds.) UMAP 2012. LNCS, vol. 7379, pp. 369–373. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-31454-4_38

    Chapter  Google Scholar 

  7. Longo, L.: Designing medical interactive systems via assessment of human mental workload. In: International Symposium on Computer-Based Medical Systems, pp. 364–365 (2015)

    Google Scholar 

  8. Moustafa, K., Luz, S., Longo, L.: Assessment of mental workload: a comparison of machine learning methods and subjective assessment techniques. In: Longo, L., Leva, M.Chiara (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 30–50. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_3

    Chapter  Google Scholar 

  9. Blankertz, B., Curio, G., Muller, K.R.: Classifying single trial EEG: towards brain computer interfacing. In: Advances in Neural Information Processing Systems, vol. 1, pp. 157–164 (2002)

    Google Scholar 

  10. Dornhege, G., Blankertz, B., Curio, G., Muller, K.R.: Boosting bit rates in noninvasive EEG single-trial classifications by feature combination and multiclass paradigms. IEEE Trans. Biomed. Eng. 51(6), 993–1002 (2004)

    Article  Google Scholar 

  11. Stevens, R., Galloway, T., Berka, C.: Integrating EEG models of cognitive load with machine learning models of scientific problem solving. In: Proceedings of 2nd Annual Augmented Cognition International Conference, pp. 55–65 (2006)

    Google Scholar 

  12. Zhang, Y.Z.Y., Owechko, Y., Zhang, J.Z.J.: Driver cognitive workload estimation: a data-driven perspective. In: Proceedings of the 7th International IEEE Conference on Intelligent Transportation Systems (IEEE Cat. No.04TH8749), pp. 642–647 (2004)

    Google Scholar 

  13. Lee, J.C., Tan, D.S.: Using a low-cost electroencephalograph for task classification in HCI research. In: Proceedings of the 19th ACM Symposium on User Interface Software and Technology, pp. 81–90 (2006)

    Google Scholar 

  14. Reid, G.B., Nygren, T.E.: The subjective workload assessment technique: a scaling procedure for measuring mental workload, vol. 52, pp. 185–218. North-Holland (1988)

    Google Scholar 

  15. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. In: Advances in Psychology, vol. 52, pp. 139–183 (1988)

    Google Scholar 

  16. Kramer, A.F.: Physiological metrics of mental workload: a review of recent progress. Multiple-task performance. Taylor & Francis, 279–328 (1991)

    Google Scholar 

  17. Wickens, C.D.: Multiple resources and mental workload. Hum. Factors 50, 449–454 (2008)

    Article  Google Scholar 

  18. Wickens, C.D., Hollands, J.G.: Engineering Psychology and Human Performance, 3rd edn. Prentice Hall, Upper Saddle River (1999)

    Google Scholar 

  19. Tsang, P.S., Velazquez, V.L.: Diagnosticity and multidimensional subjective workload ratings. Ergonomics 39(3), 358–381 (1996)

    Article  Google Scholar 

  20. Brooke, J.: SUS-A quick and dirty usability scale. Usability Eval. Ind. 189(194), 4–7 (1996)

    Google Scholar 

  21. Azevedo, A.I R.L., Santos, M.F.: KDD, SEMMA and CRISP-DM: a parallel overview. IADS-DM (2008)

    Google Scholar 

  22. Longo, L., Dondio, P.: On the relationship between perception of usability and subjective mental workload of web interfaces. In: 2015 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), pp. 345–352. IEEE (2015)

    Google Scholar 

  23. Liang, N.Y., Saratchandran, P., Huang, G.B., Sundararajan, N.: Classification of mental tasks from EEG signals using extreme learning machine. Int. J. Neural Syst. 16(01), 29–38 (2006)

    Article  Google Scholar 

  24. Müller, K.R., Tangermann, M., Dornhege, G., Krauledat, M., Curio, G., Blankertz, B.: Machine learning for real-time single-trial EEG-analysis: from brain–computer interfacing to mental state monitoring. J. Neurosci. Methods 167(1), 82–90 (2008)

    Article  Google Scholar 

  25. Yin, Z., Zhang, J.: Identification of temporal variations in mental workload using locally-linear-embedding-based EEG feature reduction and support-vector-machine-based clustering and classification techniques. Comput. Methods Programs Biomed. 115, 119–134 (2014)

    Article  Google Scholar 

  26. Zhang, J., Yin, Z., Wang, R.: Recognition of mental workload levels under complex human–machine collaboration by using physiological features and adaptive support vector machines. IEEE Trans. Hum.-Mach. Syst. 45(2), 200–214 (2014)

    Article  Google Scholar 

  27. Yin, Z., Zhang, J., Wang, R.: Neurophysiological feature-based detection of mental workload by ensemble support vector machines. In: Wang, R., Pan, X. (eds.) Advances in Cognitive Neurodynamics (V). ACN, pp. 469–475. Springer, Singapore (2016). https://doi.org/10.1007/978-981-10-0207-6_64

    Chapter  Google Scholar 

  28. Rubio, S., Díaz, E., Martín, J., Puente, J.: Evaluation of subjective mental workload: a comparison of SWAT, NASA-TLX, and workload profile methods. Appl. Psychol. 53, 61–86 (2004)

    Article  Google Scholar 

  29. Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human–robot interaction. Pattern Anal. Appl. 9, 58–69 (2006)

    Article  Google Scholar 

  30. Smith, K.T.: Observations and issues in the application of cognitive workload modelling for decision making in complex time-critical environments. In: Longo, L., Leva, M.Chiara (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 77–89. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_5

    Chapter  Google Scholar 

  31. Balfe, N., Crowley, K., Smith, B., Longo, L.: Estimation of train driver workload: extracting taskload measures from on-train-data-recorders. In: Longo, L., Leva, M. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 106–119. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_7

    Chapter  Google Scholar 

  32. Cahill, J., et al.: Adaptive automation and the third pilot: managing teamwork and workload in an airline cockpit. In: Longo, L., Leva, M. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 161–173. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_10

    Chapter  Google Scholar 

  33. Delamare, L., Golightly, D., Goswell, G., Treble, P.: Quantification of rail signaller demand through simulation. In: Longo, L., Leva, M.Chiara (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 174–186. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_11

    Chapter  Google Scholar 

  34. Byrne, A.: Mental workload as an outcome in medical education. In: Longo, L., Leva, M.Chiara (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 187–197. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_12

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. South East European University, Tetovo, 1200, North Macedonia

    Bujar Raufi

Authors
  1. Bujar Raufi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bujar Raufi .

Editor information

Editors and Affiliations

  1. Technological University Dublin, Dublin, Ireland

    Luca Longo

  2. Technological University Dublin, Dublin, Ireland

    Maria Chiara Leva

Appendix: Usability and Mental Workload Questionnaires and Independent Feature Descriptors

Appendix: Usability and Mental Workload Questionnaires and Independent Feature Descriptors

.

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Raufi, B. (2019). Hybrid Models of Performance Using Mental Workload and Usability Features via Supervised Machine Learning. In: Longo, L., Leva, M. (eds) Human Mental Workload: Models and Applications. H-WORKLOAD 2019. Communications in Computer and Information Science, vol 1107. Springer, Cham. https://doi.org/10.1007/978-3-030-32423-0_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-32423-0_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-32422-3

  • Online ISBN: 978-3-030-32423-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation