Skip to main content
SpringerLink
Log in

Workload Assessment Using Speech-Related Neck Surface Electromyography

  • Conference paper
  • First Online:
Human Mental Workload: Models and Applications (H-WORKLOAD 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1012))

Abstract

This paper presents preliminary findings of an ongoing effort to evaluate the application of face and neck surface electromyography (sEMG) to real-time cognitive workload assessment. A retrospective analysis of anterior neck sEMG signals, recorded from 10 subjects during a time-pressured mental arithmetic task with verbal responses during a previous study by Stepp et al. [52], suggests that a measure known as neck intermuscular beta coherence (NIBcoh) may be sensitive to cognitive workload and/or error commission in tasks involving speech production, with sub-second temporal resolution. Specifically, the recent reanalysis indicates that subjects exhibited significantly lower NIBcoh when they produced incorrect verbal responses as compared to NIBcoh associated with correct responses. We discuss this promising application of NIBcoh within the context of our continuing research program and introduce future experiments that will shed light on the relationships among face and neck sEMG signals, task demands, performance, cognitive effort/strain, subjective workload measures, and other psychophysiological measures.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Balfe, N., Crowley, K., Smith, B., Longo, L.: Estimation of train driver workload: extracting taskload measures from on-train-data-recorders. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 106–119. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_7

    Chapter  Google Scholar 

  2. Beatty, J.: Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychol. Bull. 91(2), 276–292 (1982). https://doi.org/10.1037/0033-2909.91.2.276

    Article  Google Scholar 

  3. Betts, B., Binsted, K., Jorgensen, C.: Small-vocabulary speech recognition using surface electromyography. Interact. Comput. 18(6), 1242–1259 (2006). https://doi.org/10.1016/j.intcom.2006.08.012

    Article  Google Scholar 

  4. Borghini, G., et al.: Assessment of mental fatigue during car driving by using high resolution EEG activity and neurophysiologic indices. In: Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society, EMBS, pp. 6442–6445 (2012). https://doi.org/10.1109/EMBC.2012.6347469

  5. Brookings, J.B., Wilson, G.F., Swain, C.R.: Psychophysiological responses to changes in workload during simulated air traffic control. Biol. Psychol. 42, 361–377 (1996). https://doi.org/10.1016/0301-0511(95)05167-8

    Article  Google Scholar 

  6. Brown, P., Farmer, S.F., Halliday, D.M., Marsden, J., Rosenberg, J.R.: Coherent cortical and muscle discharge in cortical myoclonus. Brain 122(3), 461–472 (1999). https://doi.org/10.1093/brain/122.3.461

    Article  Google Scholar 

  7. Byrne, A.: Mental workload as an outcome in medical education. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 187–197. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_12

    Chapter  Google Scholar 

  8. Cahill, J., et al.: Adaptive automation and the third pilot. In: Aircraft Technology. IntechOpen (2018)

    Google Scholar 

  9. Cain, B.: A review of the mental workload literature. In: Defence Research and Development Toronto (Canada), pp. 4-1–4-34 (2007). http://www.dtic.mil/cgi-bin/GetTRDoc?Location=U2&doc=GetTRDoc.pdf&AD=ADA474193

  10. Chan, A., Englehart, K., Hudgins, B., Lovely, D.: Myo-electric signals to augment speech recognition. Med. Biol. Eng. Comput. 39(4), 500–504 (2001). https://doi.org/10.1007/BF02345373

    Article  Google Scholar 

  11. Cheng, B., Liu, G.: Emotion recognition from surface EMG signal using wavelet transform and neural network. In: Proceedings of The 2nd International Conference on Bioinformatics and Biomedical Engineering (ICBBE), pp. 1363–1366 (2008). https://doi.org/10.1109/ICBBE.2008.670

  12. De Joux, N., Russell, P.N., Helton, W.S.: A functional near-infrared spectroscopy study of sustained attention to local and global target features. Brain Cogn. 81(3), 370–375 (2013). https://doi.org/10.1016/j.bandc.2012.12.003

    Article  Google Scholar 

  13. Denby, B., Schultz, T., Honda, K., Hueber, T., Gilbert, J., Brumberg, J.: Silent speech interfaces. Speech Commun. 52(4), 270–287 (2010). https://doi.org/10.1016/j.specom.2009.08.002

    Article  Google Scholar 

  14. Dussault, C., Jouanin, J.C., Philippe, M., Guezennec, C.Y.: EEG and ECG changes during simulator operation reflect mental workload and vigilance. Aviat. Space Environ. Med. 76(4), 344–351 (2005)

    Google Scholar 

  15. Edwards, T., Martin, L., Bienert, N., Mercer, J.: The relationship between workload and performance in air traffic control: exploring the influence of levels of automation and variation in task demand. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 120–139. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_8

    Chapter  Google Scholar 

  16. Fairbanks, G.: The rainbow passage. In: Voice and Articulation Drillbook, vol. 2 (1960)

    Google Scholar 

  17. Fairclough, S.H., Venables, L., Tattersall, A.: The influence of task demand and learning on the psychophysiological response. Int. J. Psychophysiol. 56(2), 171–184 (2005). https://doi.org/10.1016/j.ijpsycho.2004.11.003

    Article  Google Scholar 

  18. Fallahi, M., Motamedzade, M., Heidarimoghadam, R., Soltanian, A.R., Miyake, S.: Effects of mental workload on physiological and subjective responses during traffic density monitoring: a field study. Appl. Ergon. 52, 95–103 (2016). https://doi.org/10.1016/j.apergo.2015.07.009

    Article  Google Scholar 

  19. Feige, B., Aertsen, A.D., Kristeva-Feige, R.: Dynamic synchronization between multiple cortical motor areas and muscle activity in phasic voluntary movements. J. Neurophysiol. 84(5), 2622–2629 (2000). https://doi.org/10.1152/jn.2000.84.5.2622

    Article  Google Scholar 

  20. Fraser, K., Ma, I., Teteris, E., Baxter, H., Wright, B., McLaughlin, K.: Emotion, cognitive load and learning outcomes during simulation training. Med. Educ. 46(11), 1055–1062 (2012). https://doi.org/10.1111/j.1365-2923.2012.04355.x

    Article  Google Scholar 

  21. Gevins, A., Smith, M.E.: Neurophysiological measures of cognitive workload during human-computer interaction. Theor. Issues Ergon. Sci. 4(1–2), 113–131 (2003). https://doi.org/10.1080/14639220210159717

    Article  Google Scholar 

  22. Goldman, S.L., Hargrave, J., Hillman, R.E., Holmberg, E., Gress, C.: Stress, anxiety, somatic complaints, and voice use in women with vocal nodules: preliminary findings. Am. J. Speech-Lang. Pathol. 5(1), 44–54 (1996). https://doi.org/10.1044/1058-0360.0501.44

    Article  Google Scholar 

  23. Grosse, P., Cassidy, M.J., Brown, P.: EEG-EMG. MEG-EMG and EMG-EMG frequency analysis: physiological principles and clinical applications (2002). https://doi.org/10.1016/S1388-2457(02)00223-7

    Article  Google Scholar 

  24. Halliday, D., Rosenberg, J., Amjad, A., Breeze, P., Conway, B., Farmer, S.: A framework for the analysis of mixed time series/point process data: theory and application to the study of physiological tremor, single motor unit discharges and electromyograms. Prog. Biophys. Mol. Biol. 64(2–3), 237–278 (1995)

    Article  Google Scholar 

  25. Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv. Psychol. 52(C), 139–183 (1988). https://doi.org/10.1016/S0166-4115(08)62386-9

    Article  Google Scholar 

  26. Horrey, W.J., Lesch, M.F., Garabet, A.: Dissociation between driving performance and drivers’ subjective estimates of performance and workload in dual-task conditions. J. Saf. Res. 40(1), 7–12 (2009). https://doi.org/10.1016/j.jsr.2008.10.011

    Article  Google Scholar 

  27. Jacob, R.J., Karn, K.S.: Eye tracking in human-computer interaction and usability research. Ready to deliver the promises. In: The Mind’s Eye: Cognitive and Applied Aspects of Eye Movement Research, pp. 531–553. Elsevier, New York (2003). https://doi.org/10.1016/B978-044451020-4/50031-1

    Chapter  Google Scholar 

  28. Johnson, A.N., Shinohara, M.: Corticomuscular coherence with and without additional task in the elderly. J. Appl. Physiol. 112(6), 970–981 (2012). https://doi.org/10.1152/japplphysiol.01079.2011

    Article  Google Scholar 

  29. Johnson, A.N., Wheaton, L.A., Shinohara, M.: Attenuation of corticomuscular coherence with additional motor or non-motor task. Clin. Neurophysiol. 122(2), 356–363 (2011). https://doi.org/10.1016/j.clinph.2010.06.021

    Article  Google Scholar 

  30. Jordan, C., Brennen, S.: Instantaneous self-assessment of workload technique (ISA). Defence Research Agency, Portsmouth (1992)

    Google Scholar 

  31. Jou, S., Schultz, T., Walliczek, M., Kraft, F., Waibel, A.: Towards continuous speech recognition using surface electromyography. In: Ninth International Conference on Spoken Language Processing (2006)

    Google Scholar 

  32. Kamzanova, A.T., Kustubayeva, A.M., Matthews, G.: Use of EEG workload indices for diagnostic monitoring of vigilance decrement. Hum. Factors 56(6), 1136–1149 (2014). https://doi.org/10.1177/0018720814526617

    Article  Google Scholar 

  33. Kilner, J.M., Baker, S.N., Salenius, S., Jousmäki, V., Hari, R., Lemon, R.N.: Task-dependent modulation of 15–30 Hz coherence between rectified EMGs from human hand and forearm muscles. J. Physiol. 516(2), 559–570 (1999). https://doi.org/10.1111/j.1469-7793.1999.0559v.x

    Article  Google Scholar 

  34. Kristeva-Feige, R., Fritsch, C., Timmer, J., Lücking, C.H.: Effects of attention and precision of exerted force on beta range EEG-EMG synchronization during a maintained motor contraction task. Clin. Neurophysiol. 113(1), 124–131 (2002). https://doi.org/10.1016/S1388-2457(01)00722-2

    Article  Google Scholar 

  35. Lindström, B.R., Mattsson-Mårn, I.B., Golkar, A., Olsson, A.: In your face: risk of punishment enhances cognitive control and error-related activity in the corrugator supercilii muscle. PLOS One 8(6), e65692 (2013). https://doi.org/10.1371/journal.pone.0065692

    Article  Google Scholar 

  36. Longo, L.: Subjective usability, mental workload assessments and their impact on objective human performance. In: Bernhaupt, R., Dalvi, G., Joshi, A., Balkrishan, D.K., O’Neill, J., Winckler, M. (eds.) INTERACT 2017. LNCS, vol. 10514, pp. 202–223. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67684-5_13

    Chapter  Google Scholar 

  37. Marshall, S.P.: The index of cognitive activity: measuring cognitive workload. In: Proceedings of the 2002 IEEE 7th Conference on Human Factors and Power Plants, pp. 7–9 (2002). https://doi.org/10.1109/HFPP.2002.1042860

  38. Matthews, G., Reinerman-Jones, L.E., Barber, D.J., Abich, J.: The psychometrics of mental workload. Hum. Factors: J. Hum. Factors Ergon. Soc. 57(1), 125–143 (2015). https://doi.org/10.1177/0018720814539505

    Article  Google Scholar 

  39. Matthews, G., Warm, J.S., Reinerman-Jones, L.E., Langheim, L.K., Washburn, D.A., Tripp, L.: Task engagement, cerebral blood flow velocity, and diagnostic monitoring for sustained attention. J. Exp. Psychol.: Appl. 16(2), 187–203 (2010). https://doi.org/10.1037/a0019572

    Article  Google Scholar 

  40. McCracken, J., Aldrich, T.B.T.: Analyses of Selected LHX Mission Functions: Implications for Operator Workload and System Automation Goals. Technical report, Anacapa Sciences Inc., Fort Rucker, AL (1984)

    Google Scholar 

  41. Meltzner, G.S., Heaton, J.T., Deng, Y., De Luca, G., Roy, S.H., Kline, J.C.: Development of sEMG sensors and algorithms for silent speech recognition. J. Neural Eng. 15(4), 046031 (2018). https://iopscience.iop.org/article/10.1088/1741-2552/aac965/meta

    Article  Google Scholar 

  42. Meltzner, G.S., Heaton, J.T., Deng, Y., De Luca, G., Roy, S.H., Kline, J.C.: Silent speech recognition as an alternative communication device for persons with laryngectomy. IEEE/ACM Trans. Audio Speech Lang. Process. 25(12), 2386–2398 (2017). https://doi.org/10.1109/TASLP.2017.2740000

    Article  Google Scholar 

  43. Mijović, P., Milovanović, M., Ković, V., Gligorijević, I., Mijović, B., Mačužić, I.: Neuroergonomics method for measuring the influence of mental workload modulation on cognitive state of manual assembly worker. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 213–224. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_14

    Chapter  Google Scholar 

  44. Mulder, L., De Waard, D., Brookhuis, K.A.: Estimating mental effort using heart rate and heart rate variability. In: Handbook of Human Factors and Ergonomics Methods, pp. 1–20 (2004)

    Google Scholar 

  45. Neto, O.P., Christou, E.A.: Rectification of the EMG signal impairs the identification of oscillatory input to the muscle. J. Neurophysiol. 103(2), 1093–103 (2010). https://doi.org/10.1152/jn.00792.2009

    Article  Google Scholar 

  46. O’Donnell, R., Eggemeier, F.: Workload assessment methodology. In: Boff, K.R., Kaufman, L., Thomas, J.P. (eds.) Handbook of Perception and Human Performance, Cognitive Processes and Performance, vol. 2. Wiley, Hoboken (1986)

    Google Scholar 

  47. Pickup, L., Wilson, J.R., Sharpies, S., Norris, B., Clarke, T., Young, M.S.: Fundamental examination of mental workload in the rail industry. Theor. Issues Ergon. Sci. 6(6), 463–482 (2005). https://doi.org/10.1080/14639220500078021

    Article  Google Scholar 

  48. Reid, G.B., Nygren, T.E.: The Subjective Workload Assessment Technique: A Scaling Procedure for Measuring Mental Workload, North-Holland, vol. 52 (1988). https://doi.org/10.1016/S0166-4115(08)62387-0

    Google Scholar 

  49. Santiago-Espada, Y., Myer, R., Latorella, K., Comstock, J.: The Multi-Attribute Task Battery II (MATB-II) Software for Human Performance and Workload Research: A User’s Guide. Technical report, NASA, Chicago (2011)

    Google Scholar 

  50. Smith, A.P., Smith, H.N.: Workload, fatigue and performance in the rail industry. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 251–263. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_17

    Chapter  Google Scholar 

  51. Stepp, C.E., Hillman, R.E., Heaton, J.T.: Use of neck strap muscle intermuscular coherence as an indicator of vocal hyperfunction. IEEE Trans. Neural Syst. Rehabil. Eng. 18(3), 329–335 (2010). https://doi.org/10.1109/TNSRE.2009.2039605

    Article  Google Scholar 

  52. Stepp, C.E., Hillman, R.E., Heaton, J.T.: Modulation of neck intermuscular beta coherence during voice and speech production. J. Speech Lang. Hear. Res. 54(3), 836–844 (2011). https://doi.org/10.1044/1092-4388(2010/10-0139)

    Article  Google Scholar 

  53. Stroobant, N., Vingerhoets, G.: Transcranial Doppler ultrasonography monitoring of cerebral hemodynamics during performance of cognitive tasks: a review (2000). https://doi.org/10.1023/A:1026412811036

    Article  Google Scholar 

  54. Tassinary, L.G., Cacioppo, J.T.: Unobservable facial actions and emotion. Psychol. Sci. 3(1), 28–33 (1992). https://doi.org/10.1111/j.1467-9280.1992.tb00252.x

    Article  Google Scholar 

  55. Tong, S., Helman, S., Balfe, N., Fowler, C., Delmonte, E., Hutchins, R.: Workload differences between on-road and off-road manoeuvres for motorcyclists. In: Longo, L., Leva, M.C. (eds.) H-WORKLOAD 2017. CCIS, vol. 726, pp. 239–250. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-61061-0_16

    Chapter  Google Scholar 

  56. Tsang, P.S., Velazquez, V.L.: Diagnosticity and multidimensional subjective workload ratings. Ergonomics 39(3), 358–381 (1996). https://doi.org/10.1080/00140139608964470

    Article  Google Scholar 

  57. Van Boxtel, A., Goudswaard, P., Van der Molen, G., Van Den Bosch, W.: Changes in electromyogram power spectra of facial and jaw-elevator muscles during fatigue. J. Appl. Physiol. 54(1), 51–58 (1983). https://doi.org/10.1152/jappl.1983.54.1.51

    Article  Google Scholar 

  58. Van Boxtel, A.: Facial EMG as a tool for inferring affective states. In: Proceedings of Measuring Behavior, pp. 104–108. Noldus Information Technology Wageningen (2010)

    Google Scholar 

  59. Vanneste, S.: Timing in aging: the role of attention. Exp. Aging Res. 25(1), 49–67 (1999). https://doi.org/10.1080/036107399244138

    Article  Google Scholar 

  60. Voelcker-Rehage, C., Stronge, A.J., Alberts, J.L.: Age-related differences in working memory and force control under dual-task conditions. Aging, Neuropsychol. Cogn. 13(3–4), 366–384 (2006). https://doi.org/10.1080/138255890969339

    Article  Google Scholar 

  61. Warm, J.S., Tripp, L.D., Matthews, G., Helton, W.S.: Cerebral hemodynamic indices of operator fatigue in vigilance. In: Handbook of Operator Fatigue, pp. 197–207 (2012)

    Chapter  Google Scholar 

  62. Welch, P.: The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Trans. Audio Electroacoust. 15(2), 70–73 (1967). https://doi.org/10.1109/TAU.1967.1161901

    Article  Google Scholar 

  63. Wickens, C.D.: Multiple resources and mental workload. Hum. Factors 50(3), 449–455 (2008). https://doi.org/10.1518/001872008X288394

    Article  Google Scholar 

  64. Yeh, Y.Y., Wickens, C.D.: Dissociation of performance and subjective measures of workload. Hum. Factors: J. Hum. Factors Ergon. Soc. 30(1), 111–120 (1988). https://doi.org/10.1177/001872088803000110

    Article  Google Scholar 

  65. Zhang, J.Y., Liu, S.L., Feng, Q.M., Gao, J.Q., Zhang, Q.: Correlative evaluation of mental and physical workload of laparoscopic surgeons based on surface electromyography and eye-tracking signals. Sci. Rep. 7(1), 11095 (2017). https://doi.org/10.1038/s41598-017-11584-4

    Article  Google Scholar 

Download references

Acknowledgements

This material is based upon work supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. D17PC00119 (Distribution Statement: approved for public release; distribution unlimited). The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government.

Author information

Authors and Affiliations

  1. Stottler Henke Associates, Inc., Seattle, WA, USA

    Aaron Novstrup & Terrance Goan

  2. Massachusetts General Hospital, Center for Laryngeal Surgery and Voice Rehabilitation, Boston, MA, USA

    James Heaton

Authors
  1. Aaron Novstrup
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Terrance Goan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. James Heaton
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Aaron Novstrup .

Editor information

Editors and Affiliations

  1. Dublin Institute of Technology, Dublin, Ireland

    Luca Longo

  2. Dublin Institute of Technology, Dublin, Ireland

    M. Chiara Leva

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Novstrup, A., Goan, T., Heaton, J. (2019). Workload Assessment Using Speech-Related Neck Surface Electromyography. In: Longo, L., Leva, M. (eds) Human Mental Workload: Models and Applications. H-WORKLOAD 2018. Communications in Computer and Information Science, vol 1012. Springer, Cham. https://doi.org/10.1007/978-3-030-14273-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-14273-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-14272-8

  • Online ISBN: 978-3-030-14273-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation