Skip to main content
SpringerLink
Log in

Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

This paper examines the psychophysiological effects of mental workload in single-task and dual-task human-computer interaction. A mental arithmetic task and a manual error correction task were performed both separately and concurrently on a computer using verbal and haptic input devices. Heart rate, skin conductance, respiration and peripheral skin temperature were recorded in addition to objective performance measures and self-report questionnaires. Analysis of psychophysiological responses found significant changes from baseline for both single-task and dual-task conditions. There were also significant psychophysiological differences between the mental arithmetic task and the manual error correction task, but no differences in questionnaire results. Additionally, there was no significant psychophysiological difference between performing only the mental arithmetic task and performing both tasks at once. These findings suggest that psychophysiological measures respond differently to different types of tasks and that they do not always agree with performance or with participants’ subjective feelings.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Backs RW (1997) Psychophysiological aspects of selective and divided attention during continuous manual tracking. Acta Psychol 96:167–191

    Article  Google Scholar 

  2. Backs RW, Lenneman JK, Wetzel JM, Green P (2003) Cardiac measures of driver workload during simulated driving with and without visual occlusion. Hum Factors 45:525–538

    Article  Google Scholar 

  3. Boiten F (1993) Component analysis of task-related respiratory patterns. Int J Psychophysiol 15:91–104

    Article  Google Scholar 

  4. Chen JYC (2009) Concurrent performance of military and robotics tasks and effects of cueing in a simulated multi-tasking environment. Presence–Teleop Virt 18: 1–15

    Google Scholar 

  5. Collet C, Averty P, Dittmar A (2009) Autonomic nervous system and subjective ratings of strain in air traffic control. Appl Ergon 40:23–32

    Article  Google Scholar 

  6. Ćosić K, Popović S, Jovanovic T, Kukolja D, Slamić M (2007) Physiology-driven adaptive VR system: Technology and rationale for PTSD treatment. Annual Review of CyberTherapy & Telemedicine 5:179–191

    Google Scholar 

  7. Derrick WL (1988) Dimensions of operator workload. Hum Factors 30:95–110

    Google Scholar 

  8. Detenber BH, Simons RF, Bennett GG (1998) Roll 'em!: the effects of picture motion on emotional responses. J Broadcast Electron 42:113–127

    Article  Google Scholar 

  9. Feldman PJ, Cohen S, Lepore SJ, Matthews KA, Kamarck TW, Marsland AL (1999) Negative emotions and acute physiological responses to stress. Ann Behav Med 21:216–222

    Article  Google Scholar 

  10. Gwizdka J (2010) Distribution of cognitive load in web search. J Am Soc Inf Sci Technol, in press

  11. Horrey WJ, Lesch MF, Garabet A (2009) Dissociation between driving performance and drivers' subjective estimates of performance and workload in dual-task conditions. J Safety Res 40:7–12

    Article  Google Scholar 

  12. Kahneman D (1973) Attention and Effort. Prentice Hall, Englewood Cliffs

    Google Scholar 

  13. Kahol K, Panchanathan S (2008) Neuro-cognitively inspired haptic user interfaces. Multimed Tools Appl 37:15–38

    Article  Google Scholar 

  14. Liu C, Agrawal P, Sarkar N, Chen S (2009) Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. Int J Hum-Comput Int 25:506–529

    Article  Google Scholar 

  15. Liu J, Gwizdka J, Liu C, Belkin NJ (2010) Predicting task difficulty for different task types. In: Proceedings of the 73rd Annual Meeting of the American Society for Information Science & Technology. Pittsburgh, PA, in press

  16. Martinez JM, Kent JM, Coplan JD, Browne ST, Papp LA, Sullivan GM et al (2001) Respiratory variability in panic disorder. Depress Anxiety 14:232–237

    Article  Google Scholar 

  17. Mulder G, Mulder LJ, Meijman TF, Veldman JB, van Roon AM (2000) A psychophysiological approach to working conditions. In: Backs R, Boucsein W (eds) Engineering psychophysiology: Issues and applications. Lawrence Erlbaum Associates, Mahwah, pp 139–159

    Google Scholar 

  18. Nikula R (1991) Psychological correlates of nonspecific skin conductance responses. Psychophysiology 28:86–90

    Article  Google Scholar 

  19. Novak D, Mihelj M, Munih M (2010) Psychophysiological responses to different levels of cognitive and physical load in haptic interaction. Robotica, in press

  20. Octavia JR, Raymaekers C, Coninx K (2010) Adaptation in virtual environments: conceptual framework and user models. Multimed Tools Appl, in press

  21. Ohsuga M, Shimono F, Genno H (2001) Assessment of phasic work stress using autonomic indices. Int J Psychophysiol 40:211–220

    Article  Google Scholar 

  22. Parasuraman R, Cosenzo KA, De Visser E (2009) Adaptive automation for human supervision of multiple uninhabited vehicles: Effects on change detection, situation awareness and mental workload. Mil Psychol 21:270–297

    Article  Google Scholar 

  23. Parkes AM, Coleman N (1990) Route guidance systems: a comparison of methods of presenting directional information to the driver. In: Lovesey J (ed) Contemporary Ergonomics 1990. Taylor & Francis, London, pp 480–485

    Google Scholar 

  24. Pashler H, Johnston JC (1998) Attentional limitations in dual-task performance. In: Pashler H (ed) Attention. Psychology Press, East Sussex, pp 155–189

    Google Scholar 

  25. Phillips AC, Carroll D, Hunt K, Der G (2006) The effects of the spontaneous presence of a spouse/partner and others on cardiovascular reactions to an acute psychological challenge. Psychophysiology 43:633–640

    Article  Google Scholar 

  26. Posner MI, Sandson J, Dhawan M, Shulman GL (1989) Is word recognition automatic? A cognitive anatomical approach. J Cogn Neurosci 1:50–60

    Article  Google Scholar 

  27. Ritvanen T, Louhevaara V, Helin P, Väisänen S, Hänninen O (2006) Responses of the autonomic nervous system during periods of perceived high and low work stress in younger and older female teachers. Appl Ergon 37:311–318

    Article  Google Scholar 

  28. Schwerdtfeger A (2004) Predicting autonomic reactivity to public speaking: don't get fixed on self-report data! Int J Psychophysiol 52:217–224

    Article  Google Scholar 

  29. Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology (1996) Heart rate variability: Standards of measurement, physiological interpretation, and clinical use. Eur Heart J 17:354–381

    Google Scholar 

  30. Veltman JA, Gaillard AW (1998) Physiological workload reactions to increasing levels of task difficulty. Ergonomics 41:656–669

    Article  Google Scholar 

  31. Wester AE, Böcker KB, Volkerts ER, Verster JC, Kenemans JL (2008) Event-related potentials and secondary task performance during simulated driving. Accident Anal Prev 40:1–7

    Article  Google Scholar 

  32. Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon Sci 3:159–177

    Article  Google Scholar 

  33. Wilson GF, Russell CA (2003) Operator functional state classification using multiple psychophysiological features in an air traffic control task. Hum Factors 45:381–389

    Article  Google Scholar 

  34. Xie B, Salvendy G (2000) Review and reappraisal of modelling and predicting mental workload in single- and multi-task environments. Work Stress 14:74–99

    Article  Google Scholar 

  35. Zhang T, Kaber D, Hsiang S (2010) Characterisation of mental models in a virtual reality-based multitasking scenario using measures of situation awareness. Theor Issues Ergon Sci 11:99–118

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the volunteers who participated in this study. The work was funded by the EU Information and Communication Technologies Collaborative Project MIMICS grant 215756 and additionally supported by the Slovenian Research Agency.

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering, University of Ljubljana, Tržaska 25, 1000, Ljubljana, Slovenia

    Domen Novak, Matjaž Mihelj & Marko Munih

Authors
  1. Domen Novak
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Matjaž Mihelj
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Marko Munih
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Domen Novak.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Novak, D., Mihelj, M. & Munih, M. Dual-task performance in multimodal human-computer interaction: a psychophysiological perspective. Multimed Tools Appl 56, 553–567 (2012). https://doi.org/10.1007/s11042-010-0619-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-010-0619-7

Keywords

Search

Navigation