Abstract
In this study, we argue that multimodal presentations should be created in a cognitive-aware manner, especially in a high-load HCI situation where the user task challenges the full capacity of the human cognition. An experiment was conducted to investigate the cognitive effects of modalities, using a high-load task. The performance measurements and subjective reports consistently confirm a significant modality impact on cognitive workload, stress and performance. A relation between modality usage and physiological states was not found, due to the insufficient sensitivity and individual differences of the physiological measurements. The findings of this experiment can be well explained by several modality-related cognitive theories. We further integrate these theories into a suitability prediction model, which can systematically predict how suitable a certain modality usage is for this presentation task. The model demonstrates a possible approach towards cognitive-aware modality planning and can be modified for other applications.
Chapter PDF
Similar content being viewed by others
Keywords
References
Andre, E.: The Generation of Multimedia Presentations. In: Dale, R., Somers, H.L., Moisl, H. (eds.) Handbook of Natural Language Processing. Marcel Dekker, Inc., USA (2000)
Bachvarova, Y., van Dijk, B., Nijholt, A.: Towards a Unified Knowledge-Based Approach to Modality Choice. In: Multimodal Output Generation (MOG), pp. 7–15 (2007)
Baddeley, A.D.: Essentials of Human Memory. Psychology Press, USA (1999)
Baddeley, A.D., Hitch, G.J.: Working Memory. The Psychology of Learning and Motivation: Advances in Research and Theory 8, 47–89 (1974)
Boucsein, W., Haarmann, A., Schaefer, F.: Combining Skin Conductance and Heart Rate Variability for Adaptive Automation During Simulated IFR Flight. In: Harris, D. (ed.) HCII 2007 and EPCE 2007. LNCS, vol. 4562, pp. 639–647. Springer, Heidelberg (2007)
Cao, Y., Theune, M., Nijholt, A.: Modality Effects on Cognitive Load and Performance in High-Load Information Presentation. In: Intelligent User Interface (IUI), pp. 335–344 (2009)
Carr, T.H., McCauley, C., Sperber, R.D., Parmelee, C.M.: Words, Pictures, and Priming: On Semantic Activation, Conscious Identification, and the Automaticity of Information Processing. J. Exp. Psychol. Hum. Percept. Perform. 8, 757–777 (1982)
Clark, J.M., Paivio, A.: Dual Coding Theory and Education. Educational Psychology Review 3, 149–210 (1991)
Hart, S.G., Staveland, L.E.: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Human Mental Workload 1, 139–183 (1988)
Kramer, A.F.: Physiological Metrics of Mental Workload: A Review of Recent Progress. In: Damos, D.L. (ed.) Multiple-Task Performance. CRC Press, USA (1991)
Malik, M.: Heart Rate Variability - Standards of Measurement, Physiological Interpretation, and Clinical Use. Circulation 93, 1043–1065 (1996)
Mayer, R.E., Moreno, R.: Nine Ways to Reduce Cognitive Load in Multimedia Learning. Educational Psychologist 38, 45–52 (2003)
Paivio, A.: Mental Representations: A Dual Coding Approach. Oxford University Press, USA (1986)
Potter, M.C., Faulconer, B.A.: Time to Understand Pictures and Words. Nature 253, 437–438 (1975)
Scerbo, M.W., Freeman, F.G., Mikulka, P.J., Parasuraman, R., Di Nocero, F.: The Efficacy of Psychophysiological Measures for Implementing Adaptive Technology. TP-2001-211018, NASA Langley Research Center, Hampton (2001)
Stern, R.M., Ray, W.J., Quigley, K.S.: Psychophysiological Recording, 2nd edn. Oxford University Press, UK (2001)
Verwey, W.B., Veltman, H.A.: Detecting Short Periods of Elevated Workload: A Comparison of Nine Workload Assessment Techniques. Applied Experimental Psychology 2, 270–285 (1996)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2009 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Cao, Y., Theune, M., Nijholt, A. (2009). Towards Cognitive-Aware Multimodal Presentation: The Modality Effects in High-Load HCI. In: Harris, D. (eds) Engineering Psychology and Cognitive Ergonomics. EPCE 2009. Lecture Notes in Computer Science(), vol 5639. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-02728-4_1
Download citation
DOI: https://doi.org/10.1007/978-3-642-02728-4_1
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-02727-7
Online ISBN: 978-3-642-02728-4
eBook Packages: Computer ScienceComputer Science (R0)