Abstract
Students experience challenges when understanding visual information in multimedia learning. Specifically, immersive multimedia environments, such as virtual reality increase the likelihood that students undergo distractions in which information seeking during system-paced instruction occurred. Although previous studies have reviewed various cue designs to yield students’ higher attention, skepticism still exists regarding which ways cue designs can support their learning comprehension in video instruction. For this study, we sampled a total of 64 undergraduates in a university. Using video instruction performed by an animated pedagogical agent (APA), this study examined the effect of social (i.e., an APA’s conversational gestures) and cognitive (i.e., visual cue) cues on students’ learning comprehension and eye-gaze data within types of visual information (text and pictorial). Also, this study investigated how both cues promoted students’ cognitive load overall. Specific to text information processing, the results of the study confirmed that the negative prime effect of social cues undermined students’ learning comprehension and increased their cognitive load, whereas cognitive cues appeared to be supportive in video instruction. Also, this study found that students’ different visual-attention patterns appeared in pictorial information processing. In terms of pictorial information processing, the study finding implies that whereas social cues caused visual distractions and lowered learning comprehension, cognitive cues as visual cues helped learners to integrate pictorial information via visuospatial clues. Conclusively, we reported several design implications derived from the study findings.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adcock, A. B. (2004). The interaction of learner expertise and instructional role of a pedagogical agent on learner perception of agent, perceived cognitive load and task performance. (Doctor), The University of Memphis.
Amadieu, F., Mariné, C., & Laimay, C. (2011). The attention-guiding effect and cognitive load in the comprehension of animations. Computers in Human Behavior, 27(1), 36–40. https://doi.org/10.1016/j.chb.2010.05.009.
Atkinson, R. C., & Shiffrin, R. M. (1968). Human memory: A proposed system and its control processes. In K. W. Spence & J. T. Spence (Eds.), The psychology of learning and motivation: Advances in research and theory (Vol. 2). New York: Academic Press.
Atkinson, R. K. (2002). Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, 94(2), 416–427. https://doi.org/10.1037/0022-0663.94.2.416.
Atkinson, R. K., Mayer, R. E., & Merrill, M. M. (2005). Fostering social agency in multimedia learning: Examining the impact of an animated agent’s voice. Contemporary Educational Psychology, 30(1), 117–139. https://doi.org/10.1016/j.cedpsych.2004.07.001.
Ayres, P., & Paas, F. (2007). Making instructional animations more effective: A cognitive load approach. Applied Cognitive Psychology, 21(6), 695–700.
Baylor, A. L. (2011). The design of motivational agents and avatars. Educational Technology Research and Development, 59(2), 291–300. https://doi.org/10.1007/s11423-011-9196-3.
Baylor, A. L., & Kim, S. (2009). Designing nonverbal communication for pedagogical agents: When less is more. Computers in Human Behavior, 25(2), 450–457. https://doi.org/10.1016/j.chb.2008.10.008.
Baylor, A. L., & Kim, Y. (2004). Pedagogical agent design: The impact of agent realism, gender, ethnicity, and instructional role. In J. C. Lester, R. M. Vicari, & F. Paraguaçu (Eds.), Intelligent tutoring systems. ITS 2004. Lecture Notes in Computer Science (Vol. 3220). Berlin/Heidelberg: Springer.
Bickmore, T., & Cassell, J. (2005). Social dialogue with embodied conversational agents. In Advances in natural multimodal dialogue systems (pp. 23–54). Dordrecht: Springer.
Cassell, J., Pelachaud, C., Badler, N., Steedman, M., Achorn, B., Becket, T., et al. (1994). Animated conversation: Rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents. Paper presented at the proceedings of the 21st annual conference on computer graphics and interactive techniques
Castellano, G., Mancini, M., Peters, C., & McOwan, P. W. (2011). Expressive copying behavior for social agents: A perceptual analysis. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 42(3), 776–783. https://doi.org/10.1109/TSMCA.2011.2172415.
Choi, S., & Clark, R. E. (2006). Cognitive and affective benefits of an animated pedagogical agent for learning English as a second language. Journal of Educational Computing Research, 34(4), 441–466. https://doi.org/10.2190/A064-U776-4208-N145.
Cooke, L. (2005). Eye tracking: How it works and how it relates to usability. Technical Communication, 52(4), 456–463. https://doi.org/10.1016/j.jlp.2015.07.001.
Davis, R. O. (2018). The impact of pedagogical agent gesturing in multimedia learning environments: A meta-analysis. Educational Research Review., 24, 193–209. https://doi.org/10.1016/j.edurev.2018.05.002.
de Koning, B., & Tabbers, H. (2011). Facilitating understanding of movements in dynamic visualizations: An embodied perspective. Educational Psychology Review, 23(4), 501–521. https://doi.org/10.1007/s10648-011-9173-8.
de Koning, B. B., Tabbers, H. K., Rikers, R. M., & Paas, F. (2007). Attention cueing as a means to enhance learning from an animation. Applied Cognitive Psychology, 21(6), 731–746. https://doi.org/10.1002/acp.1346.
Dinçer, S., & Doğanay, A. (2017). The effects of multiple-pedagogical agents on learners’ academic success, motivation, and cognitive load. Computers & Education, 111, 74–100. https://doi.org/10.1016/j.compedu.2017.04.005.
Dunsworth, Q., & Atkinson, R. K. (2007). Fostering multimedia learning of science: Exploring the role of an animated agent’s image. Computers & Education, 49(3), 677–690. https://doi.org/10.1016/j.compedu.2005.11.010.
Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., & Scherer, K. (2011). Toward a minimal representation of affective gestures. IEEE Transactions on Affective Computing, 2(2), 106–118. https://doi.org/10.1109/T-AFFC.2011.7.
Huang, H. M., Rauch, U., & Liaw, S. S. (2010). Investigating learners’ attitudes toward virtual reality learning environments: Based on a constructivist approach. Computers & Education, 55(3), 1171–1182. https://doi.org/10.1016/j.compedu.2010.05.014.
Huk, T. (2006). Who benefits from learning with 3D models? The case of spatial ability. Journal of Computer Assisted Learning, 22(6), 392–404. https://doi.org/10.1111/j.1365-2729.2006.00180.x.
Jamet, E. (2014). An eye-tracking study of cueing effects in multimedia learning. Computers in Human Behavior, 32, 47–53. https://doi.org/10.1016/j.chb.2013.11.013.
Jarodzka, H., Scheiter, K., Gerjets, P., & van Gog, T. (2010). In the eyes of the beholder: How experts and novices interpret dynamic stimuli. Learning and Instruction, 20(2), 146–154. https://doi.org/10.1016/j.learninstruc.2009.02.019.
Jarodzka, H., van Gog, T., Dorr, M., Scheiter, K., & Gerjets, P. (2013). Learning to see: Guiding students' attention via a Model's eye movements fosters learning. Learning and Instruction, 25, 62–70. https://doi.org/10.1016/j.learninstruc.2012.11.004.
Johnson, A. M., Ozogul, G., & Reisslein, M. (2015). Supporting multimedia learning with visual signalling and animated pedagogical agent: Moderating effects of prior knowledge. Journal of Computer Assisted Learning, 31(2), 97–115. https://doi.org/10.1111/jcal.12078.
Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87(4), 329–354. https://doi.org/10.1037/0033-295X.87.4.329.
Krämer, N. C., & Bente, G. (2010). Personalizing e-learning: The social effects of pedagogical agents. Educational Psychology Review, 22(1), 71–87. https://doi.org/10.1007/s10648-010-9123-x.
Lee, H. (2007). Instructional design of web-based simulations for learners with different levels of spatial ability. Instructional Science, 35(6), 467–479. https://doi.org/10.1007/s11251-006-9010-5.
Lester, J. C., Converse, S. A., Kahler, S. E., Barlow, S. T., Stone, B. A., & Bhogal, R. S. (1997, March). The persona effect: Affective impact of animated pedagogical agents. Paper presented at the proceedings of the ACM SIGCHI conference on human factors in computing systems (pp. 359–366). ACM.
Lester, J. C., Stone, B. A., & Stelling, G. D. (1999). Lifelike pedagogical agents for mixed-initiative problem solving in constructivist learning environments. User Modeling and User-adapted Interaction, 9(1–2), 1–44. https://doi.org/10.1023/A:1008374607830.
Liu, H.-C., Lai, M.-L., & Chuang, H.-H. (2011). Using eye-tracking technology to investigate the redundant effect of multimedia web pages on viewers’ cognitive processes. Computers in Human Behavior, 27(6), 2410–2417. https://doi.org/10.1016/j.chb.2011.06.012.
Louwerse, M. M., Graesser, A. C., Lu, S., & Mitchell, H. H. (2005). Social cues in animated conversational agents. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 19(6), 693–704. https://doi.org/10.1002/acp.1117.
Mayer, R. E. (2009). Multimedia learning. Cambridge: Cambridge University Press.
Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167–171. https://doi.org/10.1016/j.learninstruc.2009.02.012.
Mayer, R. E., & DaPra, C. S. (2012). An embodiment effect in computer-based learning with animated pedagogical agents. Journal of Experimental Psychology: Applied, 18(3), 239–252. https://doi.org/10.1037/a0028616.
Mayer, R. E., & Moreno, R. (1998a). A split-attention effect in multimedia learning: Evidence for dual processing systems in working memory. Journal of Educational Psychology, 90(2), 312–320. https://doi.org/10.1037/0022-0663.90.2.312.
Mayer, R. E., & Moreno, R. (1998b). A cognitive theory of multimedia learning: Implications for design principles. Journal of Educational Psychology, 91(2), 358–368. https://doi.org/10.1037/0022-0663.91.2.358.
Mayer, R. E., & Moreno, R. (2003). Nine ways to reduce cognitive load in multimedia learning. Educational Psychologist, 38(1), 43–52. https://doi.org/10.1207/S15326985EP3801_6.
Mayer, R. E., Sobko, K., & Mautone, P. D. (2003). Social cues in multimedia learning: Role of speaker's voice. Journal of Educational Psychology, 95(2), 419–425. https://doi.org/10.1037/0022-0663.95.2.419.
McLaren, B. M., DeLeeuw, K. E., & Mayer, R. E. (2011). A politeness effect in learning with web-based intelligent tutors. International Journal of Human-Computer Studies, 69(1–2), 70–79. https://doi.org/10.1016/j.ijhcs.2010.09.001.
Moreno, R., & Flowerday, T. (2006). Students’ choice of animated pedagogical agents in science learning: A test of the similarity-attraction hypothesis on gender and ethnicity. Contemporary Educational Psychology, 31(2), 186–207. https://doi.org/10.1016/j.cedpsych.2005.05.002.
Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching: Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177–213. https://doi.org/10.1207/S1532690XCI1902_02.
Moreno, R., Reislein, M., & Ozogul, G. (2010). Using virtual peers to guide visual attention during learning: A test of the persona hypothesis. Journal of Media Psychology, 22(2), 52–60. https://doi.org/10.1027/1864-1105/a000008.
Ozcelik, E., Arslan-Ari, I., & Cagiltay, K. (2010). Why does signaling enhance multimedia learning? Evidence from eye movements. Computers in Human Behavior, 26(1), 110–117. https://doi.org/10.1016/j.chb.2009.09.001.
Paas, F., & Sweller, J. (2012). An evolutionary upgrade of cognitive load theory: Using the humarrn motor system and collaboration to support the learning of complex cognitive tasks. Educational Psychology Review, 24(1), 27–45. https://doi.org/10.1007/s10648-011-9179-2.
Paivio, A. (1990). Mental representations: A dual coding approach. Oxford: Oxford University Press.
Park, S. (2015). The effects of social cue principles on cognitive load, situational interest, motivation, and achievement in pedagogical agent multimedia learning. Journal of Educational Technology & Society, 18(4), 211–229. https://doi.org/10.1037/t15489-000.
Parong, J., & Mayer, R. E. (2018). Learning science in immersive virtual reality. Journal of Educational Psychology, 110(6), 785–797. https://doi.org/10.1037/edu0000241.
Pelachaud, C. (2009). Studies on gesture expressivity for a virtual agent. Speech Communication, 51(7), 630–639. https://doi.org/10.1016/j.specom.2008.04.009.
Ryu, J. (2012). The interaction effect of measuring stage and cognitive efficiency. Journal of Educational Technology, 28(4), 663–686.
Ryu, J., & Shin, A. (2017). The effects of integration of picture and text by spatial contiguity principle on learning and cognitive load. The Korean Journal of Educational Methodology Studies, 29(1), 103–120.
Ryu, J., & Yu, J. (2012). The effects of pedagogical agents realism on persona effect and cognitive load factors in cross-use of printed resources and mobile device. The Journal of Korean Association of Computer Education, 15(1), 55–64.
Ryu, J., & Yu, J. (2013). The impact of gesture and facial expression on learning comprehension and persona effect of pedagogical agent. Science of Emotion & Sensibility, 16(3), 281–292.
Schneider, S., Beege, M., Nebel, S., & Rey, G. D. (2018). A meta-analysis of how signaling affects learning with media. Educational Research Review, 23, 1–24. https://doi.org/10.13140/RG.2.2.15920.46081.
Schnotz, W. (2005). An integrated model of text and picture comprehension. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 49–69). New York, NY: Cambridge University Press.
Schnotz, W., & Bannert, M. (2003). Construction and interference in learning from multiple representation. Learning and Instruction, 13(2), 141–156. https://doi.org/10.1016/S0959-4752(02)00017-8.
Schroeder, N. L. (2017). The influence of a pedagogical agent on learners’ cognitive load. Journal of Educational Technology & Society, 20(4), 138–147.
Sweller, J., Ayres, P., & Kalyuga, S. (2011). Cognitive load theory. New York: Springer.
Van Gog, T. (2014). The signaling (or cueing) principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 263–278). New York, NY: Cambridge University Press. https://doi.org/10.1017/CBO9781139547369.014.
Yung, H. I., & Paas, F. (2015). Effects of cueing by a pedagogical agent in an instructional animation: A cognitive load approach. Educational Technology & Society, 18(3), 153–160.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
Cognitive load scale
-
(1) Task Demanding (TDE)
-
TDE1. (02) I felt spent after the learning task.
-
TDE2. (06) I felt the physical demands while I was learning the contents.
-
TDE3. (12) I felt physically tired during the lesson.
-
TDE4. (17) I felt exhausted when I was reading the contents.
-
(2) Mental Effort (MEN)
-
MEN1. (01) I focused on the presented material to be studied.
-
MEN2. (07) I studied hard to learn the given contents.
-
MEN3. (09) I concentrated on my mental effort when reading the material.
-
MEN4. (13) I did my best to understand the contents.
-
(3) Perceived Task Difficulty (DIF)
-
DIF1. (03) It was not easy to understand the contents in learning task.
-
DIF2. (10) The difficulty of learning content was high.
-
DIF3. (14) It was difficult for me to understand the concepts.
-
DIF4. (18) It was hard to recognize the differences between concepts.
-
(4) Self-evaluation (SEV)
-
SEV1. (04) I think that I successfully understood the learning material.
-
SEV2. (11) I am confident to properly apply what I learned.
-
SEV3. (15) I am satisfied after solving (or learning) the tasks.
-
SEV4. (19) I think that I effectively studied the learning material.
-
(5) Usability (USE)
-
USE1. (05) The material layout on the computer screen was readable.
-
USE2. (08) The computer-based materials made it easy to understand the structure of the learning content.
-
USE3. (16) The learning content was well laid out in order to understand on the computer screen.
-
USE4. (20) The computer-based material was well designed to figure out the key points of the learning content.
NOTE. The numbers in parenthesis are the item number shown on an actual survey form.
This is answered on a 7 point Likert scale from 1- ‘not true of me at all’ to 7- ‘extremely true of me’.
Rights and permissions
About this article
Cite this article
Moon, J., Ryu, J. The effects of social and cognitive cues on learning comprehension, eye-gaze pattern, and cognitive load in video instruction. J Comput High Educ 33, 39–63 (2021). https://doi.org/10.1007/s12528-020-09255-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12528-020-09255-x