Assessing the effects of different multimedia materials on emotions and learning performance for visual and verbal style learners

Abstract

Multimedia materials are now increasingly used in curricula. However, individual preferences for multimedia materials based on visual and verbal cognitive styles may affect learners' emotions and performance. Therefore, in-depth studies that investigate how different multimedia materials affect learning performance and the emotions of learners with visual and verbal cognitive styles are needed. Additionally, many education scholars have argued that emotions directly affect learning performance. Therefore, a further study that confirms the relationships between learners' emotions and performance for learners with visual and verbal cognitive styles will provide useful knowledge in terms of designing an emotion-based adaptive multimedia learning system for supporting personalized learning. To investigate these issues, the study applies the Style of Processing (SOP) scale to identify verbalizers and visualizers. Moreover, the emotion assessment instrument emWave, which was developed by HeartMath, is applied to assess variations in emotional states for verbalizers and visualizers during learning processes. Three different multimedia materials, static text and image-based multimedia material, video-based multimedia material, and animated interactive multimedia material, were presented to verbalizers and visualizers to investigate how different multimedia materials affect individual learning performance and emotion, and to identify relationships between learning performance and emotion. Experimental results show that video-based multimedia material generates the best learning performance and most positive emotion for verbalizers. Moreover, dynamic multimedia materials containing video and animation are more appropriate for visualizers than static multimedia materials containing text and image. Finally, a partial correlation exists between negative emotion and learning performance; that is, negative emotion and pretest scores considered together and negative emotion alone can predict learning performance of visualizers who use video-based multimedia material for learning.

Introduction

Educational materials are no longer limited to static text; that is, presentation is being transformed from text-based materials to multimedia materials to harness learner attention and interest. Particularly, considerable attention has focused on developing interactive multimedia materials to promote learning motivation and performance (ChanLin, 1998; Lowe, 2003). However, whether different multimedia materials with the same learning content and objectives have the same effects on learners' emotions and performance for learners with different cognitive styles must be clarified. Since each learner has a different cognitive learning style that is associated with multimedia material learning, presenting inappropriate multimedia learning materials to learners may negatively affect learning.

Emotions can affect attention, meaning creation, and the formation of memory channels. Hence, emotional status and learning are strongly correlated (LeDoux, 1994). Kort, Reilly, and Picard (2001) indicated that emotion sets with positive–negative oppositions, including anxiety–confidence, boredom–fascination, frustration–euphoria, dispirited–encouraged, and terror–enchantment, are likely relevant to learning. Additionally, many psychologists and neurologists have demonstrated that emotions and motives are important to cognitive learning (Izard, 1984). Goleman (1995) noted that depressed, angry, and anxious students have difficulty learning. According to Piaget (1989), human emotions can arise from or interfere with learning. Izard (1984) analyzed the degree to which cognitive activities are adversely affected by negative emotions and enhanced by positive emotions. In other words, identifying the cognitive and emotional states of students during instruction can facilitate the development of positive learning experiences for learners (Reilly & Kort, 2004).

Most current methods for assessing learner's emotional states are post hoc subjective assessments of emotion according to learner's self-reporting on a set of questions (Schutte, Malouff, & Bhullar, 2009). However, these methods are often not sensitive to changes in emotion. Also, how to design a set of questions that can obtain an accurate assessment of the learners' emotional states is difficult since learners may find it difficult to express their own feelings. In recent years, emotion-recognition technologies based on human physiological signals have been developed for practical application (Bi & Fan, 2011; Chen & Lee, 2011; Shen, Wang, & Shen, 2009), such that investigating the correlation among different multimedia materials, learner emotional states, and learning performance is now practicable. Particularly, many past studies demonstrated that heart rate variability (HRV) patterns are directly responsive to changes in emotional states (Latham, 2006; McCraty, Atkinson, Tiller, Rein, & Watkins, 1995; Tiller, McCraty, & Atkinson, 1996). The emWave, which is a stress detector of emotional state developed by HeartMath (http://www.heartmathstore.com/), uses HRV human physiological signals to recognize the emotions of individual learners as positive, peaceful or negative. The emWave has been successfully applied as biofeedback-based stress management tool in medical field to reduce physician stress and identified that it was confirmed as a simple and effective stress-reduction strategy for physicians (Bedell & Kaszkin-Bettag, 2010; Lemaire, Wallace, Lewin, de Grood, & Schaefer, 2011). It has also been applied in educational settings to facilitate social, emotional, and academic learning (McCraty, 2005). Using HRV human physiological signals for the objective and real time assessment of learner's emotion provides an opportunity to objectively detect small variations in emotion in real time.

By applying emWave to analyze emotional data, our previous study (Chen & Wang, 2011) demonstrated that video-based multimedia material provided to 170 Grade 5 elementary school students for energy education generated the best learning performance and most positive emotions among static text and image-based multimedia material, video-based multimedia material, and animated and interactive multimedia material. Moreover, a partial correlation exists between negative emotions and learning performance. Importantly, pretest scores and negative emotions considered simultaneously can predict learning performance of learners who use video-based multimedia material for learning. Notably, female learners are more affected by different multimedia materials than male learners (Chen & Wang, 2011). However, an important goal is to determine whether individual differences in the visualizer–verbalizer dimension affect learning performance and emotions of learners using different multimedia materials for learning. The study thus deals with emotional variations and the corresponding learning performance when learners with visual and verbal learning styles are presented different multimedia materials.