1 Introduction

Interactive narratives are a form of digital entertainment that allow users of software to interact and change stories according to their own desires and participate in a collaborative experience [9]. In a multimodal learning environment, educational elements can be presented in more than one sensory mode such as audio, music, text, picture, etc. to overcome the limitations of each modality alone [10]. Many multimodal learning environments have been developed, but there has been little work on evaluating their design [8]. As far as we know, no single walkthrough technique considers learner’s emotion in educational technologies. Earlier, we proposed a walkthrough technique for evaluating multimodal educational software by considering emotional aspects of the learners [6]. In this paper we examine our proposed evaluation technique with a case study of software for exploring a museum, and then we refine the technique.

Our evaluation technique is based on our model [5] that was inspired by Bloom’s taxonomy [1]. We adapted the three domains (cognitive, affective and psychomotor), we considered the multiple sensory and quasi-sensory modalities, and our particular focus is on the affective domain. Our model is called the MADE (Multimodal Affect for Design and Evaluation) Framework. In this model the instructors’ learning objective is shown as involving cognitive and affective aspects, and leads to linkages with sensory and quasi-sensory modalities. By quasi-sensory modalities we refer to concepts such as narrative and persuasion.

2 Case Study

Our case study is of software designed to apply narrative to help students visiting a museum. This software, Museum Explorer, was developed by Jesse Gerroir as part of a Master’s thesis [3]. The software was designed for making visits to museums better by providing narrative visualizations. Narrative should support continuity, storytelling and excitement [4]. The software supports an objective, common in relationship to field trips, to help students engage more with the subject matter by planning and reflection.

We use this software as a case study for our MADE evaluation technique called the MADE Walkthrough. This is based on Wharton et al.’s cognitive walkthrough [11], and Dormann and Biddle’s affective walkthrough [2]. It follows Kort et al.’s affective model [7]. This identifies four phases of learning and the affective character of each. The first phase is encouraging exploration with positive affect. The second phase introduces challenges, and negative affect is expected. The third phase is to support overcoming challenges and reduce the negative affect, and the fourth phase is to affirm learning and restore positive affect. In the MADE Walkthrough, the role of the evaluator is to go through some tasks, while looking at four questions.

To examine the effectiveness of the walkthrough, we recruited participants to use our technique to evaluate the software. We recruited only people with Human-Computer Interaction (HCI) evaluation experience, but not members of our own research group. We applied a qualitative approach, audio recorded and took notes for our detail analysis to identify software issues.

The participants applied the walkthrough to evaluate the software. The facilitators first started the process and gave the tasks that were chosen for testing. One person operated the software on a big screen, while the others discussed what to do and what they found. They walked through the software, answering each of the questions and considering the modalities and the teaching objectives. Figure 1 shows the Museum Explorer. On the top is a map of the visited areas and on the bottom is the visualization selection, including: the Slideshow, which is meant to display pictures in a way that is commonly used as a presentation, the Categorical, where information is arranged by the topic of the locations, the Sequential, which focused around giving a sense of time and place, and the Dramatic, to give a sense of the user’s personal experience.

Fig. 1.
figure 1

The Museum Explorer with the four visualization styles.

Full size image

3 Findings

With the first participants we ran into issues where they were focused on the usability rather than the affective dimensions, and found that the MADE Walkthrough [6] was too concerned with usability. The importance of a need for better solutions with regards to evaluating software systems beyond just their usability was clear. Therefore, to bring more emotional aspects of the users into focus (and not usability), we emphasized Kort et al.’s affective model [7] more and modified the wording in our technique, and identified new walkthrough steps. Kort et al.’s model highlights the relationships between emotions and learning, and was built on Russell’s circumplex model. This model focuses on what emotional state the learner is in, and from that, what help he/she needs, and attempts to link the emotion and the cognitive aspects of the learning process. This gave the modified MADE walkthrough (Version 2) in Table 1. We used these modifications for later participants. For each walkthrough question we now present some of the participants’ comments, highlighting those that we felt reflected on the effectiveness of the walkthrough.

Table 1. The MADE walkthrough Version 2
Full size table

First Question: Does the system use positive emotions to encourage the user to explore the learning environment? Participants typically disagreed, and their comments showed the question did influence them to consider the appropriate issues. — P3: The user has a fair amount of freedom to explore. But counteracting that is the lack of clear goals that would provide the motivation to explore. Also visualization is boring and like a form, nice to have a drag and drop interface here. — P1 & P2: There is not much information presented about the exhibits. I am wondering why we adding to the itinerary, what it does? Why we are doing it? Not clear why the need of these functionalities. — P1: Give a dynamic sort of feel, for example using animations, as circle moving to itinerary, expands up or somethings that feels it is living more. — P4: Dramatic slideshow encourages students to seek what they have missed out on and what to look for in the future. (Other participants saw this as a weakness in the software instead of a challenging experience for the learner.)

Second Question: Does the system provide more difficult material to challenge the user? The participants again did understand the intent and offered insightful comments. — P5: The act of exploring the exhibits is not challenging in itself. P3: The activities are a little bit passive, because all they doing is just clicking on things and they probably are not even reading them.

Third Question: Does the system allow the user to persevere and overcome challenges? Their comments were similar: — P3 & P5: There are no challenges to overcome inherent in freely exploring a virtual museum. — P4: It does not prod you to think about these challenges to overcome. It does present data to you, does not seem challenging at all. It does not help you string together a story. Categorical visualization provides a holistic view of the material and then its individual break down by exhibit, and it prompts learning “learn more here”. Allows for a holistic narrative, by theme instead of exhibit.

Fourth Question: Does the system give positive affective feedback to affirm successful learning? The participants explained: — P3: I do not remember any instance of positive feedback during the process of exploring the museum or creating the visualization. — P2: I expected music in dramatic, the name of dramatic more interactive entertaining. — P4: It is good for individual exhibits, but not so much as an overall theme of learning. Provides tools for students to seek out learning, no direct feedback but it encourages learning by having shareable content. Multimedia could also help with this (music, videos, YouTube links).

Overall, the participants understood how to apply the walkthrough. They were able to imagine themselves in the place of the learners and consider which of the four questions related to each situation. Despite not being familiar with the work of Kort, the participants quickly understood the steps, their rationale and the connection between affect and learning. Moreover, they were able to connect the questions with the narrative quasi-sensory modality.

4 Discussion

The participants, using our inspection method, found a number of issues in the software that might lead to significant improvements. For example: they found the software did encourage at the beginning and provide a sense of completion at the end, but had no engaging challenges in between. They also identified many places where multimodality (animation, music, video) would have been beneficial, but was not provided. The participants found the software seemed surprisingly passive and even boring (just adding pictures and comments, but no humor, no music, no video), and did not bring the engagement expected from narrative. They also found inconsistencies and lack of continuity in the interface, which also detracted from engagement. Moreover, the participants also thought the design would not be motivating for users. For example, when some participants understood the visual story idea, they responded by commenting: “it should be exciting like making a movie.”

However, as our sessions progressed, we discovered several important ways to improve our inspection methods. In a normal cognitive walkthrough there are four questions that apply to each task. In our modified MADE walkthrough (Version 2), the questions were more holistic. That is to say that in the beginning of each session the evaluators should have a general idea of the system and the tasks, as the walkthrough is following four steps of learning: exploring, challenging, overcoming, and affirmation. For each task only one of those steps might be appropriate. This perhaps caused the walkthrough to be more difficult.

Table 2. The MADE walkthrough Version 3
Full size table

Once our participants understood this, they could make helpful comments. For example, where a task clearly related to exploring, one participant said: “Yes, I think the software encourages the user to learn by providing positive emotions, it says the task I want is available to me and I can do something with it...”. Moreover, where participants recognized that the task should be more challenging, to ensure real learning was taking place, they identified that the Museum Explorer did not really have any challenges: “There are no achievements or challenges to complete... I thought it was just filling in what I visited (or what I plan to visit), which would be the closest thing to a challenge.” However, because there were no challenges, there was nothing to overcome, and therefore, it was hard to evaluate persuasion and encouragement. This can require delicate design, because different learners will find different levels of challenge possible. We felt that it would be better if all the questions applied at every step, as happens with a cognitive walkthrough.

Therefore, we propose the final version (Version 3) shown in Table 2 that can confidently be used to evaluate an educational system.

5 Conclusions

This paper focused on an evaluation technique for affect in educational technology development. The proposed evaluation method was assessed through a case study. Our participants used the technique to evaluate Museum Explorer software, which featured narrative and visual stories to support greater engagement and reflection. Participants could apply the technique and make useful comments to significantly improve the software. However, by using qualitative analysis of our observations and transcripts of participant think-aloud comments, we were able to identify several ways to improve our inspection technique. We identified a need to de-emphasize ordinary usability, and state explicitly the teaching objectives, the educational strategies and modality advantages involved.