Lessons Learned from Evaluating an Authoring Tool for Learning Objects

Abstract

Teachers and students often adopt technologies that allow new ways of teaching and learning, and multimedia resources such as slideshows, videos and games have been increasingly used in both distance and face-to-face instruction. These multimedia learning environments handle several media resources (e.g., video, image, text). One such environment is the multimedia authoring tool Cacuriá. Cacuriá allows teachers to create multimedia educational content for interactive TV and Web without requiring programming skills. In this paper we present a case study which was conducted to show how learning objects could be created using Cacuriá without requiring the user to previously know programming concepts. Finally, usability tests results show that Cacuriá can be used by teachers with effectiveness and consistency.

1 Introduction

Learning Objects (LOs) are defined as any content used to support the learning process by computer(s) [1, 2]. In the last years there was an increase in authoring tools providing teachers with different ways to create content (e.g., slideshows, videos, and games). They can be used in both forms of education: classroom and distance [3,4,5,6].

Fig. 1.

Service architecture.

Fig. 2.

Research methodology.

Cacuriá is a multimedia authoring tool that serves that purpose [7]. Cacuriá allows teachers to create multimedia educational content for interactive TV and the Web without requiring programming skills. The tool also enables the teacher to create applications which adapt to students’ interactions. Cacuriá is integrated with the iVoD (Interactive Video on Demand) service from RNP, a National Research and Educational Network responsible for promoting the development of networks in Brazil, including the development of innovative applications and services. iVoD was specified to allow the storage and distribution of LOs integrated into the authoring tool. Figure 1 describes the architecture of the iVoD service. First, a teacher creates a LO using Cacuriá and then submits it to the iVoD cloud storage. Then, students can watch and interact with the content created by the teacher with various devices with Internet access, such as tablets, computers, and smartphones. The service is used by 15 universities in different regions of Brazil and nearly 30 LOs have been created between 2015 and 2016.

The methodology used in the development of Cacuriá is presented in Fig. 2. As described elsewhere [7], the Cacuriá’s design process integrates Participatory Design [8] and Interaction Design [9] methods. In order to present the Cacuriá’s evaluation process, this paper considers that usability is an important factor which influences the quality of a software design. The lack of usability in a software system can lead users to dissatisfaction, low productivity and wasted time. In this paper, we present the results of usability tests performed with users of the tool and the main lessons learned from them. Moreover, we also show a case study which was conducted to demonstrate how learning objects can be created using Cacuriá.

This paper is structured as follows. Section 2 presents some related works. Section 3 describes the evaluation process of a multimedia authoring tool for learning objects, and Sect. 4 presents its results. Finally, Sect. 5 provides some final considerations.

2 Related Works

Usability can be defined as the “effectiveness, efficiency and satisfaction with which specified users achieve specified goals in particular environments” [10]. To ensure a system has high usability, several usability evaluation methods have been proposed by experts and researchers [11], and many works in the literature present usability evaluations of softwares for distinct domains [12,13,14,15].

Feizi and Wong performed an empirical study with 32 user interfaces designers and software developers to investigate the usability attributes of effectiveness, efficiency and satisfaction scores for learning Adobe Flash CS4 and Microsoft Expression Blend 4 [12]. Their goal was to compare the impacts of adopting different user interface styles: graphical user interface (GUI) and command-line interface (CLI). The results showed that participants perceived CLI as more difficult to learn and use and, although participants perceived GUI as simpler to learn, the results highlighted the need to provide menu labels and icons that are familiar and easy to find.

Cabada et al. [13] describe a collaborative learning environment called Educa. Educa is composed of five modules: an authoring tool, two repositories (resources and courses), a delivery engine, and a recommendation engine. The authoring tool was implemented to create adaptive learning material. The authors evaluated the authoring tool with 30 participants through a 5-point Likert scale questionnaire. The results showed that most participants “agreed” or “strongly agreed” with respect to the software interfaces’ ease to generate an intelligent tutor, learning time for using the tool, time to produce a content and the course organization.

In turn, Marchiori et al. present a system named WEEV (Writing Environment for Educational Video games), whose main goal is to facilitate educational video game development by educators [14]. This authoring tool was tested with 20 software engineering students and 9 educators. The goal of evaluation was to discover problems in the user interaction with the system and to assess the perception of the system by educators. Some users detected problems in the software such as, some saved files were unreadable, some constructs needed to be deleted in order to be modified and help panels provided lots of information of limited relevance. All the students implemented an application using the WEEV around 50 min. On the one hand, the students did not seem to value the usefulness of the software. On the other hand, the educators showed interest in using technologies that would allow them to create their own games. Nevertheless, although the evaluation took around 90 min, the educators did not have time to fully develop the test application. They found usability problems (e.g. the system was complex to use, especially in the creation of new elements). The results point to the need to help the educators to understand the metaphor used in tool and to provide sample games in order for them to understand the purpose of the system.

3 Evaluation Process

This section describes the evaluation process, which consisted of two steps: a case study and a usability test. The first step aimed to evaluate whether it is possible for experts to build interactive applications with Cacuriá. The second one aimed to evaluate the usability of Cacuriá and to investigate whether end users can successfully create an interactive application.

3.1 Case Study

The goal of the case study was to identify a range of applications that can be created using Cacuriá. We therefore analyzed the tool and identified the models of learning objects that could be created with it.

To illustrate the development process of the interactive application, we describe here the process of authoring an application about tourist spots in Rio de Janeiro. This application is called “Roteiro do dia” (Tour of the day) and, as Fig. 3 shows, it is composed of four images and five videos. It starts with an introduction video describing some places in Rio. Then, the video offers the possibility of getting to know more about two locations (Central do Brasil and Copacabana). At the end of whichever video the user may choose, two additional locations are offered (Gafieira Estudantina and Jardim Botânico) for users to obtain more information.

The temporal synchronism with the video happens in the exhibition of the images “Central do Brasil.png”, “Copacabana.png”, “Gafieira Estudantina.png” and “Jardim Botânico.png”, which are shown just in the final seconds of each video. The nonlinear authoring is characterized by offering a choice of place about which the student may want to obtain further information. This choice may appear to users as a kind of customized experience.

Fig. 3.

The “Roteiro do dia” application.

3.2 Usability Test

The evaluation was performed with 44 teachers distributed in sessions of 6 to 10 participants (Fig. 4). They had little or no experience in using authoring tools. The overall goal of the usability test was to evaluate user satisfaction, effectiveness and efficiency of the tool and to investigate whether the prototype supports teachers in the creation of interactive content.

Fig. 4.

Tests performed with the tool.

The tests were carried out using computers containing a software for capturing the actions of the participant and the authoring tool for learning objects. A folder in each computer contained a shortcut to access the tool and the media to be used when building the application. Moreover, a 30-step task script for the construction of the application was distributed among the participants (Table 1).

Table 1. Script tasks user testing.

Each session started with a brief introduction about the authoring tool. Then the “Roteiro do dia” application was run. Finally we asked participants to build the application using Cacuriá. After developing the application, each teacher answered an online multiple choice questionnaire, based on version 7.0 of the Questionnaire for User-Interaction Satisfaction (QUIS)¹, whose objective is to measure user satisfaction [16]. The questionnaire was adapted so as not to be long and to assess just what was relevant for the tool. It therefore included only 44 of the 126 QUIS questions. Moreover, the original scales were reduced from 9 to 5 points, ranging from 1, representing the user’s dislike, to 5, representing the user’s satisfaction with the corresponding aspect.

The questionnaire used in the evaluation was divided into 6 sections. The first section was related to the user identification and contains fields to enter the name, occupation, experience as a teacher or tutor, as well as programming skills. Next, the participants should answer questions to assess their overall perception of the tool. The third section aimed to evaluate the tool interface. The fourth section contained questions related to terminology and system information. The fifth section consisted of questions focused on the evaluation of learning to use the tool. The sixth and final section was associated with system capabilities such as speed, response time, correcting typos, etc. At the end of sections 3 to 6, participants could make free-form textual comments about the aspects of the tool addressed in the corresponding section.

4 Results and Findings

In this section we present the results of case study and the usability test, which show the effectiveness and efficiency of the authoring tool Cacuriá. The participants’ feedback raised new requirements and recommendations for refining the tool.

Fig. 5.

Cacuriá’s interface.

4.1 Case Study

Based on the results obtained during the Participatory, Interface Design and Implementation phases [7], we defined and developed Cacuriá as a tool to support users in generating nonlinear learning objects composed of synchronized media (e.g., video, image and text). The “Roteiro do dia” application was used to illustrate the content model generated by the tool and some other features the tool offered, such as the synchronism between media objects and the insertion of links on the scenes to demonstrate the nonlinearity of content. As the study involved skilled users, there was no need to follow the step-by-step task script provided to users in the usability testing activity.

When Cacuriá is run, the first action to be taken is to click on the video icon to choose the first video for the scene. Then, the video is added in the Library View and its first frame is shown in the Layout View. In addition, the timeline of the Temporal View receives the total duration of the video and the options in the Properties View are enabled.

Next, four scenes are added through the option “add scene” located in the Scene View. Then, the second scene is selected in the same view, in order to add the “Central do Brasil.mp4” video. Similarly, the “Copacabana.mp4”, “Gafieira Estudantina.mp4” and “Jardim Botânico.mp4” videos are inserted in the third, fourth and fifth scenes, respectively.

Then, back to the first scene to add and position the “Central do Brasil.png” and “Copacabana.png” images on the video. Next, the links from the first scene are created. The images recently added are selected and links to the second and third scene are set in the Properties View, as Fig. 5 illustrates. Similarly, the “Gafieira Estudantina.png” and “Jardim Botânico.png” images are inserted and positioned in both the second and third scenes. Lastly, a link is also configured for each image to trigger the fourth and fifth scenes, respectively.

4.2 Usability Test

The usability test was performed in seven days. Only 6 of the 44 participants failed to perform all the tasks. We noticed a certain degree of difficulty to start the application development: the average learning time was around 10 min. Most participants built the application in less than 40 min, which was the expected time. Although the tool still requires improvements in its efficiency, the results demonstrate that an adequate degree of effectiveness was achieved.

The results are presented through graphs with percentages of agreeing and disagreeing responses to each question. The use of color in the graphs aims to make it easy to identify where there are agreements on the proposed model and what problems were found in the tool. The evaluation analysis also includes the comments provided by the participants at the end of each stage. The results considered satisfactory are those for which the participant choses option 4 or 5 (green color) on the scale. But when the option 1, 2 or 3 (red and gray color) was selected, the result is classified as a problem and considered a feature to be improved.

Fig. 6.

General impressions about the tool. (Color figure online)

Figure 6 shows the results of the general impressions that the participants had after using Cacuriá. It can be observed that most users believed Cacuriá to be a useful tool (Q1). Furthermore, it was though as motivating for the construction of learning objects through the tool (Q2). More than half of participants assessed the tool features as sufficient (Q4) and both its use (Q3) and the options offered to make the activities (Q5) were deemed satisfactory. Based on these data, we notice overall positive general impressions about the tool.

According to Fig. 7, over half of the users rated the letters used in the tool interface as easy to identify (Q6), with adequate font sharpness (Q7) and good readability (Q8). In addition, most of them thought it was easy (Q10) to find the media properties and (Q9) to identify the corresponding icons to add video, image, text, shapes, as well as to publish a learning object. Regarding system colors (Q11), although they were considered appropriate in general, some users found them a little confusing. Meanwhile, most participants reported as adequate the arrangement of information (Q12) and the progression of work-related tasks (Q13).

Fig. 7.

Overall results about the user interface. (Color figure online)

The results regarding the terminology and system information are shown in Fig. 8. Most of the participants perceived as consistent the system terminology (Q14), the messages displayed on the user interface (Q19), and the terms related to the task (Q15) and the computer (Q16). They also agreed that performing an operation in Cacuriá leads to a predictable results (Q22). Moreover, the computer terms used in the tool (Q17) and displayed on the system interface (Q18) were evaluated as appropriately and precise. However, the feedback messages issued by the tool can be improved. The scores about the instructions for correcting errors (Q20) and about whether the system keeps the user informed about what it is doing (Q21) were satisfactory. Moreover, issues regarding the error messages (Q23), the phrasing of error messages (Q25) and if error messages clarify the problem (Q24) were assessed as unsatisfactory.

Fig. 8.

Overall results about the terminology and system information. (Color figure online)

As can be seen in Fig. 9, the results obtained in the evaluation of learning demonstrate that Cacuriá was perceived as easy to operate (Q26). Most of the participants positively assessed issues regarding getting started (Q27), learning advanced features (Q28), time to learn to use the system (Q29) exploration of features by trial and error (Q30), discovery of new features (Q31), straightforwardness of tasks performance (Q32), number of steps per task (Q33), logical sequence of steps to complete a task (Q34), and feedback on the completion of steps (Q35).

Fig. 9.

Overall results about the learning. (Color figure online)

Some participants made comments regarding the ease, simplicity, and speed in learning to use the tool. A participant reported that: “for people like me who have worked with some video editor or even Microsoft PowerPoint, learning is not time-consuming because the symbols follow the same standard and were well applied in Cacuriá”. Despite the satisfactory results, participants also gave interesting suggestions for the user to learn to operate Cacuriá more efficiently. The main suggestions were related to the use of a manual and videos embedded in the tool to demonstrate system features.

Figure 10 shows the evaluation results of system capabilities. Satisfactory results were obtained on issues related to system speed (Q36), response time for most operations (Q37), rate at which information is displayed (Q38), correcting typos (Q40), whether the ease of operation depends on the user’s level of experience (Q42), and whether the user could accomplish tasks knowing only a few commands (Q43). Most users considered it easy to correct their mistakes (Q39) and to use shortcuts to perform actions (Q44). Nevertheless, the support to undo operations (Q41) was classified as inadequate. Despite satisfactory results in Q39 and Q44, they were also considered as features to be improved due to the number of participants’ suggestions. In addition, many participants remarked that the tool needs to offer users an option to undo, triggered by the “Ctrl + Z” key combination, as well as more shortcuts to perform functions.

Fig. 10.

Overall results about system capabilities. (Color figure online)

5 Final Considerations

The evaluation of the current version of Cacuriá achieved satisfactory results regarding its effectiveness. The tool proved to be useful for teacher to build classes for distance learning and supplementary content for classroom teaching. The main positive results of the usability tests were related to the learnability of the tool user interface in terms of its “intuitiveness” and reduced learning time. However, some improvements would be welcome, related both to the colors used in the user interface and the feedback messages issued by the tool.

We also evaluated the use of Cacuriá in a rural region in a poor state of Brazil. The challenge was to allow teachers in those areas to create their own educational contents based on their local reality. This experience had interesting preliminary results. Teachers used the tool to create different classes. In a geometry class, teachers showed how to measure the area of polygons using the city’s main square. In a geography class, teachers talked about the pollution of the local river. Despite the regional limitations and the lack of experience by the teachers, the students described that they feel more engaged with interactive and multimedia contents than with conventional classes.

Based on the development experience of Cacuriá, it seems that the effectiveness was achieved in the first cycle, due to the inclusion of the stakeholders in the tool design process. However, more cycles are necessary to improve the tool’s efficiency. It seems that the proposed methodology can help the general usability of multimedia tools.