Keywords

1 Introduction

In the latest years, the Danish school system has experienced an increasing digitization of learning practice, in this respect teachers at different levels of education have been actively exploring different learning approaches, with the goal of activating students to be engaged and responsible for their learning. An approach that is becoming widely spread in Denmark is blended learning, which is defined as “the thoughtful integration of classroom face-to-face learning experiences with online learning experiences” [11]. Research on the field of blended learning has a relatively long story, showing that blended learning can bring positive results in giving to the students the opportunity to engage more actively with their teaching material classroom activities than it is observed in traditional lecturing settings [2].

This study is the result of a collaboration between the bachelor programme in Web Development at the Lillebaelt AcademyFootnote 1 (EAL for short) and the Occupational Therapy education from the University College Lillebaelt (UCL), a higher education institution that offers bachelor programmes in the field of education and health care, in the town of Odense (Southern Denmark). The collaboration involved the authors and students from EAL as designers and developers, together with teachers and students from UCL as clients, experts, and users. The goal of this study was to explore how to enrich the practice of blended learning at the education of occupational therapy, through the concrete case of challenging the experience of editing and using quizzes for self-assessment. Moreover, from the authors’ point of view, this work aims at continuing the line of research discussed in Marchetti and Valente [18], which investigated the creation of a learning platform to support the different needs expressed by teachers, students and developers. Building on Marchetti and Valente [18], we will complement EAL students’ findings about their UCL clients, by taking into account the different attitudes and needs that the UCL teachers might express regarding the use of digital tools for learning. We are particularly interested in the different tasks and perspectives of the three actors: UCL teachers, their students, and the developers (i.e. EAL students).

On a general level, through our collaboration with UCL, we found that the practice of online quizzes for self-assessment has become increasingly spread within the framework of blended learning, with the goal of facilitating independent practices of students’ self-reflection on own progress and peer-feedback. At the same time, the teachers saw in quizzes an opportunity to integrate more flexible and frequent evaluations during courses, avoiding the burden of extra work load. However, the use of quizzes was found to be challenging by both UCL students and teachers, mainly in relation to how online quizzes are experienced and how they affect the practice of assessment. Based on initial insights and literature review, in this study we investigate how a digital system could contribute to support blended learning within higher education. To better explain the relation among the various actors, we presented to our EAL students/developers a scenario called Quiz-R-Us in which all three actors are involved in the construction of a website for creating, sharing and taking quizzes. In Quiz-R-Us developers would implement quiz templates, to be used by quiz creators for generating and publishing specific quizzes (customized in the contents and visual style); the quiz players would then login and solve the quizzes, send feedback to quiz creators, show results to other players, with the goal of practicing to learn or memorize.

The rest of the paper presents related work (Sect. 2), the empirical study (Sect. 3), and the design and prototype testing (Sect. 4). Section 4 presents a discussion of our findings, followed by conclusions (Sect. 5).

2 Related Work

The approach of blended learning has a long history, with significant studies already conducted in the year 2000’s, such as [11, 22]. Currently a significant integration of blended learning has already taken place and blended learning has become a norm in higher education [4]. A similar integration can be found also in Denmark, where with the increasing digitization of learning practice, teachers have actively engaged in experimenting with new systems and tools to enrich their daily practices.

Blended learning can be generally defined as an integration of online, distant learning supported by educational, web-based technologies, especially content management systems, and face-to-face learning activities provided by the teachers inside the classroom [4]. Blended learning can provide the advantages of combining the flexibility of online learning with regular in-person tutoring, hence addressing the needs of different types of learners. Several studies are concerned with how blended learning can influence students in becoming more active in their learning, for instance adopting self-regulated learning strategies. The ideal scenario would be to encourage students in becoming effective self-regulated learners, who are self-motivated, able to set appropriate learning goals, and effective in managing their time [24]. A statistical, questionnaire-based comparison between online and blended learning [1], has found that the two approaches influence how students might implement strategies for self-regulated learning. For online students, strategies like time management and effort regulation were found to be positively related to grades. Blended learning students would make more use of strategies like peer learning and help seeking, where online students seem to rely more on online tutorials and other students’ online discussions on particular topics. Another similar study [24] claims that self-regulated learning is a key concept in the design of online learning environments for blended learning, where procrastination is regarded as damaging high learning performance. However, the authors found that in blended learning environments, students are stimulated in developing skills of active procrastination associated with awareness of time management, to be able to meet their deadlines, while participating to online and in-person learning activities.

Several researchers have acknowledged how blended learning can afford a rich variety of approaches and pedagogical frameworks targeting students’ different learning styles, active learning, and peer collaboration [11]. In this respect, a critical perspective was raised by an early study [11], on the complexity of implementing blended learning, given the large design space it creates and the abundance of different contexts in which it can be applied. A similar aspect has been acknowledged also by Barnard-Ashton et al. in a more recent study [1], who argue that the variety of activities afforded by blended learning is introducing a driver for change within education. Barnard-Ashton et al. have specifically investigated the factors affecting the integration of blended learning within an undergraduate curriculum in occupational therapy applying a Problem-Based Learning approach. An interesting aspect of this study was that students expressed positive feelings regarding flexible access to learning materials, some said that they found reassuring to have all the information on their curriculum available in one place, especially before exams [1, p. 7]. At the same time the possibility for self-assessment through quizzes was found beneficial and adding novelty. Similarly, Cohen and Sasson [6] have investigated learning outcomes and the attitude of students towards online quizzes in blended learning courses in an introductory physics course. The students were asked to write essays and to take online quizzes and their attitudes toward the essays and the quizzes were evaluated through a questionnaire, which showed a positive attitude in the students toward online quizzes, as their introduction was considered as an improved in learning practice. Dankbaar [8] conducted a similar analysis about the impact of quizzes and serious games within blended learning practice in medical education. Results from the study seem to confirm that the use of quizzes and blended learning did not negatively affected students learning outcomes. Moreover, Dankbaar concluded that the use of games contributed to the students’ understanding of the subject, especially for students who were new to them, and lead to longer time dedicated to studying. Interestingly Dankbaar points out that there is a need for more design-based research in the design of quizzes and games for learning to better capture the complex requirements of such systems.

Authoring of online quizzes in learning context, a central topic in our study, has been investigated for a long time, and many tools have been proposed, implemented and tested for different subjects. For example, White and Hammer [22], discuss a web-based tool to support teachers’ creation of self-grading quizzes within the area of psychology education. In Freasier et al. [10] from 2003, a web-based quiz system is presented with the aim of supporting learning for undergraduate chemistry students; in this case quizzes are considered as homework. Examples of tools for the rapid creation of web quizzes can be found in [17], where one such system is described, targeted at supporting quiz authoring for English teachers. However, teachers’ problems were not all solved, since a few years later even after the spreading of learning management systems (or LMS, which are special kinds of CMS dedicated to learning resources), we find that easy and rapid creation of quizzes was still investigated: “In most LMS/CMS it is not so quick and easy to create auto-graded quizzes in effective various formats. We have made a prototype tool enabling rapid creation of quiz data importable to Moodle, a popular open source LMS” [25]. Contemporary teachers have even more choice of tools, which are often free and of good quality. Self-grading quizzes can be created directly with Google Forms [23] in a very intuitive way, and without any knowledge of coding; moreover, the data is automatically collected in a spreadsheet for further analysis. The entire system is integrated in Google Docs and is usually considered user-friendly enough that even primary school students can work with it.

An interesting survey of authoring tools (not only for online quizzes) can be found in Haghshenas et al. [14]. One of the tools mentioned in Haghshenas et al. [14], the “Quiz Maker tool” (part of the “Articulate” authoring tool) offers a feature called branch quizzes, where the answer to a question changes which question is presented next. We have encountered a similar idea in our study, as discussed in Sect. 4.

However, despite the progress in supporting online quiz creation, in settings where teachers have low computer proficiency creating online quizzes remains complex: “[…] in three Indian peri-urban schools. We identify key issues impeding digital content creation by low computer proficiency teachers. We also present an authoring framework, Cloze, which successfully enables these teachers to create content […]” [16]. We believe that this can be maintained more in general, whenever IT competences are not part of the teachers’ background, such as the case in many primary and secondary schools in Denmark, or in cases where teachers have a particularly skeptical attitude towards the web or IT.

Many of the papers we surveyed focus on supporting teachers; the other central factor in the successful use of online quizzes is students’ motivation and engagement. Online quizzes can be made mobile-friendly, and one of the main advantages of mobile tests is motivation: “students were highly motivated and enjoyed using mobile application for testing” [20]. In the area of e-learning, online quizzes can also be made more engaging by:

  1. 1.

    Mixing quizzes with games (as in Game-Based Learning), or

  2. 2.

    Gamifying the experience of taking a quiz.

An example of mixing quiz and games can be found in Hamshire et al. [15], where a game is developed to support students starting a higher education. In that study quizzes are used in two ways: for self-assessment in relation with the learning game, but also in the design process. In Hamshire et al. [15] in fact the game was accompanied by a website: “After playing the game, students can access the companion website that includes FAQs, quizzes, videos and student case studies”. Quizzes were also used in the design of the game: “[…] three broad themes used to scaffold the game design […]. The most frequently occurring problems and concerns from these three themes were developed into quiz questions and dilemmas that became three sets of cards for use within the game, using verbatim quotes from students wherever possible.” [15]. In other cases, a quiz can be cast in a game setting, like in Dudzinski et al. [9] where a multiple choice test for pharmacy students provided the basis to build a multiplayer serious game.

Alternatively, the experience of creating and taking quizzes can be gamified. An example is found in Cheong et al. [5], where the authors created a “gamified multiple choice quiz implemented as a software tool and was trialled in three undergraduate IT-related courses.”. Interestingly, gamification can be done also without implementing complex digital games, as discussed in Boskic and Hu [3] for example, where “the instructional designers at the University of British Columbia ventured into the “unknown” and transformed a “traditional” online course on adult education into a role-playing game”. Successful gamification according to Boskic and Hu [3] requires to take “into consideration and combining four major aspects: good pedagogical principles, the suitable course, the instructor who was ready to take risks and a platform that was accessible and user-friendly”. We discuss our approach in relation to similar aspects in Sect. 3.

Another dimension of online and mobile quizzes that is related to students’ engagement is the possibility to incorporate resources other than text. The study in Schön et al. [21] discusses “a lightweight, media-supporting quiz application for mobile devices. The idea is to use the students’ own smartphones as voting devices and provide a quiz administration module which is integrated into the existing university’s e-learning platform. […] The quiz is implemented as a web application, also providing the use of media content such as pictures and videos”. The main problem in systems like the one in Schön et al. (2012) is how to establish links among the participants’ devices; the solution in Schön et al. (2012) is to use “dynamically generated Quick Response (QR) codes”. A modern platform that adopts a similar approach is KahootFootnote 2. We find Kahoot interesting because it is free and widely adopted by schools; it also provides competitive and some forms of collaborative play (like the “team mode”). We also found by reading teachers’ forums, that Kahoot can be used collaboratively in another way: dividing students in groups and letting them create parts of the same quiz. The resulting quiz can later be used by all students to prepare for an exam.

3 Learning Goals and User Study

3.1 Methods and Learning Goals for the Web Development Students

The case discussed in this paper was especially interesting as it required technical skills that EAL students are supposed to acquire through their first semester, but at the same time it required to challenge the current practice of quizzes and assessment. In this respect, the design aspect of Quiz-R-Us requires the EAL students to reflect in depth about the users’ needs and to think creatively before starting to code. This is often a challenge for first semester students, who are very concerned with product development, and in many cases they might end up recreating existing solutions, instead of really innovative ones.

Methodologically the empirical work the authors conducted together with EAL students builds on User Centred Design (Preece et al. [19]), UCD for short, and Lean UX (Gothelf et al. [12]), as required by the learning goals of the Web Development education. The class of EAL students included 17 first semester students, who were divided into four groups of circa 4–5 students, as typical in Danish higher education. Learning Lean UX together with UCD is seen as valuable at EAL, because it introduces a business twist in UCD, which typically focuses on the needs of end users and not so much on business outcomes. As explained by Gothelf et al. [12], the Lean UX methodology combines user experience design with the lean start-up philosophy, which deals with how nowadays entrepreneurs engage in continuing innovation in order to build successful businesses. In this respect, Lean UX is similar to and integrates well with UCD and agile methods for software development, requiring the developers to conduct a research on the users and clients, to develop a series of prototypes, starting from low-fidelity sketches and wireframes to end with high-fidelity implementations, which are more developed, functioning prototypes. These prototypes have to be tested iteratively with user and stakeholders, to fine tune their functional and visual qualities. At the same time, Lean UX gives emphasis to the business aspect of software development, requiring the developers to explicitly discuss how the system under development is supposed to contribute to the clients companies, for instance expanding their basis of customers or facilitating access to their service. Finally, the Lean UX method has a step-by-step structure and requires the use of specific templates for the documentation of the different stages of a design process, so that it is expected to support EAL students in properly conducting and documenting their design inquiry.

Therefore, following Lean UX, the EAL students had to start their design inquiry from a research phase about UCL and their needs as users and stakeholders. Moreover, since the EAL students are also supposed to acquire graphic design skills, theories and methods from the graphic design field were included in the project. For instance during the research stage required by the lean UX method, the students had to conduct a visual research (Dabner et al. [7]), in which graphic designers have to look at the current systems used by their clients and competitors, to take inspiration and create elegant visual solutions that also match with the branding adopted by the clients. The outcome of this research stage was a series of requirements, drafts of possible layouts, and experiments with color palettes and typography.

Afterwards the EAL students had to engage in designing a series of simple low-fidelity prototypes, which had to be iteratively evaluated through a formative evaluation (Preece et al. [19]), in order to show the main concept to the users and gain useful feedback to implement their high-fidelity prototypes, to be tested again in a final, summative evaluation together with the users. The EAL students were also required to video record every session with the users, in order to analyze the input gained from the users after the sessions and use clips and screenshots from the videos as documentation for the project, a common practice in UCD (Preece et al. [19]).

This blending of methods did not create particular issues, since the research stage of the lean UX method also requires developers to investigate current systems in use by the clients and their competitors, exploring the possible changes that could be performed on the current systems and possible business outcomes.

3.2 User Study, Early Findings and Requirements

Few months before the spring semester started, we entered in contact with the teachers of UCL and had two meetings in which we discussed their needs and wishes for new technologies to adopt in their blended learning practices. During these two meetings, we were able to arrange three further sessions: the first one was intended as a user study and was planned for early February, so that it could come after EAL students have been introduced to the methods, and after their visual research stage. In preparation to the user study each group of EAL students made a set of circa 5–6 questions to ask to the students and the teachers at UCL; the questions were about their experience with blended learning and assessment questionnaires.

The second session, scheduled for late March, was a formative evaluation, during which the EAL students had to show initial low-fidelity prototypes at UCL. Around one month later the EAL students had to present at UCL more refined high-fidelity prototypes, in which key functions should already be implemented, so to give a clearer impression on the final application and its scenario.

The analysis of the video material gathered during the interview clearly suggested that there is a strong agreement between UCL students and teachers on the positive effect of blended learning on the students’ independent engagement with their learning process confirming results from current studies [1, 2, 4, 13]. Ac-cording to our analysis, UCL students experience blended learning as more engaging and motivating than traditional learning. Regarding more specifically the experience of online quizzes, UCL uses FronterFootnote 3, a content management system that enables to share learning material and facilitate communication between teachers and students. Fronter includes modules for the making of multiple-choice quizzes for evaluation. Kahoot and Study QuizFootnote 4 are also used at UCL; Kahoot is a popular tool and it is used especially in relation to social events, during which the UCL students make quizzes for other classes and engage in group-based competitions.

Through the interviews and analysis of video recordings, we identified two categories of requirements:

  • Usability – requirements on clarity and technical issues;

  • Experience – the wishes expressed by UCL students and teachers for new user experiences.

With respect to the first category, usability, we found that the use of quizzes poses challenges in relation to the participation of the teachers and the students in creating quizzes and in managing assessment, as the teachers see in quizzes an opportunity for automatizing and reducing their burden in assessment. At the same time the students might not find quizzes engaging and challenging enough, especially when they had to take the same quizzes they were involved in making. Both UCL students and teachers said that quizzes contribute to blended learning, enabling students to check upon their own results and become more aware of their preparation, a point already discussed in [4, 5]. However, the students were more active than their teachers in providing requirements about usability, for example the students argued that they like how quizzes enable them to quickly see their results without having to wait for teachers’ evaluation; students especially liked quizzes that show results after each answer, instead of at the end of the entire quiz. The UCL students also complained that, especially on Fronter, they lack an overview of how many quizzes are available, about which topics and how long they are. They also complained that Fronter is slow when loading, and especially when using it on the mobile phone they have to login more than once, which was described as “annoying” and “demotivating” with respect to taking more quizzes. The more active contribution of students regarding usability could be motivated by the fact that teachers and students experience quizzes and their CMS from different perspectives and with different interfaces. For instance, the CMS interface for taking quizzes might be less stable than the interface teachers use for editing, especially when accessed through a mobile device, an issue that was discussed during the interviews. Moreover, teachers might access quizzes more frequently than their teachers, so that the teachers might edit their quizzes once a month while their many students access quizzes daily or weekly.

Moving towards the second category, experience, we found that UCL students and teachers agree that although social competitions through quizzes can be engaging and motivating, for example using Kahoot. However, in contrast with existing literature, the experience of taking quizzes could be also “boring” and “dry”, when quizzes are too long and only focus on memorizing terms. Both UCL teachers and students argued that quizzes would be more valuable for learning if they supported playful experiences and deeper reflections on the subjects. Unfortunately, both Fronter and Kahoot take a traditional approach to tests, focusing on classical multiple-choice questionnaires, with the addition that Kahoot has also support for uploading images and videos. The UCL teachers instead would like to make quizzes that contain a story, involving also a component of role play “in which the student can imagine to be in that situation, handling a patient”. In their view, quizzes might have questions that “are not only right or wrong”, enabling students and teachers to discuss ethical dilemmas. Use of videos and images was mentioned several times during the interviews as a desired feature, especially in relation to a storytelling approach. We also found that the UCL teachers are experimenting with their students with different forms of board and social games, where the students are involved as co-designers of the games. But since the UCL teachers (as well as the students) lack IT expertise to make these games digital, they mostly rely upon paper and role play, even though they see digital technologies as offering better support for individual assessment. Building on our data, we find a need for challenging the typical concept of multiple-choice questionnaire, to transform it into an engaging experience of reflection and peer learning.

The Quiz-R-Us scenario proved useful in starting our collaboration with UCL, however, the EAL students experienced challenges in the framing of the project and were afraid of not being qualified enough. In fact, they found the requirements of UCL people confusing and leading towards complex software solutions that they could not handle. Moreover, the EAL students also feared that they had to make themselves questionnaires and learning materials for the UCL teachers, and felt they lacked the necessary knowledge in the field of occupational therapy. We had to clari-fy to EAL students that they were not required to take the place of the UCL teachers, but that they should focus on creating a new editor for quizzes and on envisioning more creative scenarios for the making and use of quizzes. It would be the task of the UCL teachers to upload multimedia content and create engaging contents for their students. We consider the response of the EAL students as revealing the usefulness of the three-actors scenario outlined in Marchetti and Valente (2016), which forces developers (the EAL students) to perform a mental step-back, and acknowledge their clients as domain experts (here the UCL teachers and students). This initial setback translated into a slow start, but in the end the EAL students were able to propose innovative concepts that matched the requirements of their users.

3.3 Design and Prototypes

The EAL students worked in four groups and developed four different prototypes. From a technical perspective, the EAL students followed the requirements for their first semester: development of a working prototype adopting specific tools and languages, such as HTML 5, CSS, Bootstrap, jQuery and JavaScript. From a conceptual perspective EAL students experienced difficulties in handling the openness of the project, so they started from designing the layout of the login page and of the different views on the quizzes for the UCL teachers and students. In this way, they were able to brainstorm about the different roles of UCL students and teachers, and the functionalities that those roles required. For instance, all the prototypes had different views for the UCL teachers, who had to create quizzes and monitor the results of their students, and for the UCL students, who needed to access to the available quizzes, pick one, and see their results, also in comparison to other students. Through this process the EAL students started to reflect on how to create more playful and reflective experiences of quizzes for blended learning contexts.

In general, we noticed a convergence of the prototypes to turn traditional multiple-choice quizzes and individual questions into small multimedia puzzle games. One group simply proposed to introduce images and clips, instead of only text, to illustrate the content of multiple-choice questions and their possible answers. From the perspective of the group, images and videos could provide richer and more engaging information than text. A more innovative concept was proposed by another group, who envisioned a questionnaire in which UCL students had to pick the right treatment for patients, based on short video sequences or comic strips. Using this prototype, UCL teachers could take a video representing a specific case, divide it into smaller clips, and construct a questionnaire where the questions are based on one of the clips; in this way, the UCL teachers will be able to create a story throughout the questionnaire depicting one specific case. Moreover, clips could be created and used to display the possible answers that the students have to choose from; this represent a departure from the typical use of other systems like Kahoot, where videos represent questions, while answers are given in a textual form.

Similarly, another group proposed to design multiple-choice questionnaires in which a series of short clips, displaying for instance different stages of a therapy, are shown to the UCL students and have to be placed in the right order (Fig. 1). After the UCL students have placed the clips in the right sequence, the whole video will be played showing how the therapy should be performed. On the contrary in case the completed sequence is not correct, a different video will be played showing the consequences of the wrong therapy on the patients, eventually introducing some humor, like showing catastrophic scenarios for the poorly treated patients.

Fig. 1.
figure 1

Quiz where clips have to be placed in the right order.

The remaining group wanted instead to create quizzes in the form of multi-linear narrative, in which the question is represented by a video (Fig. 2), and each of the different available choices might trigger different story lines, so that which question will appear next will depend on the previous answers. In Haghshenas et al. (2012), this approach is called branch quizzes.

Fig. 2.
figure 2

Quiz in which answers lead to see the consequences related to the chosen answer.

EAL students found challenges in figuring out how to handle the logical sequences of the questions, without having to involve too many storylines, so that the editing of the questionnaire would not become too complex and time consuming to manage for the teachers. The proposed solution was to keep the multiple storylines to a minimum of two or three, and to allow confluence of few questions on the same successive question, to contain the branching factor in the quiz; for the same reason, multi-linear stories typically have various storylines sharing the same ending.

In conclusion, the EAL students tried to re-conceptualize multiple-choice quizzes based on the data that they gathered during the user study at UCL. At the same time, they strived to match the branding identity of UCL, so that the layout of the proto-types, typographic fonts and colors had clear reference to UCL main website.

4 Testing and Discussion

EAL students showed their prototypes at UCL during two testing sessions, a formative and a summative qualitative evaluation. During the first session, they showed initial concepts and simple visual prototypes, in order to gather more specific design requirements. Instead during the second session, that was run about one month later, EAL students showed semi-functioning prototypes and were able to gain more specific feedback. During the first session, the concepts presented by EAL students were conservative and focused on the usability issues discussed by the UCL students during the field study. This means that EAL students tended to stick to the traditional format of a multiple-choice quiz, eventually enriched by images, videos, and by competitive aspects (similar to those found in existing systems like Kahoot). The UCL teachers instead expected to see more refined and innovative concepts already at the formative evaluation (first session), even though we informed them about the design methods adopted in the project. As a result, the teachers provided critical feedback to EAL students, who experienced this first evaluation as unsatisfying. This initial session was, however, useful in challenging the EAL students to face the issues of creating innovative solutions.

The prototypes presented at the summative evaluation (the second session) were appreciated by UCL teachers for their conceptual and visual qualities, enabling them to see more clearly in which direction they would like to enrich their current practice of blended learning. At the same time the more complete prototypes enabled the teachers to better understand the role of the first evaluation, as they could see that their feedback had in fact been considered seriously by the EAL students. One of the teachers said to a group: “I am beginning now to see a way to use your product” and in the end of the session she added that she could understand better our method and she sympathized with the challenge that the EAL students had to face.

The UCL teachers found particularly interesting how the proposed scenarios al-lowed more interactivity than traditional multiple-choice quizzes. They liked the idea of having small puzzles embodied in the questionnaires, like having to rearrange video clips in the right order or to choose the right forms of therapy in relation to a specific situation depicted in the videos. In particular, it was appreciated how both solutions could enable UCL students to see “the consequence of choosing a wrong therapy, so that they can understand why that answer was wrong!”. These scenarios were seen as opening the possibility for the students to have in depth reflections and group discussions about how they would act in a given situation. Following up on this, the teachers asked to one of the EAL groups: “Could you imagine questions where there is no correct answer?”. According to the UCL teachers, occupational therapists often deal with ethical dilemmas, in which there is not a straightforward solution and different actions could be equally right, yet these could have different consequences on the well-being of the patients and on their relationship with their therapists. They said that it could be interesting to have questions like: “When we come home to an old lady, we have to act like her guest, but we also have to make her practice. How do we react if she is yelling at us, spitting at us?”. Possible answers could be, according to one UCL teacher: “Yell, give her a hug or walk away”. The UCL students would then have to try and see which consequences their actions have on the old lady. This kind of questions could be easily implemented, however, it would not make sense to con-sider an answer as right or wrong, instead the UCL teachers suggested that there could be a space for writing a reflection on the choice made. The UCL students could collect points from the system, the teachers or from each other, for having written meaningful reflections. Moreover, this scenario opens the possibility to compile a list of the reflections written by all the students taking the quiz, providing a kind of collective learning diary for the UCL teachers and the students. On the other hand, the individual reflections written by a single student could be stored in a personal learning diary, and help the student realize how her critical perspective changed as her critical insights on the subject matured through time. This scenario involving videos also opened an interesting discussion, both on a practical and moral level. For instance, the creation of videos could be quite time consuming for UCL teachers and students, who do not have professional skills in video editing. Moreover, a moral dilemma might arise, in having to film and use images of sick individuals, eventually children and elderly, during their daily therapy. Hence, we discussed the opportunity to leverage on the abundance of materials already available on the Internet and on social media. The UCL teachers pointed at the fact that the Danish UCL institutions have a channel on the Internet, where they publish videos online in different occasions, which could be easily used for quizzes.

At the summative evaluation the UCL teachers also asked if it could be possible to embed a story in the questionnaires. And in fact, the EAL students presented quizzes where that would be possible, but since these where incomplete, showing mainly a short series of fake questions (i.e. mock-ups), the teachers could not immediately see how they could construct a story through the questions; further explanations where needed to clarify this point.

The proposed scenarios were also appreciated from a practical perspective as they all enable the UCL students to see their scores while taking the quizzes, and this would foster more discussion when blended with other activities in the classroom. The UCL student who was present at the summative evaluation added that this could also help her in getting an overview of the questionnaire and of how she was doing.

In general, the UCL students seemed more concerned with usability and accessibility of quizzes, as they would like to access quizzes from their phone and criticized their current platform for not working properly on mobile devices and for lacking a good overview, which might be even more challenging to have on a small screen. From an experience perspective, the UCL students were mainly interested in gaining access to new quizzes that could enable them to go further with their learning and also they would like to have more social, competitive sessions supported by peer learning. The UCL teachers instead were especially interested in eliciting in their students’ deeper understandings of the topic and had the idea that this could be done by representing stories, concrete cases and ethical dilemmas in the questionnaires. Moreover, it was interesting to see the challenges that UCL teachers faced with understanding our iterative methods, so that they were clearly disappointed with the first evaluation, but even more satisfied with the final one, when they could also make sense of the simple visualizations they saw during the first evaluation.

4.1 Discussion

We faced various challenges during this project. The first was the fact that UCL students did not participate actively in the cooperation. During the user study we were able to involve five students from UCL, plus the two teachers who were our contact. Unfortunately, in the formative and summative evaluations we only had one UCL student participating. In this respect our design requirements were based on the perspectives of both teachers and students, while the testing was covered the teachers’ perspective, with only one student being present. This situation was generated by an issue of coordinating the cooperation between large groups of two different institutions. Both EAL and UCL students were in fact intensively booked with teaching activities and assignments from their schools, at the same time the teachers were busy with different teaching and coordination activities. Therefore, it was hard to create a reasonable schedule that enabled all the participants to meet and discuss the project.

A second challenge was to support the EAL students in framing the project correctly. Designing and developing a system like Quiz-R-Us is different from developing of a learning game, where the main focus is on the creation of the learning contents as well as the digital game. Here the EAL students found themselves more in the position of designers developing digital design tools for others, who work in areas that are alien to the developers (i.e. the UCL teachers). Therefore, we find that this case enabled us to better grasp how the three roles (developers, UCL teachers and their students), are in a relation of co-dependence where each of them affects the other two, making it difficult to set clear boundaries on how authorship is allotted and shared.

Finally, despite the challenges that we faced in coordinating the project, interesting insights were gained through the design process. First of all, that at UCL both teachers and students desired a more playful and engaging experience and our developers gained in experience from being exposed to the different perspectives of different groups of users, and eventually designed more innovative solutions.

The prototypes and scenarios created in the course of the project tend to confirm current studies [1, 4, 5], showing that online, mobile quizzes are effective tools to support learning in blended learning environments. However, in opposition to existing studies, several conditions were individuated, for positively evaluated quizzes:

  • Quizzes that do not require technical skills to be created, and take into account the roles and needs of all groups of users;

  • That provide partial results (instead of summative scores) because that allows discussion in the classroom and potentially deeper as well as peer learning;

  • That can use a larger palette of media contents, to engage quiz takers and to give more expressivity to the authors;

  • That have questions that follow multiple storylines, showing consequences of decisions, instead of just providing scores;

  • That can be stretched towards simulations, games or playful experience, to pro-vide more narrative and visual elements, and some form of interaction.

Especially with respect to the last point, during this project we realized that authoring quizzes is a familiar activity for teachers, as solving them is a typical activity for students. Quizzes are usually textual, linear and not very interactive, and are typically much less complex than digital games; however, we were positively surprised when the teachers started discussing of a story with multiple lines embedded in the questions and answers of a quiz. By trying to imagine all the alternative ways in which a student could “navigate” a similar quiz, the teachers were actually talking about the gameplay of a simple simulation game. So, starting from a known activity, we were able to involve teachers in a discussion that resembles game design, in a much more natural way than we expected.

Regarding the different attitudes of UCL teachers regarding the use of technology in the classroom, we have confirmed results from our previous studies [Marchetti and Valente] in which we found that teachers’ attitudes fall into three categories: the technology concerned, who would minimize their use of technologies; the mediators, who would actively search for what technologies or media are available and creatively contextualize them in their teaching activities, and the designers, who would like to develop creative experiences for their students on their own, but would recur to role play paper-based, low-tech solutions, like technology concerned teachers, when required coding skills that they do not have. We noticed in fact that UCL teachers have generally a positive and creative attitude towards using technologies in their teaching activities. Thus, they would generally switch between acting as mediators and designers, as they at times use technologies and media that are available but are also willing to design new experiences for their students. In such case, they would then recur to paper-based games and role play, as in [18], to avoid the need for programming. However, the teachers said that they would like to use digital solutions to enrich their blended learning practice and students’ engagement with the learning material. In Quiz-R-Us we empower our users, both teachers and students, enabling them to take and edit new quizzes. With quizzes, editing and taking the test are comparably similar activities, and we believe this helped teachers (who usually are not programmers) to bridge the gap between specification and execution. In sandbox games (like MinecraftFootnote 5) we find an analogous situation, where players can act as level designers via the same mechanics they normally use when playing.

In agreement with existing studies [1, 8], we found that blended learning is actually appreciated for supporting a variety of different approaches, hence not providing specific guidelines about “how” to sensibly combine face-to-face with distant learning practices, an aspect that was criticized by other researchers [6]. In our study, we found that at UCL blended learning is appreciated as an open-ended framework, hence the recurring statements acknowledging how different media, technologies and social activities can be easily contextualized within blended learning. This openness of blended learning is interpreted rather as an opportunity to frame playful learning activities. During user study and evaluations, UCL teachers were pointing at playful learning and the use of games as preferred resources to enrich their approach to blended learning. In this respect, even if quizzes are perceived as requiring improvement, they are in fact seen by UCL students and teachers as an integral element of blended learning, existing in both digital and non-digital formats, as a distant as well as a social, face-to-face activity, bridging between individual and social learning activities.

5 Conclusions

In this study, we have investigated how to enrich the practice of blended learning and the use of quizzes within the education of occupational therapy. We found that blended learning often involves quizzes, and that quizzes can pose complex questions in relation to how teachers and students experience assessment.

Interestingly, UCL teachers and students perceive quizzes as an integral practice in their approach to blended learning supporting, both in face-to-face and distant learning practice, yet in the beginning of our study neither of them could point at precise design requirements. This offered an opportunity to creatively explore innovative concepts, challenging the current notion of multiple-choice questionnaires, in order to support the needs of both groups of users. Moreover, quizzes served as a concrete base to start a productive dialogue about a future playful scenario.

One of our most difficult tasks required balancing the learning goals of our EAL students with the development of good solutions for the UCL teachers and students. The EAL students for instance experienced the initial phases of project as confusing, as they feared that the project was too complex for their technical expertise and that they lacked the necessary knowledge in the field of occupational therapy. We interpreted this as a problem in recasting their role from developers of a system like a learning game to developers of a tool supporting the two groups of users. The EAL students eventually appropriated this new role and realized that they were in fact required to define both a prototype and to envision future scenarios for the users. Further in the development process, the focus on the different needs of teachers and UCL students helped in creating successful prototypes. In this sense through this project we (and our EAL students) could see more clearly how the three roles are in a relation of co-dependence with each other. The discussions that started in the course of the project provided important insights about how to look at quizzes in novel ways, tailor them better for the style of teaching used at UCL, and make them more informative and engaging for the students.

The main weakness of our study was the limited involved of UCL students in occupational therapy, who were supposed to participate in testing the prototypes and to contribute with critical insights on the proposed scenario. This means that our results from testing are more related to the teachers’ perception of our proposed scenarios.

Through this design-based investigation we found that blended learning is acknowledged as a valuable approach to learning, enabling for a variety of different activities. However, we also found that at UCL, blended learning is reinterpreted more specifically as a framework for playful learning activities, where quizzes provide a valuable resource for playful engagement both in distant and face-to-face learning activities.

We are currently in the course of conducting a new iteration, in order to continue to investigate this intertwining between blended and playful learning, focusing on a better involvement of their students and testing of more complete prototypes. Within this new iteration, we are also cooperating with local companies, working on a digital platform for blended learning.