1 Introduction

Student engagement has long been recognized as integral to both performance and retention of learning [4]. Nevertheless, it is still an ongoing research and policy topic to understand the aspects that could lead to improve students engagement [2]. Digital Teaching Platforms (DTPs) in the classroom have been identified as a tool for engaging students, however only if suitable pedagogical methods and Learning Objects are adopted to meet the teachers pedagogical plan and classroom needs [7].

In this paper we evaluate the proof-of-concept implementation of a novel tablet-based DTP, designed from the ground up to support teachers in Brazilian classrooms. After in-loco interviews with specialists and students, classroom observations and review of the literature, we found two significant research opportunities: (1) the existent solutions rely on a stable broadband internet connection, which in Brazil is not always available; and (2) teachers indicated that they did not know how to use the existent platforms due to an over-abundance of features and lack of support. The principal requirements were thus an easy-to-use out-of-the-box platform that enables teachers to incorporate the use of tablets into their regular lesson plan and stimulates student engagement.

We developed a proof-of-concept implementation of the DTP consisted of four principal components: (i) a light-weight content server, running on a Raspberry Pi, (ii) a wifi router, (iii) one tablet per person (students and teacher), and (iv) a SmartTV or interactive whiteboard and tested in two different ways: in a usability lab where we identify hurdles in how students navigate through the content, and in field trials where we evaluate the response from both students and teachers when using the system.

The DTP we developed can be categorized in the scope of Mobile Learning, as described by Pereira et al. [8]. They found that through the evolution of Learning Models, traditional learning techniques could be enhanced by Electronic Learning and Mobile Learning. Our principal difference from current works in this area regards the user-centric research in order to build our software solution.

The work of Samuel et al. [10] presents IGLOO, a mobile learning technology that intends to support educators and students in learning environments. IGLOO consists of an administrative interface and a mobile application to run of the educator’s device. The administrative interface is used by the educators in order to create quizzes. The educators interface will receive those quizzes by SMS or Bluetooth. Their evaluation is based on achieved tasks rather than the student performance. The difference from our work is that we are proposing a solution that encompasses different aspects regarding educational activities; for example, the association of the educational contents with the program of the discipline.

Potts et al. [9] propose an m-learning application for administering quizzes that offers a multi touch interface compatible with Android and iOS devices. After the students answers tests, the results are sent to the educator by email. The presented work, developed a prototype that is flexible to provide such feedback during a class. In our point of view, this approach provides more dynamic classes and improves the communication between teachers and students.

We believe that this is the first work that collects best-practices and usability results for a digital teaching platform in the classroom. In the next section we describe the study in the usability lab, and in Sect. 3 describe further research in classrooms. We then conclude the article highlighting the principal findings from both experiments.

2 Usability Tests

2.1 Methodology

The usability lab study evaluated the DTP with 14 students (1 female and 13 male) with ages between 18 and 35 years old and enrolled in vocational education programs. The study was conducted with one student at a time, and each test took approximately 45 min to 1 h. The task analysis was based on the 5 quality attributes of usability proposed by Nielsen [5]: learnability, efficiency, memorability, errors and satisfaction. The test was designed to verify issues regarding the natural process, and teaching methodology, by emulating a classroom environment: each test was conducted with two mediators, one of whom acted as a teacher to enhance the immersion of the subject. The test simulated a lesson on Brazilian biomes (see Fig. 1), a topic that has nothing to do with the students’ studies, but was indicated as an interesting general topic. The other mediator acted as interviewer and thus played a more traditional role in usability studies.

Fig. 1.
figure 1

The educational content in the DTP Player. Note that the UI is in English to enhance understanding, but we used a localized Portuguese version for the actual tests.

Full size image

Five activities were scripted, that are related to different types of content available in the platform (e.g. watching a video, studying a slideshow, and answering exercises). As this is performed in a usability lab, all movements, speeches, questions and errors in the experience of the student-subjects were recorded. Eye tracking was used to identify attention hotspots in heat maps and gaze plots. After completing all the tasks, each subject was invited to reflect on their general experience, and what went well or not. Each user answered questions regarding their experience in each activity on a Likert scale from 1 to 7.

The study was divided into three phases: introduction (self-exploration), completing the set of tasks, and an interview and debriefing.

  • Phase 1: The experimental process started with a short warm-up. We received the participants and asked them questions about their technology preferences and hobbies, trying to relax them and make them as comfortable as possible. They were invited to explore the system by themselves.

  • Phase 2: Two researchers conducted the experiment, one acting as a teacher and the other monitoring the eye tracker results. The users were invited to sit and be positioned adequately in front of the X2-30 eye tracker [11]. It was quickly explained what would be requested, and what the students would do through the Digital Teaching Platform. The evaluation was conducted following the sequence of tasks divided by the activities described below:

    • – video tasks: tasks related to interacting with the video player

    • – reading texts: tasks related to textual content

    • – slide navigation: tasks related to paging through a gallery of educational slides

    • – navigation: tasks related to navigation, in which the students had to find the teacher’s page

    • – exercises: tasks related to answering exercise questions

  • Phase 3: After completing the tasks all the users were invited to watch their session, which was recorded. The goal was to enhance the results and comprehend why specific areas were the focus of attention. The study was concluded with an interview and questionnaire.

2.2 Results

Watch Video. Most users like to see the title and the description of the video and had no problem watching the video. An example of the focus from eye tracking can be seen in Fig. 2. However, the users did suggest a reorganization of the page: they expect the features “Like”, “Share” and “Notes” below the video, as opposed to next to it; we noticed that YouTubeFootnote 1 is the main reference for the arrangement of elements on the screen. Furthermore, some users expect functionality that is not currently offered, such as easy access to content that they “liked” for future access. Moreover, they would like to be able to take notes while watching the video (the video pauses when the users pull up a keyboard). Another feature that users would like is more cohesion between the different contents, such as direct hyperlinks from the video to texts, or slides, that give more details.

Fig. 2.
figure 2

Heatmap generated by the X2-30 eye tracker during a video task.

Full size image

Reading Texts. All users consider texts to be very important for the application, despite large texts on the screen being unattractive. The digital text —together with the price and a lighter backpack — are mentioned as the most positive and practical points for tablet usage in a classroom, and they leave the users satisfied. The users like to have a zoom feature for reading. According to the users the figures in the text draw more attention while reading. Most users would like to have more options for note taking and highlighting, and mention a tool bar with options to make it possible to select, copy, paste, highlight and bold parts of the text. The users would like to have an option to consult a dictionary for the meaning the words they do not know. Most users would like to send the text to their personal email, or otherwise access the texts outside of the application.

Slide Navigation. Swiping between slides did not cause problems, but users indicated they would like further navigation options, such as a bar with thumbnails and previous/next buttons. Moreover, as with viewing videos, most users want it to be easier to take notes. There was also some confusion of the UI elements, where we noted that the icon for viewing the slides in a timed slide show draws more attention than the icon for viewing the slides in full screen.

Exercises. The users felt there was feedback missing when answering exercises: they were confused that there was no confirmation that their answer had been accepted, or that the quiz did not proceed automatically to the next question. They also missed buttons to navigate forward and backward through the questionnaire, in addition to the current option of clicking on any question to open it. Finally, the users would like the ability to take notes while doing the exercises (for instance, to write down the intermediate solutions in a math question). The users very much liked receiving feedback of their performance — an optional screen for the teacher displays the correct answers and a percentage of the students who gave each answer.

Navigation. The button to navigate straight to the teacher’s page was identified by all, but not associated with that action: most students thought it accessed a special page for the teacher and ignored it. They navigated to the teacher’s page by navigating back to the class overview page and finding the content the teacher had opened. They all identified it as an important functionality once told about its use.

Relevance Map. A relevance map of application was created according to the users expectations. We asked the users how relevant to a classroom content application they judged all activities they had seen, answered on a 7-point likert scale, with 1 being irrelevant, and 7 extremely relevant. The main areas and the “teacher’s page” feature were considered for the analysis. In general, the content and functionality was seen as very relevant by the users. To validate the social features (Like, Share and Notes) a ranking of preferences was created. The users were asked about “what they do after accessing an interesting content”. The ranking was created according to the users decisions and choices. These rankings can be seen in Fig. 3.

Fig. 3.
figure 3

The relevance of all the functionality tested to the educational process, according to the students; and the ranking of what social action they would take with interesting content.

Full size image

3 Field Trials

In the field trials, we tested the platform with teachers and students of middle-school mathematics in three different school environments: two public schools in Manaus, AM, and an institute providing extracurricular school activities for students in Campinas, SP. The students were between 13 and 16 years old. In addition to observing the classes and capturing automatic interactions, we conducted on-the-spot satisfaction surveys asking the participants to fill out a questionnaire after the classes, using the User Engagement Scale [6], and conducted interviews with some of the students and the participating teachers.

The field trial focused on mathematics classes on the topic “functions”, and the content was created by the respective teachers, using a slideshow, video, questionnaire and interactive content. The content was divided into two classes: introductory knowledge and enrichment on the topic. Using this content, the two class hours were used to evaluate two aspects of the DTP:

  • to measure the satisfaction with the DTP on the first contact (first impression);

  • and to evaluate specific usage issues that appeared in a classroom setting.

3.1 Methodology

We used an adaptation of the Experience Sampling Method (ESM) [3], a method originally developed to perceive people’s emotional response to events, and since adapted to measure users’ satisfaction, particularly in situated mobile scenarios [1]. While there are different ways of applying the method, we adopt a straightforward application for situations where both the evaluator and the subject are in the same physical location: the evaluator asks the subject to point out his or her current emotional state (with regards to satisfaction) on a printed card (Fig. 4), which gives a choice of five states ranging from very dissatisfied to very satisfied at the time of the app use.

Fig. 4.
figure 4

The five emotional states the users could indicate on a card for applying the ESM method.

Full size image

In addition to asking students about their emotional state during the class, we asked all participants (including the teacher) to participate in a short self-reporting survey based on the User Engagement Scale (UES). The UES is a self-report measure that builds upon earlier work in the area of educational multimedia. Many applications of the method were found and analyzed by O’Brien and Cairns, who suggested that “the UES is a “good” measure that can assist researchers in capturing the users’ perception of information interactions, and be used in mixed methods studies to help make sense of, for example, behavioral data” — O’Brien and Cairns [6]. The original survey they proposed has 31 questions, which is too long for the quick application of a survey after the class. We adapted the survey to focus on the dimensions that O’Brien and Cairns indicated were most stable, and reduced the survey further by not asking the questions in their negative form, resulting in six questions on the dimensions aesthetics, usability and focused attention, and we added a final question that conflates some of the other dimensions into what we call perceived experience, as a way to measure the user’s engagement with the app. The questions can be found in Table 1. The questions were answered on a Likert scale from 1–5, with a similar emotional aid as in the ESM method.

Table 1. Questions used in the user engagement survey during the field trials, categorized by dimension.
Full size table

Finally, we augmented the self-report methods with observation and interviews with some of the students and the teachers. A summary of the times and methods applied is provided in Fig. 5.

Fig. 5.
figure 5

Overview of the field trials and the evaluation methods applied throughout this experiment

Full size image

3.2 Results

The principal findings are that both the students and the teachers were very satisfied with the usage of the DTP in the classroom, and showed a high level of engagement, in particular in the dimensions of aesthetic appeal and perceived experience, as showed in Fig. 6.

The DTP app is an easy, focused, interactive and richer way to learn, according to the students we asked. One finding was that they reported that in a traditional setting without tablets they had to copy the teacher’s notes manually, whereas on the tablet with the DTP this is unnecessary, and does not “waste their time”: they can concentrate on the content itself and use the time to comment, take their own notes and answer the teacher’s questions. Students think that is also improves the teacher’s performance: “He explains better and students be more engaged”. Finally, the students report that they were required to participate in the exercises: “in the DTP app we must do the exercise. In the textbook we can pretend”.

Specific usability issues also arose, most importantly: (i) the login and material download process is too slow, and cost the teacher minutes that could have been spent teaching. (ii) it is essential that students notes (whether made with a stylus or a keyboard) are prominently displayed together with the content.

A more general observation is that the students (and teachers) expect a more collaborative behavior: they want students to contribute to the class content and activities; proposing exercises, topics, videos and also suggesting approaches to the teachers. All of this appears to be more evident and easy in a digital setting. In their opinion, during the experiment with the DTP, the students interact more with each other because of the tablets. We observed this in particular when the exercises were corrected, and the students’ integration became effusive and very participatory. Creating more tools for sharing ideas and content among students and teachers is a direction we intend to explore in a future version.

Fig. 6.
figure 6

Satisfaction at the moment of use (left) and the outcome of the survey for each dimension (right)

Full size image

4 Conclusion

Our research is preliminary, nevertheless we conclude that despite usability issues of the proof-of-concept, the DTP appears to motivate teachers performance and engage the students in a classroom setting. User feedback indicates that our DTP offers advantages in class time optimization and interactivity. The findings show that tablet usage is an important tool to motivate students in the classroom setting. We must however place a caveat; while the students had all used tablets in the classroom before, it was not common practice, and the novelty may play a part in the positive evaluation: long-term evaluations are scheduled to better evaluate this and to identify additional features to incorporate in the DTP.