Keywords

1 Introduction

Educational robotics has garnered significant interest in recent years to teach students not only the fundamentals of robotics, but also core Computer Science (CS) concepts [5] and Computational Thinking (CT) competencies [2, 3]. However, participation in an ER Learning Activity (ERLA) does not automatically ensure student learning [17], with the design of the activity playing a key role towards the learning outcomes [6]. Indeed, the lack of understanding as to how specific instructional approaches impact student learning in ER activities has been raised at multiple occasions [16]. Many researchers have even evoked the need to have an operational model to understand how to foster CT skills [10] within the context of ER activities [1, 9]. To that effect, Chevalier & Giang et al. [3] developed the Creative Computational Problem Solving (CCPS) model for CT competencies using an iterative design-oriented approach [13] through student observations. The resulting 5 phase model (Fig. 1) helped identify and understand students’ cognitive processes while engaging in ER learning activities aimed to foster CT skills. By analysing the students’ behaviour through the lens of the CCPS model to understand the students’ thought processes, both teachers and researchers may have a means of action and intervention in the classroom to foster the full range of cognitive processes involved in creative computational problem solving. The authors concluded that a validation by teachers was essential to ensure that they could “effectively take advantage of the model for their teaching activities”, not only at the design stage, but also to guide specific interventions during ER learning activities.

This article reports the findings of a study involving 334 in-service and pre-service primary school teachers, with the purpose of evaluating their perception of the model and investigating whether their own needs as users of the model are met. The Utility, Usability and Acceptability framework [18] for computer-based learning environment assessment was leveraged, as it has been previously used for the evaluation of teachers’ perception of the use of educational robots in formal education [4]. More formally, we address the following questions:

  • RQ1: What is the perceived utility of the CCPS model?

  • RQ2: What is the perceived usability of the model?

  • RQ3: What is the acceptability of the model by teachers?

2 Methodology

To evaluate the model, the study was conducted with 232 in-service and 102 pre-service teachers participating in the mandatory training program for Digital Education underway in the Canton Vaud, Switzerland [5] between November 2019 and February 2020. The inclusion of both pre-service and in-service teachers within the context of a mandatory ER training session helps ensure the generalisability of the findings to a larger pool of teachers, and not just experienced teachers and/or pioneers who are interested in ER and/or already actively integrating ER into their practices [4]. During the ER training session, the teachers participated in an ER learning activity (see Lawnmower activity in Fig. 1 [3]) which was mediated by the CCPS model. During this activity, the teachers worked in groups of 2 or 3 to program the event-based Thymio II robot [12] to move across all of the squares in the lawn autonomously. As the robot is event-based, the participants “have to reflect on how to use the robot’s sensors and actuators to generate a desired [behaviour]” [3], which requires that the participants leverage many CT-related competencies. So that teachers understand how the CCPS can be used to mediate an ER learning activity, and similarly to Chevalier & Giang et al. (2020) [3], a temporary access blocking to the programming interface was implemented at regular time intervals.

Fig. 1.
figure 1

The CCPS model (left) and lawnmower activity setup with Thymio (right) [3].

Full size image

After the activity, the teachers participated in a debriefing session where they were asked to express what they had to do to solve the problem and the trainer grouped these comments into categories relating to CT, the CCPS model and transversal skills. The teachers were then presented the CCPS model itself and its 5 phases, together with the results of the study conducted by Chevalier & Giang et al. [3] to provide concrete testimony as to the effectiveness of the model when applied in classrooms. Finally, in an overarching conclusion about fostering CT competencies during ER activities, the trainer provided guidelines on how to design ER learning activities and intervene accordingly.

The Utility, Usability and Acceptability framework [18] then served as the basis for the teachers’ evaluation of the CCPS model. As utility “measures the conformity of the purpose of the device with the users’ needs” [4, 18], in the present context we consider utility with respect to student learning. Two perspectives are adopted : 1) how the use of the model helps foster transversal skills that are part of the mandatory curriculumFootnote 1 and 2) how certain intervention methods may help promote reflection in the different phases of the model. Usability on the other hand considers “the ease of use and applicability of the device” [4, 18] by the teacher, which is why the CCPS model in this case is considered in accordance with the “professional and technical actions” that teachers make use of in their daily practicesFootnote 2. Finally, acceptability “measures the possibility of accessing the device and deciding to use it, the motivation to do so, and the persistence of use despite difficulties” [4, 18]. In this case, we consider acceptability with respect to what the teachers intend to do with the model with increasing levels of appropriation. To measure the aforementioned constructs, a set of questions pertaining to each dimension was developed (see Table 1) with most responses being provided on a 4-point Likert scale (1 - strongly disagree, 2 - disagree, 3 - agree, 4 - strongly agree).

Table 1. Utility, usability and acceptability survey [18]. Utility in terms of transversal skills considers 5 dimensions: collaboration (COL), communication (COM), learning strategies (STRAT), creative thinking (CREA) and reflexive processes (REFL)
Full size table

3 Results and Discussion

3.1 RQ1 - Utility

Educational robotics learning activities are often considered to contribute to the development of a number of transversal skills (e.g., collaboration, problem solving etc...) [2]. While this perception is also shared by teachers who are pioneers in robotics [4], it is important to ensure that ER activities mediated using the CCPS model and designed to foster CT skills, are perceived by teachers at large as contributing to the development of transversal skills. The results of the survey showed that teachers found the ER learning activity with the Thymio useful to engage in transversal skills (Fig. 2), in particular collaboration, reflexive processes, learning strategies and communication, with only creative thinking being less perceived by teachers. This is coherent with the fact that the ER Lawnmower activity was conceived to promote students’ use of transversal skills to help the emergence of related CT competencies and suggests that the use of the CCPS model in designing ER learning activities helps teachers see and strengthen the link between ER, transversal skills and CT, confirming the results of [4] with teachers who were novices. Although in the mandatory curriculum teachers are taught to evaluate transversal skills, little indication is provided as to how to foster them. The use of ER learning activities informed and mediated by the CCPS model can provide a concrete way to foster skills already present in the curriculum and ensure that students acquire the desired competencies.

Fig. 2.
figure 2

Teachers’ perceived utility (with respect to fostering transversal skills), usability and acceptability of the CCPS model. For each transversal skills we report the mean and standard deviation \(\mu \pm \)std, and Cronbach’s \(\alpha \) measure of internal consistency.

Full size image

To clarify the link between the CCPS model and the employed intervention methods, the teachers were asked to select a maximum of three intervention methods that they believed were useful to engage in each of the phases of the CCPS model (see Fig. 3). The element which emerges as the most relevant for all the phases of the model is the possibility of manipulating the robot, thereby reinforcing the role of physical agents in fostering CT skills [8]. This however is dependent on the fact that the Thymio robot provides immediate visual feedback through the LEDs in relation to each sensor’s level of activation. This highlights once more the importance of constructive alignment in ER learning activity design [7] which stipulates the importance of the robot selection in relation to the desired learning outcomes. The second most popular choice was to write down the observations, likely because this constitutes a means of specifying what happens with the robot in the environment. The written observations then become a “thinking tool” that supports modelling and investigation [15]. Surprisingly, and although the teachers were introduced to the fact that unregulated access to the programming interface tends to lead to trial and error behaviour [3], programming was often selected as being useful to foster the different CT phases, whilst not being able to program was one of the least selected. Only in the case of idea generation did both programming and not programming receive an equal number of votes. We believe that the frequent selection of programming is due to the fact that the question was based on their experience as the participants, and therefore the need for a high sense of controllability [14], in the ER learning activity and not on their experience as a teacher leading the activity in the classroom.

Fig. 3.
figure 3

Teachers’ perception of the link between intervention methods and the different phases of the CCPS model. For each phase of the model and intervention method, the proportion of teachers having selected the approach as relevant is shown.

Full size image

To summarise, on the one hand, teachers perceive the usefulness of promoting transversal skills that they are familiar with, as they are already part of the curriculum. On the other hand, they do not perceive what research has shown to be useful to promote CT competencies, likely because it was not part of the curriculum until now. Therefore, experimentation in their classrooms is necessary, as well as further training to help them acquire a more critical view of ER learning activities and to understand the impact of specific intervention methods on the development of students’ skills.

3.2 RQ2 - Usability

With respect to the usability of the CCPS model in the teaching profession, responses were globally positive (see Fig. 2, \(\mu = 2.89\), std \(= 0.78\), Cronbach’s \(\alpha = 0.82\)). Teachers believed that the CCPS model could be used to plan and intervene during an ER learning activity (over 80% of positive responses), likely due to the guidelines provided during the theoretical presentation. However, the link between the CCPS model and student learning was less evident for teachers: 66% believed it could be used to regulate student learning and 60% that it could be used to evaluate student learning. Both constructs are related by the need to assess students and understand where they stand in terms of the overall learning objectives. Although this shows that teachers need to be taught how to identify the phases in which the students are to be able to use the CCPS model to its full extent, this is also highly linked to the difficulty found in both the ER and CT literature in terms of assessment of learning and/or transversal skills [8, 9].

3.3 RQ3 - Acceptability

The question of acceptability here targets teachers’ intent to use the CCPS model in their practices. Intent to use is considered at progressive levels of appropriation (see Fig. 2), which is why it is not surprising to find that while teachers might be willing to conduct the same ER learning activity in their classrooms (64%), less are willing to adapt the activity (40%), create their own custom one (32%), and conduct a more complex one (20%). One can put this in relation with the Use-Modify-Create (UMC) progression [11] which was developed to scaffold student learning in CT contexts. Teachers need to start by using the model in a given ER learning activity to gain in self-efficacy. Only then will they be able to progress to the next stage where they feel comfortable adapting the use of the model to their teaching style and to the individual students. Finally, teachers will reach a level where they create their own ER pedagogical interventions to foster CT competencies. One must however note that intent is likely influenced by external factors (e.g. time or access to the robots, frequent barriers to the introduction of ER in formal education [4, 5]).

4 Conclusion

Provided the prominent role that teachers play in the integration of ER and CT in formal education, this study investigated teachers’ perception of an operational model to foster Computational Thinking (CT) competencies through ER activities: the Creative Computational Problem Solving (CCPS) model [3]. Three research questions were considered in a study with 334 pre-service and in-service primary school teachers: What is the perceived utility (RQ1), usability (RQ2) and acceptability (RQ3) of the CCPS model? While teachers found that the activity design and intervention methods employed were useful to foster transversal skills (RQ1), their perception of the utility of the intervention methods on the different cognitive processes defined by the CCPS model (RQ1) was somewhat unexpected. In terms of the usability (RQ2), teachers perceived how they could design an activity and intervene using the model, but were less able to perceive how the model could be used to assess where the students were in terms of learning and regulate the activity to mediate their learning. The findings of RQ1 and RQ2 support the importance of training teachers to recognise and understand the different cognitive processes to intervene adequately and be able to differentiate their teaching per student, rather than adopting a unique strategy for an entire class.

To help teachers implement ER learning activities in the classroom and gain in autonomy to create their own activities that foster CT skills (RQ3), it seems relevant to alternate between experimentation in classrooms and debriefing during teacher training and go beyond providing pedagogical resources. To conclude, the operationalisation of ER to foster CT skills must also consider the key role that teachers have to play in the introduction of any such model and its application in formal education.