1 Introduction

Development and acquisition of the teamwork competence (TWC) is essential in educational and professional contexts because: (1) teamwork involves sharing of information and discussion among students to build mental models in a cooperative way, ultimately contributing to the improvement of students’ learning [1, 2]; (2) companies seek that prospective employees have developed the TWC because members of an organization are working together in groups to achieve common goals [3]; (3) the application of the Bologna process positions TWC as a key competence that student should develop in Higher Education.

Despite all the institutional efforts to promote and foster TWC, assessment of TWC acquisition is a difficult task. Most of the time, the assessment of TWC is determined by the final result or work that teams deliver, ignoring what happens during the different stages of the process, variability between the amount and relevance of individual contributions, or without any consideration about the different stages of the teamwork process.

From the above, a systematic and scientific study and assessment of TWC requires methodologies that facilitate quantitative assessment. The present work describes the application and evaluation of a methodology (CTMTC, Comprehensive Training Model of the Teamwork Competence) that both improves TWC and facilitates its objective assessment. CTMTC takes into account the group results and how each individual has acquired the competence. The methodology relies on the analysis of learning evidences from data generated by the use of IT-based learning tools by student teams during a project development [4]. The application of the methodology entails that teams develop the project in several stages adapted from the International Project Management Association (IPMA) [5, 6].

The objectives of this research are to present and compare the tools used in courses following the CTMTC methodology, to evaluate students’ perception about the implementation of the CTMTC, and to show how the CTMTC contributes to improve TWC and student assessment in teamwork-based courses. In order to do so, the study presents the application of the CTMTC methodology to three degree and one masters’ courses, describes the implementation of the CTMTC on those courses, and compares the results with previous editions of the courses that did not incorporate the CTMTC.

The study is structured as follows. Section 2 describes the CTMTC methodology. Section 3 explains the practical implementation of CTMTC in the courses. Section 4 illustrates in more detail one of these implementations. Section 5 compares the implementation described in Sect. 4 with outcomes from previous editions of the course and evaluates the methodology implementation from a qualitative perspective. Finally, Sect. 6 summarizes the main results and the conclusions derived from the research.

2 The CTMTC Methodology

TWC assessment is complex, especially when examining how the individual acquires this competence. There are several methods to assess TWC acquisition but they focus mainly in the assessment of group outcomes. Such methods evaluate the individual acquisition of the competence relies on observation of students’ perceptions through opinion surveys, self-evaluation questionnaires or peer evaluations to measure individual evidences [7–9]. Ultimately, these methods evaluate the knowledge students have about their task, their peers’ tasks, and project goals and milestones. However, this approach may not be the best, because other elements have a high impact in group outcomes: leader behavior, cooperation between peers, problems between team members, performance of each member, etc. [10]. It is difficult to collect this kind of information, which requires a continuous monitoring of how the group carries out the activity. In order to ease and streamline this process, this study presents CTMTC and its associated tools.

CTMTC is a proactive method that draws on three aspects of group-based learning: teamwork phases (mission and goals, responsibility maps, planning, implementation and organization of documentation), collaborative creation of knowledge, and cloud computing technologies (wikis, forums, social networks and cloud storage systems) [4]. In the CTMTC, faculty continuously monitors team members’ collaboration and individual evidences along the teamwork phases. Monitoring also enables teachers to guide students’ individual learning. CTMTC allows teachers to do partial summative assessments of TWC [11]. This method has been tested in prior settings [10–13].

Assessment of students’ individual learning evidences requires a continuous exploration of what they are doing during the learning experience, and a global analysis once the course finishes. Controlled environments with a small number of students allow teachers to easily perform monitoring and analysis but in courses with higher number of students, it requires the use of automated tools due to its time-consuming nature. This study includes one of these applications, designed as a learning analytics tool that accesses, analyzes and structures the information about students’ interactions stored in Learning Management Systems (LMS). Analysis and presentation of the information follows the requirements of CTMTC: the learning analytics tool presents the information about the group members’ forum interaction organized by course, group, thread and person. This information allows instructors to check the evolution of the group, average interaction of its members and the level of cooperation between them. The tool also shows each member’s individual participation in a thread, if the leader is acting as expected, etc. The main benefit of the application of this tool is that it reduces the effort of assessing group and individual interaction and it makes it easier to apply CTMTC [13–15]. As next section shows, the application of the methodology and the technologies used in each of the courses vary from one to another.

3 Implementation of the CTMTC Methodology

All the courses under examination follow the CTMTC, with teamwork occurring online along the course (using forums and wikis/Dropbox to communicate and share contents/files, respectively). Every four weeks the students must make a presentation of their work. Apart from interacting with other group members, students interact with the different technological systems used. These interactions leave traces that teachers may use as evidences, such as forum posts and WhatsApp messages, achievement of the CTMTC phases (mission and goals, map of responsibility, planning, etc.) and outcomes (final deliverables). At the end of the process, students must publish the final result of their work. Optionally, they can record a video (with a maximum length of 10 min) to present the result, describe the way the team organized and coordinated, and show individual and group work evidences.

One of the main benefits of the CTMTC is its flexibility. For this study, the authors adapted it to the specific needs and features of each of the courses where CTMTC implementation was decided at the University of León (4 completed courses, with 5 more ongoing courses using CTMTC at this moment). The four completed courses are part of the Bachelor of Science and Master of Science degrees on Computer Science. Each course has a different number of students and specific course goals, and therefore the CTMTC methodology needs adaptation to adhere to the requirements of each course.

  • Operating Systems. It is a second year course of the Bachelor of Science on Computer Science degree, with a total of 110 students. The application of CTMTC focuses on one activity that has a weight of 22 % of the final score. Although choice of team members and coordinators is open, the group must choose one of the three possible topics for the work. Groups have 3 or 4 members, who use the LMS forums to interact between them; additionally, some of the students also use instant messaging tools such as WhatsApp. Each group publishes its partial outcomes in the LMS wiki and deliver their final outcome using Moodle LMS assignment block.

  • Accessibility. It is a fourth year course of the Bachelor of Science on Computer Science degree, with a total of 71 students. In this course, the application of the CTMTC focuses on one activity with a 60 % weight of the final grade. The instructor determines the composition of the groups, but students can freely designate their coordinator/leader and choose the topic between different possible projects. Team members use the LMS forums and Wiki for interaction and publication of final results, respectively. Optionally, they may also send partial results of their work to the teacher.

  • Dedicated and Embedded Architectures. This is a fourth year course of the Bachelor of Science on Computer Science degree, with a total of 15 students. CTMTC covers all course activities. Groups consist of a number of members between 2 and 4, for a total of 6 groups. Students can freely choose their group and their coordinator. The project involves the same task for all groups. Students interact through a forum, use the wiki to publish their results and Moodle assignment block to deliver the final project.

  • High Performance Computing. It is a first year course of the Masters in Computer Science degree, with a total of 8 students. CTMTC is applied to a task with a weight of 60 % of the final grade. Given the low number of students, they were all included in one group, and student tracking did not require the use of learning analytics tools. Group members freely choose their coordinator. The choice of tools to use to complete the project is also open. The application of CTMTC includes GitHub as a repository, Moodle Wikis as a means to publish group results and gitter as a space for communication and interaction.

The above list shows the adaptation of CTMTC to the specific characteristics of each course. The following section describes with a higher level of detail one of the courses and compares the outcomes after application of CTMTC with those of previous editions of the course.

4 The Application of CTMTC in the Operating Systems Course

Operating Systems is a first semester course in the second year of the Bachelor of Science on Computer Science degree. In the past five years, there were more than 100 students enrolled per year. The course duration is 60 face-to-face hours (20 of them are theory and the remaining 40 h correspond to lab classes). The objective of the course is the acquisition of basic knowledge about how an operating system works, and therefore it is strongly oriented to practice. The main topics covered include abstraction of process, and analysis of problems related with concurrency and process scheduling. Completion of tasks requires the use of C Programming Language and processes and threads libraries.

Students’ final grade comprises two parts: theory (35 % of the final grade) and practice (65 % of the final grade). Practice includes two assignments: the interim assignment (35 % of the final practice grade) and the final assignment (65 % of the practical grade). The application of the CTMTC focuses on the interim assignment (with a total weight of 22.8 % over the final grade). Interestingly, the interim deliverable is not mandatory (in other words, students can pass the course regardless of completing it or not).

The 106 participating students freely formed groups (a total of 28 groups), with 3–4 members per group, and they named the person in charge of group coordination.

The teachers proposed three possible activities/projects for the groups to choose, and thus different groups could have the same project as final assignment. Each of these groups that completed the interim assignment had to follow the phases of the CTMTC methodology, using Moodle Forum as main space for interaction. In the Moodle Forum a group member can only read messages and threads created by their group partners, and teachers may read all the messages in every group, and also read them on a per group basis (as an example, Fig. 1 shows the list of threads for group 16). When the interaction between students takes place in other systems, such as WhatsApp for instant messaging, students have to upload the conversation as attached files in forum posts.

Fig. 1.
figure 1

Group 16 interactions in the Moodle Forum

Full size image

Students display the outcomes of the activities they complete on a wiki. The wiki is private for group members and teachers. The work done in the wiki has to incorporate the phases described in CTMTC, information about the activity and the name of the group coordinator. Each phase is associated to a particular page in the wiki and a link to the forum thread where the discussion about the issue took place. Figure 2 shows one of these wikis. Finally, the students deliver the final result of their project through a Moodle assignment block.

Fig. 2.
figure 2

Wiki including group 16’s CTMTC outcomes

Full size image

For an effective development of the course, teachers need not only to assess the final outcomes of each group, but also the outcomes of each member of the group. In order to do so the teachers can use the forum to see group members’ interaction, and the wiki and a Moodle assignment to see the outcomes of the group work. The vast amount of data available makes it difficult to analyze these evidences. For example, this course has 28 wikis (with more than 4 pages each) and a forum with 245 threads and 1520 posts. Manually analyzing the work of a group and its group members is a time-consuming task for instructors. For example, a manual inspection of each group’s activity takes lasted between 40 min and 1 h (this time does not include assessment). That is, it requires between 19 and 28 h to complete the inspection of every group in the course.

In order to face this problem, course teachers use a learning analytics tool that provides a quick and effortlessly way to retrieve the information required for assessment of groups’ and individuals’ outcomes (Fig. 3 depicts general information about course and groups returned by the learning analytic tool for the interim assignment).

Fig. 3.
figure 3

Information returned by the learning analytics tool in the Operating Systems course

Full size image

Two rubrics enable student assessment of both group and individual work. The rubric for group work assessment builds on prior studies using the CTMTC [4, 10, 12, 13], while the rubric for individual assessment focuses on individual acquisition of TWC and takes into account previous research related to teamwork behavior [10, 12, 13]. In this sense, teamwork behaviour refers to performing individual activities that contribute to the team work process: interpersonal behaviours (conflict and problem solving, collaboration, communication) and management behaviours (assuming leadership, establishing goals, planning tasks, coordinating the other members in the group) [16, 17].

The next section evaluates the application of CTMTC in the Operating Systems course, quantitatively (final grades) and qualitatively (student perceptions about CTMTC), and compares the results with those obtained in previous editions of the course that did not incorporate CTMTC.

5 Case Study

Table 1 shows participation, average assignment grades and average course grades in the last three editions of the course (the most recent being the one implementing CTMTC). Regarding participation, Table 1 reveals high participation levels in the interim assignment in previous editions (around 80 %), rising to nearly complete participation (96.36 %) after application of the CTMTC. The observation of assignment average grade returns interesting results because the value in the 2014/15 edition of the course is much higher than the one from the previous year and the one from the CTMTC course; however, it is worth noting that the cause of this unexpected result might be that the 2014/15 edition used peer-review methods. Finally, Table 1 also shows that average course grade is higher in the course using CTMTC. A possible explanation is that the phases defined by CTMTC (i.e., definition of the mission and goals, planning, etc.) may help to better plan and manage the group project. Because the study only includes one edition of the course using the CTMTC, future editions of the course using the CTMTC would confirm these results.

Table 1. Participation, average assignment grade and average course grade of the last three editions of the course Operating Systems.
Full size table

As mentioned earlier, a qualitative evaluation of the CTMTC in this study also entails students’ perception about the application of the methodology. In order to collect this information participants were asked three open questions:

  1. Q1.

    “In your opinion, what are the main advantages of the CTMTC?”

  2. Q2.

    “What are the main problems you have encountered with the use of CTMTC?”

  3. Q3.

    “What additional tools/systems (not included in the CTMTC) have you used in the course?”

The qualitative analysis consists of an examination of the text from the responses given by participants. This procedure includes grouping responses based on topic-proximity criteria for Q1 (advantages of the application of the methodology or the tools employed by CTMTC), Q2 (problems with application of the methodology), and Q3 (tools used for assignment completion). After classification, we combined the results in a matrix in order to extract conclusions [18]. Table 2 shows the first 20 (out of 106) responses to the three questions.

Table 2. Students’ perception about the use of CTMTC (truncated to N = 20 students)
Full size table

Even though Table 2 just shows 20 results, most of the students perceive the methodology as something positive, highlighting advantages related to teamwork behaviors, as described by Tasa et al. [17]. Some responses correspond to interpersonal behaviors (communication, problem solving, collaboration, organization, knowledge sharing, helping other group members, etc.) and other have to do with management behavior (distribution of tasks, deadlines, publication of results, planning, leadership, etc.).

Students perceive that under the CTMTC they actually work as a group, and therefore the methodology helps them to successfully complete the project. Interestingly, participants are second year degree students with no prior knowledge about project management, and thus they perceive the idea of working together and organizing their work as something positive. On the other hand, a group of students do not find the methodology useful, showing a preference for individual work – three of them were part of a group where they did not had any prior relationship with the rest of the group members–, and only 9 out of 106 students (8.49 %) have negative opinions regarding the usefulness of the methodology.

More than a third of participants (34.5 %) state that they do not encounter any problem with CTMTC, while 44 % of students claim they have problems related to the communication tools used in the course. The latter group finds that forums cannot accurately represent actual interaction between group members, and express their preference for integration of instant messaging tools (messaging tools are allowed as group work evidence on the condition that students upload their conversations to the forum, an option that only 2 of the 28 groups chose). Some students (13 % of the total) report problems with other members of the group. The main causes of these problems are perception of different levels of implication between members, unfair/uneven distribution of tasks, and missing or nearly missing deadlines. These are all common problems related to coordination and teamwork. Other problems mentioned include higher complexity and difficulty to apply the methodology than project difficulty, unfair grade weighing for leadership because group leaders have more workload, or lack of knowledge about what they should publish on the wiki.

Concerning the use of tools, 73.6 % of participants report satisfaction with the tools that application of CTMTC demands. A majority of the rest suggest the use of instant messaging tools that provide information about real time interaction instead of using the forum for interaction reporting. A reason for this perception might be that the University of Leon’s Moodle instance does not publish the post immediately after students post it, and therefore there is a delay that slows down communication. Finally, six students propose the use of code repositories for each group member to upload their code so teachers may check out individual contributions.

6 Conclusions

TWC is essential for educational institutions and in order to improve the employability of students. This study describes and evaluates the application of the CTMTC methodology for student assessment in group project-based learning in Higher Education.

The study is part of a research project aiming to compare the effectiveness of different methodologies applied to courses that involve work group and collaborative learning. The research focuses on the application of CTMTC and reveals how its flexibility enables adaptation to courses with very different characteristics (e.g. number of students, types of assignments) by allowing the integration of different tools. The research details a case study of one specific context of application of the CTMTC.

The case study suggests that application of CTMTC helps increasing students’ participation and final grades. Furthermore, students have a positive perception of the application of CTMTC, and they feel that it helps them improve project management, planning, distribution of tasks, and setting of deadlines and milestones. This perception suggests that CTMTC may help developing teamwork behaviors that students shall require in other courses and in their jobs in the future [17]. Some students report that some of the problems related to teamwork, such as communication, collaboration, and motivation, are not solved by the application of CTMTC. Even though most of the students claim that the tools used in the course are appropriate for the methodology, some of them point out problems that relate directly to the tools required for the application of CTMTC. From a course design view, instructors should find ways to find complementary technical solutions that cover this gap and address this issue in following editions of the course.

This study is part of an ongoing and constantly evolving line of work on course design and implementation of methodologies to improve collaborative project-based learning. Future avenues of research should address the influence of project selection, a characterization of how students use the tools (and which ones best suit their knowledge and abilities), or teachers’ perception of the application of CTMTC. It would also be of interest to establish a comparison between the results returned by the learning analytics system and other tools used for TWC assessment, such as self-perception questionnaires and peer review techniques [7–9].