Elsevier

Computers & Education

Volume 90, 1 December 2015, Pages 80-94
Computers & Education

Teacher regulation of cognitive activities during student collaboration: Effects of learning analytics

https://doi.org/10.1016/j.compedu.2015.09.006Get rights and content

Highlights

  • It was examined whether learning analytics (LA) support teachers during CSCL.

  • LA visualized students' cognitive activities.

  • LA did not improve detection of students' problems nor lowered cognitive load.

  • LA increased the frequency and probability of teacher interventions.

  • It is hypothesized that LA increase teachers' confidence of their diagnoses.

Abstract

By collaboratively solving a task, students are challenged to share ideas, express their thoughts, and engage in discussion. Collaborating groups of students may encounter problems concerning cognitive activities (such as a misunderstanding of the task material). If these problems are not addressed and resolved in time, the collaborative process is hindered. The teacher plays an important role in monitoring and solving the occurrence of problems. To provide adaptive support, teachers continuously have to be aware of students' activities in order to identify relevant events, including those that require intervention. Because the amount of available information is high, teachers may be supported by learning analytics. The present experimental study (n = 40) explored the effect of two learning analytics tools (the Concept Trail and Progress Statistics) that give information about students' cognitive activities. The results showed that when teachers had access to learning analytics, they were not better at detecting problematic groups, but they did offer more support in general, and more specifically targeted groups that experienced problems. This could indicate that learning analytics increase teachers' confidence to act, which in turn means students could benefit more from the teacher's presence.

Introduction

Computer-supported collaborative learning (CSCL) is an instructional strategy in which collaboration among students is supported by technology. It is based on the idea that collaboration is beneficial for learning. By collaboratively solving a task, students are challenged to share ideas, express their thoughts, and engage in discussion (Stahl, Koschmann, & Suthers, 2006). Learning during CSCL is seen as an interactive, constructive, and largely self-regulated process. Students' learning activities can be categorized into cognitive activities (i.e., related to the content of the task, for example structuring and analyzing task material), social activities (for example, the occurrence of discussion in terms of agreement and disagreement and participation rates of group members), and regulative activities at both the cognitive and social level (for example, discussing strategies for solving the task) (Janssen et al., 2007, Kaendler et al., 2014, Vermunt and Verloop, 1999, Weinberger and Fischer, 2006).

Digital learning environments designed for collaborative learning generally integrate tools for carrying out the task as well as for communication between group members. Together, these tools facilitate the types of student activities mentioned above because they support the sharing of resources and provide an opportunity for communication within the group (Erkens, Jaspers, Prangsma, & Kanselaar, 2005). In the present study, collaboration occurs through a digital learning environment in which students have access to task materials, share a text-editor with their group members, and communicate through a chat facility. Providing these tools, however, does not guarantee that students will adequately finish their task, nor a high quality of discussions (Kirschner and Erkens, 2013, Pargman, 2003, Rummel and Spada, 2005). During CSCL, teachers act as a facilitator of students' activities (Kaendler et al., 2014). Teachers can for example offer thoughts that deepen or broaden the discussion and keep track of the progress that groups of students are making on the task. To do so, it is important that teachers are able to identify all relevant events, including those that require intervention. Because of the generally rapid pace of activities within synchronous CSCL settings and the large amount of available information, supporting student activities is a demanding task. In the present study, we focus on teacher regulation of groups' cognitive activities, which are important because they are directly related to for example knowledge acquisition and of which it is known that students may experience problems (Weinberger & Fischer, 2006). We explore a way of supporting the teacher, namely by visualizations of the collaborating groups' activities. The sections below describe students' cognitive activities, the teacher's role during CSCL, and how the teacher may be supported while regulating students' cognitive activities.

The present article is situated in the context of a collaborative writing task. Groups of students in secondary education synchronously communicate with each other through a chat tool and share a text editor to write an essay based on historical sources, which are all provided within the learning environment. The cognitive activities involved in this task include evaluating and discussing the task material, writing the essay, and reading historical sources. At the level of regulative activities, the groups have to agree on a strategy for completing the task and to monitor their progress. As stated before, students largely self-regulate their activities, but it is known that problems may occur that could negatively influence students' learning gains or the quality of the group product. Two of those problems are described in this section.

The first problem concerns discussion of task material within groups. Researchers generally distinguish between on-task and off-task communication within group discussions (see De Wever, Schellens, Valcke, & Van Keer, 2006, for a review). Discussing the content of the task is most clearly related to knowledge acquisition (Weinberger and Fischer, 2006, Cohen, 1994; see also Carroll's Time-On-Task hypothesis, Carroll, 1963, quoted in Baker, Corbett, Koedinger, & Wagner, 2004). Because of the informal character of synchronous chat communication, students may stray off-task during discussions, which could lead to decreased learning gains. When groups do stay on-task, there is another potential difficulty, namely that the discussion has insufficient breadth (Baker, Andriessen, Lund, Van Amelsvoort, & Quignard, 2007). That is, discussions may be superficial or one-sided when the topic of the discussion lingers on only a limited set of the concepts that are relevant to the task. Limited breadth of discussion could also mean less depth, because the students did not take into account all possible explanations or viewpoints and did not connect these views to each other (Baker et al., 2007). So, the content of group discussions, in terms of on- and off-task behavior and the concept coverage or breadth of the discussion, is one cognitive aspect that teachers can help students to regulate.

The second problem is concerned with how students alternate between cognitive activities. While solving the task, students continuously alternate between writing and discussing (Rummel & Spada, 2005), and engage in activities such as outlining, composing, and reviewing the written text. The groups of students may choose different strategies for writing, such as parallel exploration of the material followed by integration of ideas, or continuous joint construction of text (Onrubia & Engel, 2009). For all strategies, it is important that time is managed in an adequate way. Groups may thus need help to monitor their progress while they engage in the multiple cognitive activities involved with collaborative writing.

If these problems are not addressed and resolved in time, the collaborative process is hindered. The teacher plays an important role in monitoring and solving the occurrence of problems as will be explained below.

The change toward the use of collaborative learning in education also requires changes on the part of teachers. In case of CSCL, teacher regulation takes shape by monitoring the learning activities of students as they independently work with other students on their group assignments, and intervening with feedback and assistance when needed (Anderson et al., 2001, Kaendler et al., 2014). When the educational goals are to analyze, evaluate, and synthesize knowledge, leading the students towards interaction and experimentation, teacher regulation is more loose (Salinas, 2008, Vermunt and Vermetten, 2004). Even though there is more loose teacher regulation during CSCL, the teacher maintains an important role (Kaendler et al., 2014). One of the teacher's tasks is to monitor the occurrence of problems and to help to resolve them. When problems arise or students do not make enough progress, teachers can offer their assistance. Many researchers have tried to analyze the effects of teaching activities on learning outcomes or the quality of group products during CSCL (for example Hsieh & Tsai, 2012 and Onrubia & Engel, 2012, see Van Leeuwen, Janssen, Erkens, & Brekelmans, 2013, for an overview). From these studies, it appears that the effectiveness of teaching is largely determined by the adaptivity (content and timing) of teacher interventions (Gibbs & Simpson, 2004). Each group of students has different needs, to which the teacher should adapt (Van Leeuwen et al., 2013, Coll et al., 2014). Thus, as a result of correctly timed and correctly chosen interventions, teacher regulation of CSCL can effectively help collaborating groups.

To provide adaptive support, teachers have to continuously be aware of students' activities in order to identify relevant events, including those that require intervention (Kaendler et al., 2014). The teacher has to maintain an overview of events, a characteristic that in subsequent sections will be called a diagnosis of the current situation. From the previous description of the cognitive activities, it becomes apparent that diagnosing those activities is a demanding task. There are multiple groups to monitor, and to work on the group assignment each of them engages in multiple activities in multiple tools. The pace of the groups' discussions can be fast. Like regular classrooms, the simultaneity of the situation (Doyle, 2006) means that there is a large amount of information to tend to. This may lead teachers to experience high cognitive load, i.e., diagnosing multiple groups of students demands a large amount of cognitive capacity (Paas, Tuovinen, Tabbers, & Van Gerven, 2003). When cognitive load is too high, this may cause teachers to fall back on automated courses of action, which means accurate and up-to-date diagnosing of student performance does not occur (Feldon, 2007). A possible consequence is that teachers use their existing knowledge about students to decide on an intervention instead of using information about the current situation (Feldon, 2007, Schwarz and Asterhan, 2011). This could mean that the intervention is not optimally aligned with students' needs. It is therefore expected that an increase in cognitive load could lead to less adaptation to students' needs.

To lower cognitive load, teachers may be supported by the addition of learning analytics tools. At the intersection of technical analysis of data and the learning sciences, learning analytics (LA) is defined as “the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs” (Siemens & Gasevic, 2012). In the present case, this means that teaching may be supported by analyzing student activities and reporting these analyses back to the teacher. When student activities are summarized and visualized in an easily understandable way, this could lower the time and effort needed by teachers to monitor all groups of students' activities. The LA can enhance the teacher's knowledge or awareness of each groups' activities (De Groot et al., 2007, Voyiatzaki and Avouris, 2014).

A large body of research has focused on supporting students with tools that enhance students' awareness of the activities of their group members. So called group awareness tools can provide students with information about social or cognitive characteristics of the group such as knowledge, attention and attitudes of each group member (Buder & Bodemer, 2008). These awareness tools may guide students in their collaboration by stimulating discussion of the displayed information (Dehler, Bodemer, Buder, & Hesse, 2011) and by giving the groups of students feedback about the state of the collaboration concerning a range of indicators (Harrer et al., 2006, Jermann et al., 2005).

Partly, the functions that group awareness tools fulfill for students are also applicable to the teacher, because both students and teachers can use them to monitor the students' activities (Verbert et al., 2014). However, such tools can also have a function specifically for either students or teachers. For example, consider a LA tool that visualizes and compares a characteristic of collaboration in each group (for example, the number of solved tasks). Students and teachers will derive information from this tool for different purposes. Shown to the students, such an overview for example creates opportunities for social comparison, which might motivate students to set higher standards for themselves (Janssen et al., 2007, Michinov and Primois, 2005). The teacher, on the other hand, uses the information to maintain an overview of the class and possibly as an indication of which groups might need additional help. As stated, this way the LA can contribute to lowering the cognitive load associated with monitoring the activities of multiple groups at the same time. Many articles have described how such tools that support teachers may be designed, making use of techniques such as data mining to discover patterns between student activities in learning environments and their resulting learning gains (cf. Jermann et al., 2005, Romero and Ventura, 2010). However, the step of actually providing teachers with these tools and, consequently, studying how they are used, is rarely taken (Chatti et al., 2012, Papamitsiou and Economides, 2014). A few exceptions are described here. Dyckhoff, Zielke, Bültmann, Chatti, and Schroeder (2012) developed diagnostic tools in collaboration with teachers. Teacher interviews revealed that the tools were perceived as being useful, especially because it allowed teachers to observe whether changes in learning materials or teaching strategies led to changes in student behavior. Mazza and Dimitrova (2007) also created diagnostic tools, and besides interviewing teachers also performed a small scale experimental study (with 6 participants) which showed that teachers could more quickly and more accurately gain an overview of students' activities when they used the tools. Casamayor, Amandi, and Campo (2009) found that when a teacher was notified of possibly problematic situations during students' collaboration, the teacher intervened more and was able to solve more conflicts. A qualitative study that focused both on diagnosing and intervening was conducted by Schwarz and Asterhan (2011). These authors described the choices of a single teacher concerning diagnosis and interventions and how the LA tools support the teacher to achieve adaptivity (“approximate attunement”, p. 436) to the needs of each group. In particular, the tools gave the teacher an overview of activities, which resulted in the mental space needed to carefully consider the appropriate intervention at the appropriate time. Similarly, Chounta and Avouris (2014) found that as class size increased, teachers had more need of LA to retain overview. After teachers focused on a particular group or on an individual student, the LA helped them to regain an overview of the rest of the class. Thus, LA seem to be a promising direction for supporting teachers during regulation of synchronous CSCL. However, research in this area is still scarce and primarily small scale. There is thus a need for further systematic examination of the effects of LA on teacher regulation of CSCL. The goal of this article is to add to this knowledge base, specifically focusing on the effects of LA on teacher diagnosing and intervening. The effects of two learning analytics tools are studied that provide teachers with information about students' cognitive activities. Two teacher supporting tools were added to the existing collaborative learning environment called VCRI (explained in more detail in the method section).

The first tool, called the Concept Trail (CT), is a timeline that displays the topics that groups are discussing in the Chat window. The CT marks the occurrence of a predefined set of task-related concepts and their synonyms by putting dots on the timeline. The development of the topics in the group discussion therefore becomes visible at a glance. As the group discussion progresses, older Chat messages will move upwards out of sight, but the CT still displays the entire history. The CT is integrated into the existing Chat windows, so that the development of the discussion becomes visible each time the teacher looks at a group's discussion.

Fig. 1 shows a screenshot of a Chat window with the integrated CT on top. On the left side of the CT, five concepts are displayed. The dots on the timeline indicate that the concept is mentioned in the Chat conversation. Furthermore, in the conversation itself the concepts are highlighted in yellow, so that it is immediately visible in what part of the message the concept was used. The grey bar within the CT corresponds to the current timeframe of the Chat conversation and can be moved by the teacher so that the teacher can easily jump to another episode within the conversation.

The CT gives information about cognitive activities in two ways. First of all, it can inform the teacher whether a group is on-task or not. As was described earlier, the more students are on-task, the more likely they are to acquire knowledge and to engage in meaningful discussion. Secondly, the teacher can not only monitor how often concepts are mentioned, but also the concept coverage of the discussion. For example, it might be the case that the group only covers one or two important aspects of the task, and that there are no dots on the timeline for the other concepts. It could also be the case that dots are displayed for multiple concepts within the same time frame, which could indicate that students try to connect concepts to each other. The CT thus allows the teacher to monitor the discussion and to give suggestions to deepen or broaden the discussion by mentioning specific concepts. The resulting increased specificity of interventions is desirable because it helps learners to understand how well they are performing and what still needs to be accomplished (Voerman, Meijer, Korthagen, & Simons, 2012).

The second teacher supporting tool is the availability of Progress Statistics (PS), which informs teachers of the progress of each group in terms of the number of written words in the text editor and the chat tool. Fig. 2 displays a screenshot of the PS of the collaboratively written texts. The five bars represent the progress of the five groups concerning the number of words they have written in the text editor. The horizontal line indicates the class average. The same type of PS are available for the number of words posted in the chat conversation, which can be selected from the menu on the left.

The PS give teachers an indication of the amount of activity within the groups. The class average enables teachers to see at a glance whether there are any groups that show deviating amounts of activity, i.e., groups that are behind as well as those that have more written text or more chat utterances than the others. Also, teachers may compare the activity in the chat with the activity in the text editor for each group, thereby monitoring whether groups balance their activities between writing and discussing.

It is important to note two things about the teacher supporting tools. First of all, the tools on their own do not give a complete picture of all groups' cognitive activities. The quality of the written texts, for example, does not necessarily correlate with the amount of words written. Neither does mentioning a concept mean that a student understands its content. Together, however, the tools give an aggregated overview of the situation. Although the tools are based on relatively simple measures, the information shown can be an indication for the teacher whether more time needs to be spent on a particular group. This brings us to the second point. The supporting tools do what their name suggests: support the teacher, but not replace or decide for the teacher. Whether intervention is necessary remains to the teacher to decide.

The following research questions were formulated:

What is the effect of teacher supporting tools that show information on concept coverage in discussions and task progress of collaborating groups on:

  • 1)

    Teachers' diagnosis of concept coverage within discussions and the task progress of collaborating groups?

  • 2)

    The frequency, focus, and specificity of teacher interventions?

  • 3)

    Teachers' self-reported cognitive load?

Section snippets

Design

An experimental study with a between and within subjects design was conducted to test the effects of the Concept Trail and the Progress Statistics. Participants were randomly assigned to the control (no LA tools) or the experimental (LA tools) condition. In both conditions, participants went through vignettes of three collaborative situations. The independent variable is therefore the absence or presence of LA tools. The dependent variables are the participants' diagnosis, interventions and

Results

First, information is given about the usage of the supporting tools (the Concept Trail (CT) and the Progress Statistics (PS)). Then, each research question is answered.

Discussion

Teacher regulation of CSCL is a demanding task. Multiple collaborating groups perform multiple activities, which makes continuous diagnosis a task that could create high cognitive load. The present study examined whether teacher supporting tools could assist teachers by visualizing analyses of students' cognitive activities. The main research question was what the effects are of teacher supporting tools on teachers' diagnosis, interventions, and experienced cognitive load during CSCL.

References (62)

  • R. Mazza et al.

    CourseVis: a graphical student monitoring tool for supporting instructors in web-based distance courses

    International Journal of Human-Computer Studies

    (2007)
  • N. Michinov et al.

    Improving productivity and creativity in online groups through social comparison process: new evidence for asynchronous electronic brainstorming

    Computers in Human Behavior

    (2005)
  • J. Onrubia et al.

    Strategies for collaborative writing and phases of knowledge construction in CSCL environments

    Computers & Education

    (2009)
  • M.F. Salinas

    From Dewey to Gates: a model to integrate psychoeducational principles in the selection and use of instructional technology

    Computers & Education

    (2008)
  • A. Solimeno et al.

    The influence of students and teachers characteristics on the efficacy of face-to-face and computer supported collaborative learning

    Computers & Education

    (2008)
  • A. Van Leeuwen et al.

    Teacher interventions in a synchronous, co-located CSCL setting: Analyzing focus, means, and temporality

    Computers in Human Behavior

    (2013)
  • A. Van Leeuwen et al.

    Supporting teachers in guiding collaborating students: Effects of learning analytics in CSCL

    Computers & Education

    (2014)
  • A. Van Leeuwen et al.

    Teacher regulation of multiple computer-supported collaborating groups

    Computers in Human Behavior

    (2015)
  • J. Vermunt et al.

    Congruence and friction between learning and teaching

    Learning and Instruction

    (1999)
  • L. Voerman et al.

    Types and frequencies of feedback interventions in classroom interaction in secondary education

    Teaching and Teacher Education

    (2012)
  • A. Weinberger et al.

    A framework to analyze argumentative knowledge construction in computer-supported collaborative learning

    Computers & Education

    (2006)
  • T. Anderson et al.

    Assessing teaching presence in a computer conferencing context

    Journal of Asynchronous Learning Networks

    (2001)
  • C.S.C. Asterhan et al.

    Online moderation of synchronous e-argumentation

    International Journal of Computer-Supported Collaborative Learning

    (2010)
  • C.S.C. Asterhan et al.

    Small-group, computer-mediated argumentation in middle-school classrooms: the effects of gender and different types of online teacher guidance

    British Journal of Educational Psychology

    (2012)
  • M. Baker et al.

    Rainbow: a framework for analyzing computer-mediated pedagogical debates

    International Journal of Computer-Supported Collaborative Learning

    (2007)
  • R.S. Baker et al.

    Off-task behavior in the cognitive tutor classroom: when students “Game the system”

  • J. Buder et al.

    Supporting controversial CSCL discussions with augmented group awareness tools

    International Journal of Computer-Supported Collaborative Learning

    (2008)
  • J. Carroll

    A model for school learning

    Teachers College Record

    (1963)
  • C.K.K. Chan

    Bridging research and practice: implementing and sustaining knowledge building in Hong Kong classrooms

    International Journal of Computer-Supported Collaborative Learning

    (2011)
  • M.A. Chatti et al.

    A reference model for learning analytics

    International Journal of Technology Enhanced Learning

    (2012)
  • I.-A. Chounta et al.

    Towards the real-time evaluation of collaborative activities: Integration of an automatic rater of collaboration quality in the classroom from the teacher's perspective

    (2014)
  • Cited by (75)

    • A systematic review of visual representations for analyzing collaborative discourse

      2021, Educational Research Review
      Citation Excerpt :

      Therefore, students need explicit guidance on how to use language effectively and regulate group interactions (Belland et al., 2013; Cohen & Lotan, 2014; King, 2008; Miller & Hadwin, 2015; Scheuer et al., 2010). There is also the need to support teachers in monitoring multiple groups and adopting dialogic strategies (Kazemitabar et al., 2016; van de Pol et al., 2019; van Leeuwen et al., 2015). Therefore, understanding the nature of collaboration is a challenge for researchers, whereas monitoring the collaborative process and making adjustments as needed is a challenge for students and teachers.

    • Learning analytics dashboards for adaptive support in face-to-face collaborative argumentation

      2021, Computers and Education
      Citation Excerpt :

      Fourth, students should understand that the dashboard system is not for surveillance but for better learning support. In F2F learning environments, students and instructors share the same physical space, and the system can augment instructors' influence by allowing them to monitor and control student activities (Van Leeuwen, Janssen, Erkens, & Brekelmans, 2015). If instructors were to interfere with students' learning too frequently based on the information reported by the system, students would think they were being watched.

    • Learning Analytics: a bibliometric analysis of the literature over the last decade

      2021, International Journal of Educational Research Open
      Citation Excerpt :

      The use of LA can bring important challenges and advantages to educational institutions (Howell, Roberts, Seaman & Gibson, 2018; Ifenthaler, 2017). It can be used to predict students’ grades (Agudo-peregrina, Iglesias-pradas, Conde-gonzález & Hernández-garcía, 2014; Hernández-García, González-González, Jiménez-Zarco & Chaparro-Peláez, 2015; Strang, 2017) to strengthen students retention in institutions (De Freitas et al., 2015), to predict students’ risk of failure in courses (Gašević, Dawson, Rogers & Gasevic, 2016; Scholes, 2016; D. Tempelaar, Rienties, Mittelmeier & Nguyen, 2018; D. T. Tempelaar, Rienties & Giesbers, 2015), to identify students’ learning gaps (Nyland, Davies, Chapman & Allen, 2017; Van Leeuwen, Janssen, Erkens & Brekelmans, 2015), to manage decision making in educational institutions (Niet, Diaz & Montenegro, 2016; Popoola et al., 2018), to support active learning environments (Kwong, Wong & Yue, 2017), to help students and teachers make decisions about the learning process (Ali, Asadi & Ga, 2013; Aljohani et al., 2019; Fidalgo-Blanco, Sein-Echaluce, García-Peñalvo & Conde, 2015; Jovanović, Gašević, Dawson, Pardo & Mirriahi, 2017; Ma, Han, Yang & Cheng, 2015; Ruipérez-Valiente, Muñoz-Merino, Leony & Delgado Kloos, 2015; Schumacher & Ifenthaler, 2018; Sedrakyan, Malmberg, Verbert, Järvelä & Kirschner, 2020), and to improve e-assessment (Azevedo, Oliveira & Beites, 2019), among others (Chen, Chang, Ouyang & Zhou, 2018; Conde, Colomo-Palacios, García-Peñalvo & Larrucea, 2018; Hammouda & Kamel, 2007; Hernández-Lara, Perera-Lluna & Serradell-López, 2019; Serrano-laguna, Martínez-ortiz, Haag, Regan & Johnson, 2017; Shoufan, 2019). Considering the above, it can be concluded that understanding LA is an important asset for educational organizations and an important area for research.

    View all citing articles on Scopus
    View full text