To evaluate the effectiveness of the above software interventions in promoting positive changes in students’ social and programming behaviors, we conducted a series of mixed-methods empirical studies.
4.1 Method
4.1.1 Design.
Our original empirical evaluation was designed as a within-subjects quasi-experimental study [
61,
75] in a CS1 course at our home institution. As they completed programming assignments, students in the study were exposed during the first half of the course to a Control treatment in which they did not have access to the interventions. During the second half of the course, students were exposed to an Experimental treatment in which they did have access to the interventions. Table
3 presents a week-by-week breakdown of course topics included in each treatment.
As it turned out, students in the study varied considerably in the extent to which they interacted with the interventions. Some students’ lack of engagement with the intervention created gaps in the experimental treatment data, leading to inconclusive results regarding the impact of the interventions.
Given the inconclusive results of our original study, we decided to modify the design: Rather than trying to assert, through a controlled experiment, a causal link between students’ interaction with the interventions and their social behaviors, social connections, attitudes, and academic performance, we performed a follow-up study to identify possible correlations between students’ level of interaction with the interventions in the final four weeks of the course and the outcome variables. Below, we present the follow-up study.
4.1.2 Participants.
Participants included 41 students (33 male, 8 female, age range 18 to 30 y.o., M = 22 y.o.) enrolled in two offerings of the CS1 course at our home institution, a large research university in the western U.S. These included all 26 students enrolled in the summer 2017 offering, along with 15 of the 19 students enrolled in the summer 2018 offering. Participants were a mix of computer science majors (46%) and non-computer science majors (54%) from engineering, mathematics, and business fields. The first author served as the course instructor for both course offerings.
4.1.3 Courses and Materials.
The two courses involved in the study were nearly identical. Focusing on computer programming in the C language, each course condensed the same material normally covered in a sixteen-week semester into an eight-week period. Moreover, each course had the same instructor, textbooks, labs, assignments, and exams.
Students in the courses had access to a nearly identical set of course materials, all of which were available online through the course LMS. These materials consisted of handouts, prompts for 12 programming labs, prompts and grading rubrics for eight individual programming assignments, weekly code samples, weekly lecture slides, weekly quizzes, and a midterm and final exam.
Each course adopted an identical plagiarism and academic integrity policy. High-level collaboration was encouraged, but code copying and pair programming were not allowed. To ensure that students’ code submissions were original, the instructor submitted students’ submissions to the MOSS plagiarism detection software [
1]. All exams were administered in-class under the instructor's supervision.
Two additional course materials were central to this study. First, during the final four weeks of each course, students used a
Social IDE and
LMS that included the interventions described in Section
3. Second,
online surveys were used to gauge student attitudes before and after students’ exposure to the interventions. The surveys consisted of questions from the
MSLQ: Motivated Strategies for Learning Questionnaire [
57], C++ Self-Efficacy [
63], Classroom Community scales [
69], and the Sociability Scale [
37].
4.1.4 Data Collected.
We gathered four types of evaluation data on participants in this study:
8.
Demographic data, including participants’ gender, age and academic major.
9.
Attitudinal data through online surveys administered before and after exposure to the interventions.
10.
Log data on participants’ programming processes and social activities within the Social IDE and the LMS (see Table
4).
11.
Learning outcomes data: students’ grades for programming assignments, course participation, labs, exams and the overall course.
4.1.5 Procedure.
At the beginning of the two courses in which this study took place, the course instructor briefly advertised the study and invited students to participate. The advertisement emphasized that participation in the study would not affect students’ course activities; the only difference was that students who consented to participate would release their data for research purposes. Those who chose to participate completed a written informed consent form, which also included basic demographic questions. The study protocol and informed consent form were approved by our Institutional Review Board.
Prior to their exposure to the interventions, and again at the end of the course, students were required to complete the same online attitudinal survey to gauge changes in their attitudes.
During the final four weeks of the course, students were required to complete individual programming assignments using the Social IDE and LMS. To incentivize online social participation during the programming assignments, the course instructor established a requirement of two posts and replies per week.
Students’ posts and replies were not graded on quality. However, any posts or replies that did not contain academic or social content (e.g., “Post 1: Obligatory post”) were removed from the analysis.
At the end of the course, the survey data, log data and grades data of those students who consented to participate in the study were collected for analysis.
4.2 Results
To explore the extent to which students interacted with the interventions (RQ 1), Figure
14 presents a histogram of the frequency of each number of total interactions. On average, students interacted with the interventions 12.83 times (
sd = 9.27) during the four-week period in which the interventions were available. As indicated by the figure and reflected in the high standard deviation, students’ level of interaction with the interventions varied considerably, with five students having fewer than five interactions and another five students having more than 25 interactions.
To shed further light on students’ interaction with the interventions, Table
5 presents students’ level of interaction with each of the eight interventions described in Section
3. For each interaction type, the table shows the average number of interventions generated; (b) of those generated, the average percentage interacted with at least once; and (c) the average number of times the intervention was interacted with, given that any intervention could be interacted with more than once.
As Table
5 indicates, students did not interact at all with the interventions that were generated in response to programming errors. In contrast, interventions that prompted students to reach out to others who said they were available, or that asked students to view or change their availability status, elicited the highest levels of interaction. In the middle were interventions that encouraged self-reflective and topical posts elicited interaction; these were interacted with between three and seven percent of the time.
In the remainder of this section, we use the nonparametric Spearman's rho (
rs) to test for correlations relevant to RQ 2 and RQ 3. Based on conventions in social science and educational research, we set the threshold value for statistical significance to
p < 0.05. Since we are testing for correlations between a single variable and up to seven others, we need to consider whether to guard against type I error (e.g., using Bonferroni correction). Given that we (a) have a small sample size, (b) are performing a small number of planned comparisons, (c) are performing nonparametric tests in sequence, and (d) are most interested in the results of the individual tests (as opposed to a single omnibus test), we take Armstrong's [
6] and Perneger's [
55] advice
not to adjust for type I error. In addition, we interpret
rs values of below 0.3 as
weak correlations,
rs values between 0.3 and 0.6 as
moderate correlations, and
rs values above 0.6 as
strong correlations [
2].
To examine the association between students’ pre-intervention attitudes and their level of interaction with the interventions (RQ 2), Table
6 presents correlations between students’ pre-attitudes and interaction level. As Table
6 indicates, no significant correlations were found between students’ attitudes at the start of the course and their interaction level. It is notable, however, that the correlation with self-sociability was over seven times stronger than with any other pre-intervention attitudinal measure. Thus, while students’ preexisting attitudes toward self-efficacy, community connectedness, self- and peer-learning appear not to be predictive of the extent to which they interacted with the interventions, there is a hint in the data that students’ preexisting attitudes toward self-sociability may be related to their level of interaction with the interventions.
To explore whether students’ level of interaction with the interventions might be associated with increased social and programming behaviors, improved attitudes toward learning, and increased learning outcomes (RQ 3), we conducted a further series of correlational analyses. Table
7 presents correlations between students’ interactions with the interventions and their (a)
posting activities—counts of feed posts, replies and helpful marks given to replies, (b)
browsing activities—counts of events directed toward browsing activity feed content (searches, viewing of post replies and post details); and (c)
programming activities—build, clipboard, debug, code editing, build error, save, and assignment submission events that took place within the IDE. As the table shows, there are statistically significant correlations between intervention interactions and both types of social activities, but not between intervention interactions and programming activities. The significant correlations are moderate in size. This finding provides evidence that interaction with the interventions was positively associated with the social behaviors they were designed to promote.
Table
8 examines correlations between students’ interactions with the interventions and their attitudes at the end of the course. These results identify a statistically significant correlation between students’ level of interaction with the interventions and their end-of-course attitudes toward peer learning. The strength of this correlation falls in the weak range. This suggests that students who interacted with the interventions more extensively showed an increased willingness to enlist their peers in the learning process.
Table
9 considers correlations between students’ interactions with the interventions and their academic course performance, as gauged by key course assessments. As can be seen, a significant correlation exists between students’ intervention interactions and their programming assignment grades. The size of the correlation is moderate. This finding suggests that interaction with the interventions is associated with positive performance in the course programming assignments.
Finally, to investigate how interaction with the interventions might relate to students’ social connectedness with their peers, we analyzed students’ post-reply relationships as social network graphs using Gephi, a
social network analysis (SNA) tool [
23]. Table
10 presents correlations between students’ intervention interactions and five key SNA metrics: (weighted) degree, closeness, betweenness, and eigenvector centrality. Taken together, these metrics provide a sense of
social centrality by indicating the number of connections around a specific social participant [
14]. As shown in Table
10, statistically significant correlations exist between students’ intervention interactions and both the weighted degree and eigenvector centrality metrics. Both correlations are moderate in strength. This finding indicates that interaction with the interventions was positively associated with the formation of stronger, more closely coupled social networks.
4.3 Discussion
Based on social learning theory and driven by log data automatically collected through the IDE, we designed the interventions presented in Section
3 to foster increased social interaction by encouraging students to ask questions, answer their peers’ questions, and engage in self-reflection within the context of individual programming assignments. In turn, we predicted that such increased social interaction would lead to improved learning outcomes. While large inconsistencies in students’ engagement with the interventions rendered our original quasi-experimental results inconclusive, the follow-up correlational study presented here gained useful insights into the three research questions posed in Section
1. Below, we revisit these research questions in light of our results.
4.3.1 RQ 1: To What Extent Will Students Interact with Interventions?
On average, students interacted with the interventions about 12 times during the four weeks they were available. This comes out to about three times per weekly programming assignment. However, as Figure
14 reveals, students exhibited a large variance in their interaction with the interventions; indeed, the standard deviation was 9.27.
Further exploration of students’ interaction with specific intervention types revealed three broad levels of interaction. First, students did not interact at all with the interventions that were generated in response to programming errors. This could have at least three possible explanations: (1) there were usability problems with those interventions (e.g., they lacked relevance or visibility); (2) students did not feel comfortable asking their peers for help on specific error messages; or (3) students regarded other help-seeking approaches (e.g., searching for help online) as easier or more effective.
Second, interventions that encouraged self-reflective and topical posts elicited modest interaction, with interaction rates of three to seven percent. One possible explanation for this is that the prompts came at an opportune time for students—when they were done with an assignment or were in a productive period of programming. However, making reflection posts required a level of effort that many students were unwilling to make. If students had perceived the effort of performing a reflection post as worthwhile—e.g., by making such reflection a course requirement, or by making the case that engaging in and sharing one's self-reflections are valuable activities in and of themselves—then perhaps students would have been more inclined to respond to these interventions.
Finally, interventions that prompted students to reach out to others who said they were available, or that asked students to make themselves available for help, elicited the highest levels of interaction (48-68% response rates). We can infer from this data that students are more comfortable reaching out to students who said they are available, and, conversely, that students who are informed of a need for help are willing to make themselves available. We are encouraged by this finding, which suggests that struggling students in early computing courses can be motivated to reach out for help if they know others are willing to help, and that students can be motivated to help others if they know there is a need. Such help-seeking and help-giving behaviors are pillars of vibrant social learning communities.
4.3.2 RQ 2: Are Students’ Preexisting Attitudes Correlated with Their Intervention Interaction?
Prior to the intervention, our study elicited students’ attitudes related to self-efficacy [
63], sense of community [
69], self- and peer-learning, and self-sociability. These attitudinal variables were carefully selected based on their perceived relevance to students’ choice to engage with the interventions. While our study was unable to identify any significant associations between these variables and students’ level of interaction with the interventions, the fact that the correlation with self-sociability was found to be over seven times greater than the correlation with the other variables was notable, suggesting that further study of the relationship between self-sociability and intervention interaction may be warranted.
In addition, given the wide variance observed in students’ interaction with the interventions, it would be helpful to identify additional variables that might predict students’ level of interaction. Indeed, knowing what factors might influence students’ choice to respond to the interventions could help better tailor the interventions for wider student use. This is clearly an area for future research.
4.3.3 RQ 3: Is Students’ Intervention Interaction Associated with Increased Social Behaviors, Improved Attitudes Toward Learning, and Higher Learning Outcomes?
In response to the core research question investigated by our study, we performed a series of correlational tests to identify associations between students’ usage of the interventions and their social behaviors, programming behaviors, post-intervention attitudes, academic achievement, and connectedness to their peers. These tests revealed that students’ level of interaction with the interventions was significantly correlated with five items:
1.
Their posting events in the activity feed (strength: moderate)
2.
Their browsing events in the activity feed (strength: moderate)
3.
Their post-intervention attitude toward peer learning (strength: weak)
4.
Their performance on course programming assignments (strength: moderate)
5.
Their connectedness with peers based on the Weighted Degree and Eigenvector Centrality metrics from Social Network Analysis (strength: moderate)
Recall that, based on the social learning theory that guided this research, we designed the interventions to promote increased social activity, hypothesizing that students’ programming performance would improve if they became more socially active by asking questions, answering questions, and self-reflecting on the programming process. The five significant positive correlations of weak to moderate strength identified in this research appear to provide limited support for the design goals and learning theory that drove this study: Increased interaction with the interventions was positively associated with increased social activity, more positive attitudes toward learning with the help of peers, higher performance on course programming assignments, and increased social connectedness. According to social learning theory, learning thrives when students are more active socially, are more positive about learning with the help of their peers and develop stronger connections to their peers.
However, it is important to keep in mind that the evidence produced in this study is correlational, not causal. We cannot conclude that the interventions themselves caused the increases in social behavior, more positive attitudes, and improved programming performance we observed. Rather, we can conclude that students who interacted with the interventions more frequently tended to be more active socially, tended to have more positive attitudes regarding peer learning, tended to be more socially connected, and tended to perform better on programming assignments. It could be the case that students who were a priori more socially active, more positive about peer learning, more socially connected, and higher academic achievers would tend to interact more frequently with the interventions. Clearly, the nature of the relationship between intervention interaction, social activity, attitudes, social connectedness, and programming performance remains an important open question for future research.