Analyzing productive learning behaviors for students using immediate corrective feedback in a blended learning environment

Highlights

• We analyzed students' interactions with a simple feedback tool for online homework.

• Higher engagement with the feedback tool is positively correlated to performance.

• Getting the correct answer on the first try is negatively correlated to performance.

• Interactions with the feedback explains 25% of variance in cumulative grades.

Abstract

Undergraduate classes in many science and engineering courses are utilizing asynchronous computer platforms to host educational materials such as lecture videos or discussion forums. These platforms also have the ability to provide immediate feedback to students on formative assessment tasks such as homework problems, reading questions, or weekly quizzes. Although there have been a number of studies on computer-based feedback, there is more we need to know about how students interact with immediate feedback, and how those interactions influence their learning. In this study, we characterize introductory physics students' interactions with one computer-based immediate simple corrective feedback tool, the “checkable answer feature” (CAF), powered by the institutional version of the edX platform. We investigate how much students interact with the CAF, the patterns of interaction, and, ultimately, how these patterns are associated with course performance. We utilize rich quantitative data, including a large volume of server tracking logs that show students’ use the CAF, as well as performance metrics. Our findings show certain patterns of engagement with feedback reflect productive study strategies and significantly predict higher performance. The findings provide guidance for instructional practice and the continued development of online feedback tools in introductory STEM courses.

Introduction

In educational settings, feedback serves the crucial role of closing the gap between students’ current understanding and desired learning (Shute, 2008). Timely and informative feedback can help learners recognize and correct misconceptions, motivate them to acquire knowledge, and increase their confidence and motivation to learn (Epstein et al., 2010). Yet in the context of higher education, it is often not possible for instructors to provide timely feedback to every student individually. This is especially true in first-year foundational courses due to the large number of students usually enrolled. But feedback delivered on computer-based platforms such as edX (www.edx.org), Coursera (www.coursera.org), and FutureLearn (www.futurelearn.com) offer a potential solution. They can provide students with automatic immediate feedback on formative assessment tasks such as reading questions, homework problems, and short quizzes. These platforms enable the detailed recording of the activities the students perform online. In turn, these data logs can be re-assembled to give researchers an in-depth look into how students are learning the concepts and skills instructors want them to master.

The specific feedback mechanism we focus on in this study is the “checkable answer feature” (CAF) included in a suite of online materials in an introductory blended learning course in physics, which we call PHYS101, at an elite private university in the northeast United States. The CAF, a rudimentary intelligent tutoring system, gives students immediate feedback on whether or not their answers to online homework problems are correct. Once the student enters their answer, if it is correct, the system returns a green tick mark, and if it is wrong, they see a red X (see Fig. 1). In addition, students use the CAF to see if their answers to multi-part problems, which they have to write out, are correct before they submit their traditional, paper-based homework assignments.

It is evident that formative feedback can foster improved achievement and enhanced motivation to learn, while also supporting deep and self-regulated learning (Crisp and Ward, 2008, Koh, 2008, Wolsey, 2008). Studies of courses that are wholly online (Crisp and Ward, 2008, Gijbels et al., 2005, Sorensen and Takle, 2005, Van der Pol et al., 2008, Vonderwell et al., 2007, Wang et al., 2008) show that the effective use of online formative assessment can engage students and teachers in meaningful educational experiences, providing them with opportunities to collaboratively identify learning needs and devise strategies to meet them. Other studies have investigated the effectiveness of online or computer-assisted feedback tools in blended learning environments (e.g., Angus and Watson, 2009, Chung et al., 2006, Feldman and Capobianco, 2008, Lin, 2008, Wang et al., 2008). Foundational work by Van der Kleij, Feskens, and Eggen (2015) analyzing a significant number of studies indicates that elaborated feedback is more effective than simple corrective feedback. This effect is more pronounced in higher level learning and in certain disciplines, as, for example, math versus science. Interestingly, several studies, including recent work by Van der Kleij, do not observe a difference between corrective feedback and elaborated feedback (Mazingo, 2006, Van der Kleij et al., 2012).

This raises the possibility that how students engage with feedback may have a significant influence on how feedback affects learning outcomes. While, there is evidence that the amount of time spent with feedback is mediated by student attitudes and motivations (Narciss and Huth, 2006, Van der Kleij et al., 2012), to our knowledge there are no quantitative studies of how the level and patterns of student engagement with corrective feedback can affect learning outcomes. Our study addresses this gap by addressing the following research questions:

RQ1: How much do students interact with simple corrective feedback in a blended undergraduate physics class, and what are the patterns of engagement?

RQ2: (a) What patterns of engagement with simple corrective feedback comprise productive study behaviors and are associated with stronger performance, and (b) what patterns are negatively correlated with performance?

Computer-based feedback systems are becoming more sophisticated and of growing benefit in higher education. Individual feedback can be more easily scaled to large numbers of learners because each student can interact with information that is instantaneously provided, and the material can be focused on the individual student's cognitive needs, study strategies, and preferences for how material is presented. By combining advances in artificial intelligence with content experts' knowledge of difficult concepts and common misconceptions, the material can also address the stumbling blocks that interfere with learning in particular disciplines. Feedback can be delivered when and where it is most convenient for the student (a dorm room, on-campus common study space, or Starbucks, for that matter). The ever-increasing use of these online educational platforms, coupled with rich, large-scale data gathering tools, enables an unprecedented window into student use of computer-based feedback mechanisms. Recognizing these interactions can help instructors deliver more targeted interventions, as well as help platform developers create the features that can help maximize the potential of these online systems. This study also contributes to understanding how a large, detailed online dataset can reveal patters of student study behaviors. In turn, this demonstrates ways in which data mining can be used in to complement traditional statistics and generate new knowledge. Finally, correlating online feedback behaviors with course performance sheds light on how self-regulated learning and productive struggle contribute to positive student outcomes.