Elsevier

Computers in Human Behavior

Volume 36, July 2014, Pages 401-411
Computers in Human Behavior

Using erroneous examples to improve mathematics learning with a web-based tutoring system

https://doi.org/10.1016/j.chb.2014.03.053Get rights and content

Highlights

  • Middle school students learned to solve decimal problems with a web-based tutoring system.

  • ExErr group received erroneous examples to correct and explain.

  • PS group received problems to solve and explain.

  • ExErr group outperformed PS group on a delayed test and on judging answer correctness.

  • PS group reported liking the lessons better than the ExErr group.

Abstract

This study examines whether asking students to critique incorrect solutions to decimal problems based on common misconceptions can help them learn about decimals better than asking them to solve the same problems and receive feedback. In a web-based tutoring system, 208 middle school students either had to identify, explain, and correct errors made by a fictional student (erroneous examples group) or solve isomorphic versions of the problems with feedback (problem-solving group). Although the two groups did not differ significantly on an immediate posttest, students in the erroneous examples group performed significantly better on a delayed posttest administered one week later (d = .62). Students in the erroneous examples group also were more accurate at judging whether their posttest answers were correct (d = .49). Students in the problem-solving group reported higher satisfaction with the materials than those in the erroneous examples group, indicating that liking instructional materials does not equate to learning from them. Overall, practice in identifying, explaining, and correcting errors may help students process decimal problems at a deeper level, and thereby help them overcome misconceptions and build a lasting understanding of decimals.

Introduction

Students learn and understand mathematics at a deeper level when they are prompted to make judgments about what is wrong in other students’ erroneous solutions. This proposal, which we call the erroneous examples hypothesis, is the focus of the present study. In particular, the objective of this experiment is to examine the effectiveness of mathematics instruction based on erroneous examples compared to traditional problem solving when both are incorporated into an online tutoring system.

To examine this hypothesis, we focus on the content area of decimals, because this is a critical sub-domain of mathematics, essential for almost all of more advanced mathematics. Students often have difficulty mastering decimals and misconceptions are common and persistent (Irwin, 2001, Resnick et al., 1989, Sackur-Grisvard and Léonard, 1985), with difficulties enduring even into adulthood (Putt, 1995, Stacey et al., 2001). We focus on online tutoring in order to contribute to evidence-based design principles for improving computer-based learning of mathematics. Participants in the problem-solving control group solved problems such as filling in the next two numbers in a sequence (i.e. 2.97, 2.98, 2.99, ___, ___). In contrast, participants in the erroneous examples group were presented with isomorphic questions in which a fictional student had solved the same problem incorrectly due to a misconception such as treating the two sides of the decimal as separate (i.e., 2.97, 2.98, 2.99, 2.100, 2.101) and were asked to identify and correct errors. This study investigated whether students who are exposed to erroneous examples (the erroneous examples group) would have a better learning outcome than students who get practice in solving the same problems (the problem-solving group).

According to cognitive theory of multimedia learning and cognitive load theory from which it is derived, the learner can engage in three kinds of cognitive processing during learning (Mayer, 2009, Mayer, 2011, Moreno and Park, 2010, Sweller, 1999, Sweller et al., 2011): extraneous processing (or extraneous load) is cognitive processing that does not support the instructional goal and is often caused by poor instructional design; essential processing (or intrinsic load) is cognitive processing required to mentally represent the material as presented and is caused by the complexity of the material; and generative processing (or germane load) is cognitive processing aimed at making sense of the presented material such as reorganizing it and integrating it with relevant prior knowledge and is caused by the learner’s motivation to exert effort to understand the material. The challenge of instructional design is to promote generative processing, without burdening learners with too much essential and extraneous processing that overloads their limited working memory capacity.

Worked-out examples (also called worked examples), which consist of a problem statement, the steps taken to reach a solution, and the final solution, have been used effectively to help manage essential processing and decrease extraneous processing (Cooper and Sweller, 1987, McLaren et al., 2008, McLaren and Isotani, 2011, Renkl, 2005, Renkl, 2011, Renkl and Atkinson, 2010, Zhu and Simon, 1987). Worked examples achieve this by focusing the student’s attention on the correct solution procedure to follow (Sweller, Ayres, & Kalyuga, 2011), which helps the student avoid searching their prior knowledge for solution methods and lessens extraneous processing. The freed up cognitive resources can then be used for generative processing, in particular, understanding and eventually automatizing the steps in a problem’s solution procedure. In a classic experiment on worked examples, Cooper and Sweller (1987) found that learning to solve algebra equations by studying worked examples, paired with problems to solve, resulted in faster transfer test performance than simply solving all the same problems. A variety of subsequent studies and reviews have documented the superiority of instruction using worked examples (McLaren and Isotani, 2011, Renkl, 2005, Renkl, 2011, Sweller et al., 2011).

However, one possible issue with worked examples is that although students may free up cognitive resources, this does not mean that the freed cognitive capacity will be used for generative processing (Renkl & Atkinson, 2010). One way to encourage generative processing is through self-explanation, in which learners explain the instructional materials to themselves. Chi, Bassok, Lewis, Reimann, and Glaser (1989) found that good problem solvers are more likely to generate self-explanation statements while thinking out loud when studying worked examples of physics problems. In addition, other research has shown the importance of explicitly prompting for self-explanation (Hausmann & Chi, 2002). While worked examples can be used to reduce processing demands, self-explanation prompts can be used to encourage deeper processing leading to better performance on transfer items (Atkinson et al., 2003, Hausmann and Chi, 2002).

A less studied way to encourage deeper processing while using worked examples is to present students with incorrect (i.e., erroneous) examples. Erroneous examples have been studied by a few learning science researchers (e.g., Durkin and Rittle-Johnson, 2012, Große and Renkl, 2007, Siegler, 2002, Tsovaltzi et al., 2010) and involve most of the same steps as a worked example except one or more of the steps is incorrect. In these past studies, students typically must locate the error(s), explain the error(s), and then make appropriate corrections. Erroneous examples may encourage students to engage in generative processing, as they explain to themselves why a particular part of the problem is incorrect (Durkin & Rittle-Johnson, 2012). Erroneous examples may also help students focus on each step of a solution method separately to identify where the error occurred.

On the other hand, erroneous examples may also place additional processing demands on learners, overloading working memory with extraneous processing. For instance, a student working with an erroneous example may search for what is wrong in the example by trying to represent internally both the correct and incorrect solution steps and compare those steps (Große & Renkl, 2007). One possible way to relieve this processing demand is to highlight the error in the problem for the student (Große & Renkl, 2007). Tsovaltzi et al. (2010) found that providing erroneous examples of fraction problems, along with providing computer-generated help in finding and correcting the error, was more effective than presenting erroneous examples with no help. Given what is known from this past research, the erroneous examples materials used in the present experiment indicate where an error has occurred in the decimal problems.

Some of this past research suggests that erroneous examples can facilitate learning of mathematics. For example, Durkin and Rittle-Johnson (2012) found that students who compared worked and erroneous examples of decimal problems learned more than students who compared pairs of correct worked examples. Students who compared worked and erroneous examples were also twice as likely to discuss correct concepts as students who compared worked examples only. In addition, Siegler (2002) found that having students self explain both correct and incorrect examples of mathematical equality was more beneficial for learning a generalizable procedure than self-explaining correct examples only. In Kawasaki (2010), 5th grade students benefited when the teacher contrasted both a correct and an incorrect solution to a math problem, especially when the student also committed the same kind of error.

One issue with using erroneous examples for learning is that the prior knowledge level of the learner can interact with the effectiveness of the erroneous examples. Große and Renkl (2007) found that high prior knowledge (but not low prior knowledge) college students benefitted from lessons containing both correct and incorrect solutions to probability problems rather than seeing only correct solutions. In addition, the high prior knowledge students did not benefit from having errors highlighted, presumably because they were able to identify the errors on their own. Low prior knowledge individuals did significantly better when the errors were highlighted than when they were not. This research suggests that erroneous examples may not be effective for students who do not already have a basic grasp of the instructional material while high prior knowledge students may benefit by processing the information at a deeper level.

Similarly, if working memory is overloaded by the essential processing demands of the task, erroneous examples may not encourage deeper processing. This may have occurred, for instance, in an earlier study by our group (Isotani et al., 2011) with similar materials to those used in the present study. Middle school students learned decimals by either solving practice problems, or by having to piece together self-explanation sentences for either correct worked examples or a combination of correct and erroneous examples. There were no significant differences among the three groups on either an immediate posttest or delayed posttest. The possible benefits of erroneous examples may have been offset by a self-explanation interface that inadvertently induced cognitive load. The students were prompted to create self-explanations of why solutions are incorrect by being presented with the start of a sentence and then being asked to complete the sentence from two pull-down menus. Instead of focusing on the mathematic content, students may have devoted too much of their cognitive processing to selecting the correct sentence segments and reviewing the completed sentences. For the present study a simpler self-explanation interface was used by the erroneous examples group, aimed at minimizing extraneous processing while fostering generative processing.

Finally, it is important to note that much of the prior research on erroneous examples has focused on multi-step mathematics problems. Worked examples, which are excellent at showing multi-step problem solutions, were sometimes used as control conditions to examine the possible benefits of using erroneous examples. In contrast, the present study focuses on simpler decimal problems that (for the most part) require single-step solutions (e.g. comparing two decimals to decide which one is larger). Therefore, worked examples are not a good control condition for the present study; standard problem solving constitutes a better control to contrast with the effects of erroneous examples. Furthermore, previous studies, including the classic work of Cooper and Sweller (1987), have used problem solving as the control condition in a variety of learning situations and an analysis of mathematics textbooks indicates that presenting students with problems to solve is still the de facto standard in mathematics instruction (Hiebert et al., 2005, Mayer et al., 1995). Therefore, in order to produce a clear comparison between erroneous examples and the instructional approach most commonly used for decimal instruction, we focused on problem solving as the control group in the present study.

Persistent misconceptions in students’ decimal knowledge must be overcome so students can move on to more advanced mathematics. Yet, teaching students decimals is complicated by the fact that teachers are not always aware of common misconceptions and may misattribute incorrect answers to the wrong underlying misunderstandings (Stacey et al., 2001). Pre-service teachers in their study were aware of the misconception that longer decimals are larger, i.e., what Isotani et al. (2011) have called Megz; however, few were aware of the shorter-is-larger misconception (i.e., called Segz) and often make those errors themselves.

Based on an extensive literature review, Isotani et al. (2011) created a taxonomy of misconceptions that represents 17 distinct misconceptions. The present study focuses on four of these misconceptions, the ones that prior research has shown are most common and contributory to other misconceptions: Megz (“longer decimals are larger”, e.g., 0.23 > 0.7), Segz (“shorter decimals are larger”, e.g., 0.3 > 0.57), Negz (“decimals less than 1.0 are negative” e.g., .07 goes to the left of 0 on a number line), and Pegz (“the numbers on either side of a decimal are separate and independent numbers”, e.g., 11.9 + 2.3 = 13.12). Our approach in this study is to address these common misconceptions directly; all of the problems presented to students in our study target one or more of these four misconceptions.

For the current study we streamlined the materials from Isotani et al. (2011) to help learners more easily focus on explaining and fixing errors in erroneous examples. The materials were altered so that students only had to complete sentences with a single choice for their self-explanation statements, as opposed to having to complete a sentence with two pull-down choices. Taking a cue from work by Johnson and Mayer (2010), in which students performed better on an embedded transfer test when they selected an explanation rather than having to generate one on their own, we propose that providing the explanation statements for misconceptions, rather than having learners generate their own, will relieve processing demands.

In addition, in order to focus the present study on a comparison of erroneous examples to the most common control, as well as to maximize statistical power, we simplified the design to two groups: erroneous examples and problem solving. Thus, this study examines whether erroneous examples can encourage deeper understanding than problem solving.

Our assumption is that three basic conditions are necessary to lead to learning from erroneous examples. First, to avoid embarrassment and demotivation, the errors should be examples of another student’s errors, not their own. Second, the erroneous examples should be interactive and engaging. Using computer-based materials, students are prompted for explanations, asked to find and correct errors, and given feedback. Finally, enough guidance and structure should be provided to minimize extraneous processing and manage essential processing during learning with erroneous examples.

Taking those three points into consideration, the two groups in this study were presented with isomorphic problems, but with different ways of interacting with those problems. Students in the erroneous examples group were presented with an incorrect solution, prompted to explain and correct the error and reflect on the correct answer, and received feedback on their responses. Students in the problem-solving group were asked to solve the same problems, reflect on the correct answers and explain their solution, and received feedback on their work. The additional steps in the erroneous examples condition of explaining and correcting the error were intended to improve learning outcomes by encouraging learners to engage in generative processing of decimal principles. A comparison of the steps presented by the two conditions can be seen in Fig. 1.

From the perspective of generative learning theory (Mayer, 2009, Mayer, 2011), the theoretical rationale for using erroneous examples is that they offer enough challenge to foster generative processing in learners—that is, cognitive processing aimed at making sense of the material by mentally reorganizing it and connecting it with relevant prior knowledge—while providing enough structure and guidance to not overload the learner with extraneous processing. In contrast, problem solving can cause learners to engage in so much extraneous processing that they fail to abstract the underlying solution principle used for solving the problem (Sweller et al., 2011). Based on research and theory on generative learning techniques in the science of learning, deep learning may be most sensitive to delayed rather than immediate tests (Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013).

Hypothesis 1

Thus, we expect the erroneous examples group to engage in deeper cognitive processing during learning than the problem-solving group, yielding superior posttest performance, particularly on delayed tests as has been found with other generative techniques such as the testing effect (Dunlosky et al., 2013, Johnson and Mayer, 2009).

Hypothesis 2

By virtue of their training in evaluating student solutions, we also expect the erroneous examples group to outperform the problem-solving group on correctly assessing their level of confidence for their solutions of posttest problems.

Section snippets

Participants and design

The participants were 208 middle-school students (101 boys and 107 girls) from the same Pittsburgh area school. One hundred and five students were in the 6th grade and 103 were in the 7th grade. The students’ ages ranged from 11 to 13 years old (M = 11.99, SD = .72). In a between subjects design, 100 students served in the erroneous examples group and 108 students served in the problem-solving group.

Are the groups equivalent on basic demographic characteristics and prior knowledge measures?

Based on t-tests and chi-square results, the two groups did not differ significantly (with p < .05) on average age, proportion of boys and girls, or proportion of 6th and 7th graders.

Concerning pretest score, due to an error in data logging for four of the test problems, the data for those problems was removed from the pretest, immediate posttest, and delayed posttest scores making the total possible score out of 46. The first column of Table 2 shows the mean (and standard deviation) of the

Empirical contributions

Consistent with Hypothesis 1, the primary finding is that the ErrEx group significantly outperformed the PS group on the delayed posttest, both when pretest score was used as a covariate (d = .62) and when outliers were eliminated (d = .49). However, the superiority of the ErrEx group over the PS group (d = .38) did not reach statistical significance on the immediate posttest. This pattern of results is a major new contribution to the research literature on erroneous examples and is consistent with

Acknowledgments

This project was supported by Award No. R305A090460 (PI: B. McLaren) from the Institute of Education Sciences (IES) of the U.S. Department of Education. We also thank the Pittsburgh Science of Learning Center, NSF Grant 0354420, for technical support of our work.

References (40)

  • A. Eryilmaz

    Effects of conceptual assignments and conceptual change discussions on students’ misconceptions and achievement regarding force and motion

    Journal of Research in Science Teaching

    (2002)
  • Goguadze, G., Sosnovsky, S., Isotani, S., & McLaren, B. M. (2011). Evaluating a Bayesian student model of decimal...
  • R.G.M. Hausmann et al.

    Can a computer interface support self-explanation?

    International Journal of Cognitive Technology

    (2002)
  • W.L. Hayes

    Statistics

    (1994)
  • J. Hiebert et al.

    Mathematics teaching in the United States today (and tomorrow): Results from the TIMSS 1999 video study

    Education Evaluation and Policy Analysis

    (2005)
  • K.C. Irwin

    Using everyday knowledge of decimals to enhance understanding

    Journal for Research in Mathematics Education

    (2001)
  • Isotani, S., Adams, D., Mayer, R. E., Durkin, K., Rittle-Johnson, B., & McLaren, B. M. (2011). Can erroneous examples...
  • C.I. Johnson et al.

    A testing effect with multimedia learning

    Journal of Educational Psychology

    (2009)
  • M. Kawasaki

    Learning to solve mathematics problems: The impact of incorrect solutions in fifth grade peers’ presentations

    Japanese Journal of Developmental Psychology

    (2010)
  • R.E. Mayer

    Multimedia learning

    (2009)
  • Cited by (97)

    View all citing articles on Scopus
    View full text