Jean M. Griffin
ABSTRACT
There is general agreement that people can learn from their own mistakes, but the idea of intentionally incorporating errors into curricula is controversial. Some think it will introduce misconceptions, while others think it could help students learn concepts, avoid common errors and misconceptions, and reduce anxiety about errors. This paper develops a theoretical framework that supports this technique, and reviews the experiences and experiments of over a dozen research teams that have employed it for mathematics and computer science education. It also reports on a 14-week study that investigates this issue, where undergraduates learning to program with Python solved web-based practice problems during weekly labs. Approximately 80% of the problems were the same for all students. For the remaining 20%, the treatment group got problems with carefully designed bugs in supplied code. The control group got similar problems without bugs. According to two pre/post measures, the treatment group learned as much as the control group, with no statistical difference in learning gains. Instructional designs, design principles, and future research plans are discussed.
- J. L. Booth, Karin E. Lange, Kenneth R. Koedinger, and Kristie J. Newton. 2013. Using example problems to improve student learning in algebra: Differentiating between correct and incorrect examples. Learning and Instruction 25 (2013), 24--34.Google Scholar
Cross Ref
- R. Borasi. 1994. Capitalising on errors as "springboards for inquiry": A teaching experiment. Journal for Research in Mathematics Education 25, 2 (1994), 166--208.Google ScholarCross Ref
- Karen Brennan, Christan Balch, and Michelle Chung. 2011. Creative Computing. (2011). http://scratched.gse.harvard.edu/guide/files/CreativeComputing20141015.pdfGoogle Scholar
- Ann L Brown. 1992. Design Experiments: Theoretical and Methodological Challenges in Creating Complex Interventions in Classroom Settings. Journal of the Learning Sciences 2, 2 (1992), 141--178.Google ScholarCross Ref
- Renee Bryce. 2011. Bug Wars: A Competitive Exercise to Find Bugs in Code. Journal of Computing Sciences in Colleges 27, 2 (2011), 43--50. Google ScholarDigital Library
- Xingliang Chen, Antonija Mitrovic, and Moffat Mathews. 2016. Do Erroneous Examples Improve Learning in Addition to Problem Solving and Worked Examples?. In Proceedings of International Conference on Intelligent Tutoring Systems (ITS '16). Springer, Cham, 13--22. Google ScholarDigital Library
- Xingliang Chen, Tanja Mitrovic, and Moffat Mathews. 2016. Do Novices and Advanced Students Benefit from Erroneous Examples Differently? Proceedings of 24th International Conference on Computers in Education (2016).Google Scholar
- Michelene T. H. Chi and Miriam Bassok. 1989. Learning from Examples via Self-Explanations. In Learning and Research Development Center Technical Report No. 11. University of Pittsburgh, 1--37.Google Scholar
- Ruth Clark, Frank Nguyen, and John Sweller. 2006. Efficiency in Learning, Evidence-Based Guidelines to Manage Cognitive Load. Wiley, San Francisco, CA.Google Scholar
- Allan Collins. 1990. Towards a Design Science of Education, Technical Report No. 1. Center for Technology in Education. Center for Technology in Education, Center for Technology in Education, New York NY, 1--9.Google Scholar
- Jere Confrey. 1990. What constructivism implies for teaching. In Journal for Research in Mathematics Education. Monograph Vol 4. NCTM, 107--122.Google Scholar
- Kathryn Cunningham, Cherry Street, Sarah Blanchard, Barbara Ericson, and Mark Guzdial. 2017. Using Tracing and Sketching to Solve Programming Problems. Proceedings of the thirteenth International Workshop on Computing Education Research Workshop (ICER '17) (2017), 164--172. Google ScholarDigital Library
- Quintin Cutts. 2012. Quintin Cutts - Teaching Programming: Too much doing, not enough understanding. https://www.youtube.com/watch?v=Pim4aYfiZiYGoogle Scholar
- Lionel E Deimel and J Fernando Naveda. 1990. Reading Computer Programs: Instructor's Guide and Exercises. Technical Report August.Google Scholar
- Brian Dorn and M. Guzdial. 2006. Graphic designers who program as informal computer science learners. In Proceedings of the second International Workshop on Computing Education Research (ICER '06). ACM, New York, NY, 127--134. Google ScholarDigital Library
- Allen B. Downey. 2016. Think Python, 2nd Edition. O'Reilly Media Inc., Sebastopol CA.Google Scholar
- B. du Boulay. 1986. Some Difficulties of Learning to Program. Journal of Educational Computing Research 2, 1 (1986), 57--73.Google ScholarCross Ref
- B. du Boulay, T. O'Shea, and J. Monk. 1981. The black box inside the glass box: presenting computing concepts to novices. International Journal of Man-Machine Studies 14 (1981), 237--249.Google ScholarCross Ref
- Kelley Durkin and Bethany Rittle-Johnson. 2012. The effectiveness of using incorrect examples to support learning about decimal magnitude. Learning and Instruction 22, 3 (jun 2012), 206--214.Google ScholarCross Ref
- Barbara J. Ericson, Steven Moore, Briana Morrison, and Mark Guzdial. 2015. Usability and Usage of Interactive Features in an Online Ebook for CS Teachers. Proceedings of the tenth Workshop in Primary and Secondary Computing Education (WiPSCE '15) (2015), 111--120. Google ScholarDigital Library
- L. Festinger. 1957. A theory of cognitive dissonance. Stanford University Press, Stanford, CA.Google Scholar
- L. Festinger. 1962. A theory of cognitive dissonance (Vol. 2). Stanford University Press, Stanford CA.Google Scholar
- Sally Fincher and Marian Petre (Eds.). 2004. Computer Science Education Research. Taylor & Francis, London; NY. Google ScholarDigital Library
- Sue Fitzgerald, Gary Lewandowski, Renée McCauley, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: finding, fixing and flailing, a multi-institutional study of novice debuggers. In Computer Science Education, Vol. 18. 93--116.Google ScholarCross Ref
- Michael Frese and Alexandra Altmann. 1989. The Treatment of Errors in Learning and Training. In Developing Skills With Information Technology. John Wiley, 65--86.Google Scholar
- David Ginat. 2008. Learning from Wrong and Creative Algorithm Design. ACM SIGCSE Bulletin 40, 1 (2008), 26--30. Google ScholarDigital Library
- David Ginat and Ronit Shmallo. 2013. Constructive Use of Errors in Teaching CS1. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (SIGCSE 2013). ACM, 353--358. Google ScholarDigital Library
- A. C. Graesser. 2009. Inaugural Editorial for Journal of Educational Psychology. Journal of Educational Psychology 101, 2 (2009), 259--261.Google ScholarCross Ref
- Jean M. Griffin. 2019. Constructionism and De-Constructionism: Opposite yet Complementary Pedagogies. Constructivist Foundations 14, 3 (2019), 234--243. https://constructivist.info/14/3/234.griffin.pdfGoogle Scholar
- Jean M. Griffin, Eliot Kaplan, and Q. Burke. 2012. Debug'ems and Other Deconstruction Kits for STEM Learning. In Proceedings of the Integrated STEM Education Conference (ISEC '12), IEEE 2nd. IEEE, 1--4.Google Scholar
- Cornelia S Große and Alexander Renkl. 2004. Learning From Worked Examples: What Happens If Errors Are Included? Instructional Design for Effective and Enjoyable Computer-Supported Learning (2004), 356--364.Google Scholar
- Cornelia S. Große and Alexander Renkl. 2007. Finding and fixing errors in worked examples: Can this foster learning outcomes? Learning and Instruction 17, 6 (dec 2007), 612--634.Google ScholarCross Ref
- Idit Harel. 1988. Software Design for Learning: Children's Construction of Meaning for Fractions and LOGO Programming (Doctoral Dissertation). Ph.D. Dissertation. MIT.Google Scholar
- Brian Harrington and Nick Cheng. 2018. Tracing vs. Writing Code: Beyond the Learning Hierarchy. In Proceedings of the 49th Technical Symposium on Computer Science Education (SIGCSE '18). ACM, 423--428. Google ScholarDigital Library
- Tim Heemsoth and Aiso Heinze. 2013. The impact of incorrect examples on learning fractions: A field experiment with 6th grade students. Instructional Science 42, 4 (dec 2013), 639--657.Google Scholar
- Daniel Hoffman, Ming Lu, and Tim Pelton. 2011. A web-based generation and delivery system for active code reading. In Proceedings of the 42nd ACM Technical Symposium on Computer Science Education (SIGCSE '11). 483--488. Google ScholarDigital Library
- P. Hubwieser, J. Magenheim, A. Muhling, and A. Ruf. 2013. Towards a conceptualization of pedagogical content knowledge for computer science. In Proceedings of the ninth International Conference on Computing Education Research (ICER '13). ACM, New York NY, USA, 1--8. Google ScholarDigital Library
- Patrick Hulin, Andy Davis, Rahul Sridhar, Andrew Fasano, Cody Gallagher, Tim Leek, and Brendan Dolan-Gavitt. 2017. AutoCTF: Creating Diverse Pwnables via Automated Bug Injection. In 11th {USENIX} Workshop on Offensive Technologies - WOOT '17. Google ScholarDigital Library
- Walter Kaess and David Zeaman. 1960. POSITIVE AND NEGATIVE KNOWLEDGE OF RESULTS ON A PRESSEY-TYPE PUNCHBOARD. Journal of Educational Psychology 60, 1 (1960), 12--17.Google Scholar
- Irvin Katz and John R. Anderson. 1987. Debugging: An Analysis of Bug-Location Strategies. Human-Computer Interaction 3, 4 (1987), 351--399. Google ScholarDigital Library
- Päivi Kinnunen, Robert McCartney, Laurie Murphy, and Lynda Thomas. 2007. Through the eyes of instructors: a phenomenographic investigation of student success Categories and Subject Descriptors. In Proceeding of the third International Workshop on Computing Education Research - ICER '07. ACM, 61--73. Google ScholarDigital Library
- David Klahr and Sharon McCoy Carver. 1988. Cognitive objectives in a LOGO debugging curriculum: Instruction, learning, and transfer. In Cognitive Psychology, Vol. 20. 362--404.Google ScholarCross Ref
- Amruth N. Kumar. 2015. Solving Code-Tracing Problems and its Effect on CodeWriting Skills Pertaining to Program Semantics. In ITiCSE '15. 314--319. Google ScholarDigital Library
- Amruth N. Kumar and Neeraj Singhal. 2000. Using Java to Help Students Practice Problem-Solving. Proceedings of The Second International Conference and Exhibition on The Practical Application of Java (PA JAVA '00) (2000), 205--216.Google Scholar
- D. M. Kurland, Roy D. Pea, C. Clement, and R. Mawby. 1986. A Study of the Development of Programming Ability and Thinking Skills in High School Students. Journal of Educational Computing Research 2, 4 (1986), 429--458.Google ScholarCross Ref
- Michael J Lee. 2015. Teaching and Engaging with Debugging Puzzles (Doctoral Dissertation). Ph.D. Dissertation. University of Washington.Google Scholar
- Michael J. Lee, Faezeh Bahmani, Irwin Kwan, Jilian Laferte, Polina Charters, Amber Horvath, Fanny Luor, Jill Cao, Catherine Law, Michael Beswetherick, Sheridan Long, Margaret Burnett, and Andrew J. Ko. 2014. Principles of a Debugging-First Puzzle Game for Computing Education. In Proceedings of IEEE Symposium on Visual Languages and Human-Centric Computing, VL/HCC. IEEE, 57--64.Google Scholar
- Michael J Lee and Andrew J Ko. 2011. Personifying Programming Tool Feedback Improves Novice ProgrammersÊij Learning. In Proceedings of the seventh International Workshop on Computing Education Research (ICER '11). ACM, 109--116. Google ScholarDigital Library
- Alex Lishinski, Aman Yadav, and Richard Enbody. 2017. Students' Emotional Reactions to Programming Projects in Introduction to Programming: Measurement Approach and Influence on Learning Outcomes. In Proceedings of the thirteenth International Workshop on Computing Education Research Workshop (ICER '17). ACM, 30--38. Google ScholarDigital Library
- Raymond Lister, Elizabeth S. Adams, Sue Fitzgerald, William Fone, John Hamer, Morten Lindholm, Robert McCartney, Jan Erik Moström, Kate Sanders, Otto Seppälä, Beth Simon, and Lynda Thomas. 2004. A Multi-National Study of Reading and Tracing Skills in Novice Programmers. ACM SIGCSE Bulletin 36, 4 (2004), 119. Google ScholarDigital Library
- Raymond Lister, Colin Fidge, and Donna Teague. 2009. Further Evidence of a Relationship between Explaining, Tracing and Writing Skills in Introductory Programming. ACM SIGCSE Bulletin 41 (2009), 161. Google ScholarDigital Library
- M. Lopez, J. Whalley, P. Robbins, and R. Lister. 2008. Relationships Between Reading, Tracing and Writing Skills in Introductory Programming. In Proceedings of the 4th International Workshop on Computing Education Research (ICER '08). ACM, 101--112. Google ScholarDigital Library
- Linxiao Ma, John Ferguson, Marc Roper, Isla Ross, and Murray Wood. 2008. Using Cognitive Conflict and Visualisation to Improve Mental Models Held by Novice Programmers. In Proceedings of the 39th Technical Symposium on Computer Science Education - SIGCSE '08. ACM, NY, 342--346. Google ScholarDigital Library
- L. M. Mann, M. C. Linn, and M. Clancy. 1994. Can Tracing Tools Contribute to Programming Proficiency? The LISP Evaluation Modeler. Interactive Learning Environments 4, 1 (1994), 96--113.Google ScholarCross Ref
- Renée McCauley, Sue Fitzgerald, Gary Lewandowski, Laurie Murphy, Beth Simon, Lynda Thomas, and Carol Zander. 2008. Debugging: a review of the literature from an educational perspective. Computer Science Education 18, April 2015 (2008), 67--92.Google ScholarCross Ref
- Bruce M. McLaren, Deanne M. Adams, and Richard E. Mayer. 2015. Delayed Learning Effects with Erroneous Examples: A Study of Learning Decimals with a Web-Based Tutor. International Journal of Artificial Intelligence in Education 25, 4 (2015), 520--542.Google ScholarCross Ref
- Erica Melis. 2005. Design of Erroneous Examples for ActiveMath. Artificial Intelligence in Education (2005), 451--458. Google ScholarDigital Library
- Erica Melis, Eric Andres, Jochen Budenbender, Adrian Frischauf, Paul Libbrecht, Martin Pollet, and Carsten Ullrich. 2001. ActiveMath: A Generic and Adaptive Web-Based Learning Environment. International Journal of Artificial Intelligence in Education (IJAIED) 12 (2001), pp.385--407.Google Scholar
- Brad Miller and D. Ranum. 2014. Runestone Interactive: Tools for Creating Interactive Course Materials. In Proceedings of the First ACM Conference on Learning @ Scale (L@S '14). ACM, 213--214. Google ScholarDigital Library
- Marvin Minsky. 1994. Negative Expertise. International Journal of Expert Systems 7, 1 (1994), 13--19. Google ScholarDigital Library
- Andrés Moreno and Mike Joy. 2014. Defining and Evaluating Conflictive Animations for Programming Education: The Case of Jeliot ConAn. In Proceedings of the 45th ACM Technical Symposium on Computer Science Education - SIGCSE '14. ACM, 629--634. Google ScholarDigital Library
- L A O'Hara and R J Sternberg. 1999. Creativity and Intelligence. In Handbook of Creativity, R J Sternberg (Ed.). Cambridge University Press.Google Scholar
- Stellan Ohlsson. 1996. Learning from error and the design of task environments. International Journal of Educational Research 25, 5 (1996), 419--448.Google ScholarCross Ref
- F. Oser, C. Näpflin, C. Hofer, and P. Aerni. 2012. Towards a theory of Negative Knowledge (NK). Almost-mistakes as drivers of episodic memory amplification. Professional and Practice-based Learning 6 (2012), 53--70.Google ScholarCross Ref
- F. Oser and M. Spychinger. 2005. Lernen ist schmerzhaft: Zur Theorie des Negativen Wissens und zur Praxis der Fehlerkultur. Beltz, Weinheim.Google Scholar
- Seymour Papert. 1980. Mindstorms: Children, Computers, and Powerful Ideas. Basic Books, Inc. Google ScholarDigital Library
- Roy D. Pea, E. Soloway, and J. G. Spohrer. 1987. The buggy path to the development of programming expertise. Focus on Learning Problems in Mathematics 9, 1 (1987), 5--30.Google Scholar
- D. Perkins and F. Martin. 1989. Fragile Knowledge and Neglected Strategies in Novice Programmers. In Studying the Novice Programmer, E. Soloway and J Spohrer (Eds.). Lawrence Erlbaum Associates, Hillsdale, NJ, 213--229. Google ScholarDigital Library
- Jannik Pewny and Thorstein Holz. 2016. EvilCoder: Automated Bug Insertion. Proceedings of the 32nd Annual Conference on Computer Security Applications (2016), 214--225. Google ScholarDigital Library
- Jean Piaget. 1980. Experiments in contradictions. University of Chicago Press, Chicago.Google Scholar
- G Polya. 1957. How to Solve It. Doubleday, Garden City, NJ.Google Scholar
- Brad Richards. 2000. Bugs as Features: Teaching Network Protocols Through Debugging. Proceedings of the thirty-first SIGCSE Technical Symposium on Computer Science Education (SIGCSE '00) 32, 1 (2000), 256--259. Google ScholarDigital Library
- Anthony Robins. 2010. Learning edge momentum: a new account of outcomes in CS1. Computer Science Education 20, 1 (2010), 37--71.Google ScholarCross Ref
- Robert S. Siegler. 2002. Microgenetic Studies On Self-Explanation. In Microdevelopment: Transition Processes in Development and Learning, Nira Granott and Jim Parziale (Eds.). Cambridge University Press, 31--58.Google Scholar
- E. Soloway. 1986. Learning to program = learning to construct mechanisms and explanations., 850--858 pages. Google ScholarDigital Library
- E. Soloway and J. C. Spohrer (Eds.). 1989. Studying the novice programmer. L. Erlbaum Associates, Hillsdale, NJ. Google ScholarDigital Library
- Juha Sorva, Ville Karavirta, and Lauri Malmi. 2013. A Review of Generic Program Visualization Systems for Introductory Programming Education. ACM Transactions on Computing Education (TOCE '13) 13, 4 (2013), 1--68. Google ScholarDigital Library
- J. Sweller. 1988. Cognitive Load During Problem Solving: Effects on Learning. Cognitive Science 12, 2 (1988), 257--285.Google ScholarCross Ref
- J. Sweller and Graham A. Cooper. 1985. The Use of Worked Examples as a Substitute for Problem Solving in Learning Algebra. Cognition and Instruction 2, 1 (1985), 59--89.Google ScholarCross Ref
- J Gregory Trafton and Brian J Reiser. 1993. The Contributions of Studying Examples and Solving Problems to Skill Acquisition. In Proceedings of the 15th Annual Conference of the Cognitive Science Society, M. Polson (Ed.). Lawrence Erlbaum, Hillsdale, NJ:, 1017--1022.Google Scholar
- Dimitra Tsovaltzi, Erica Melis, Bruce M McLaren, Ann-Kristin Meyer, Michael Dietrich, and George Goguadze. 2010. Learning from Erroneous Examples: When and How do Students Benefit from them?. In Proceedings of the European Conference on Technology Enhanced Learning, LNCS (vol. 6383). Springer, Heidelberg. Google ScholarDigital Library
- Jeroen Van Merriënboer. 1990. Strategies for Programming Instruction in High School: Program Completion vs. Program Generation. Journal of Educational Computing Research 6, 3 (1990), 265--285.Google ScholarCross Ref
- Kurt VanLehn. 1999. Rule-learning events in the acquisition of a complex skill: An evaluation of CASCADE. Journal of the Learning Sciences 8, 1 (1999), 71--125.Google ScholarCross Ref
- Anne Venables, Grace Tan, and Raymond Lister. 2009. A Closer Look at Tracing, Explaining and Code Writing Skills in the Novice Programmer. Proceedings of the fifth International Workshop on Computing Education Research Workshop (ICER '09) (2009), 117--128. Google ScholarDigital Library
- J. L. Whalley, R. Lister, E. Thompson, T. Clear, P. Robbins, P. K. Kumar, and C. Prasad. 2006. An Australasian Study of Reading and Comprehension Skills in Novice Programmers, using the Bloom and SOLO Taxonomies. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. Australian Computer Society, Inc., 243--252. Google ScholarDigital Library
- R. R. Wilcox. 2016. Understanding and Applying Basic Statistical Methods Using R. John Wiley & Sons, Hoboken NJ.Google Scholar
- Benjamin Xie, Greg L Nelson, and Andrew J Ko. 2018. An Explicit Strategy to Scaffold Novice Program Tracing. 2 (2018).Google ScholarDigital Library
- Peng Zhang, Jules White, and C Schmidt. 2016. HoliCoW: Automatically Breaking Team-based Software Projects to Motivate Student Testing. In IEEE/ACM 38th International Conference on Software Engineering Companion. IEEE, 436--439. Google ScholarDigital Library
Index Terms
- Designing Intentional Bugs for Learning
Recommendations
Do bugs lead to unnaturalness of source code?
ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software EngineeringTexts in natural languages are highly repetitive and predictable because of the naturalness of natural languages. Recent research validated that source code in programming languages is also repetitive and predictable, and naturalness is an inherent ...
What’s the problem? interrogating actors to identify the root cause of concurrency bugs
AGERE 2021: Proceedings of the 11th ACM SIGPLAN International Workshop on Programming Based on Actors, Agents, and Decentralized ControlPrograms written using Communicating Event-Loops (CEL) concurrency model do not suffer from low-level data races by design but are not exempt from other concurrency bugs, such as behavioral deadlocks and message order violations.
When programmers need ...
An empirical study of bugs in build process
SAC '14: Proceedings of the 29th Annual ACM Symposium on Applied ComputingSoftware build process translates source codes into executable programs, packages the programs, generates documents, and distributes products. In this paper, we perform an empirical study to characterize build process bugs. We analyze bugs in build ...
Comments