skip to main content
10.1145/3545945.3569801acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
research-article

Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1

Published: 03 March 2023 Publication History

Abstract

In this paper, we make three contributions related to the selection and use of distractors (lines of code reflecting common errors or misconceptions) in Parsons problems. First, we demonstrate a process by which templates for creating distractors can be selected through the analysis of student submissions to short answer questions. Second, we describe the creation of a tool that uses these templates to automatically generate distractors for novel problems. Third, we perform a preliminary analysis of how the presence of distractors impacts performance, problem solving efficiency, and item discrimination when used in summative assessments. Our results suggest that distractors should not be used in summative assessments because they significantly increase the problem's completion time without a significant increase in problem discrimination.

References

[1]
Mary J Allen and Wendy M Yen. 2001. Introduction to measurement theory. Waveland Press.
[2]
Elizabeth L Bjork, Robert A Bjork, et al. 2011. Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. Psychology and the real world: Essays illustrating fundamental contributions to society 2, 59--68 (2011).
[3]
Elizabeth Ligon Bjork, Jeri L Little, and Benjamin C Storm. 2014. Multiple-choice testing as a desirable difficulty in the classroom. Journal of Applied Research in Memory and Cognition 3, 3 (2014), 165--170.
[4]
Derek C Briggs, Alicia C Alonzo, Cheryl Schwab, and Mark Wilson. 2006. Diag- nostic assessment with ordered multiple-choice items. Educational Assessment 11, 1 (2006), 33--63.
[5]
Paul Denny, Andrew Luxton-Reilly, and Beth Simon. 2008. Evaluating a new exam question: Parsons problems. In Proceedings of the fourth international workshop on computing education research. 113--124.
[6]
Yuemeng Du, Andrew Luxton-Reilly, and Paul Denny. 2020. A review of research on parsons problems. In Proceedings of the Twenty-Second Australasian Computing Education Conference. 195--202.
[7]
Barbara Ericson, Austin McCall, and Kathryn Cunningham. 2019. Investigating the affect and effect of adaptive parsons problems. In Proceedings of the 19th Koli Calling International Conference on Computing Education Research. 1--10.
[8]
Barbara J Ericson. 2016. Dynamically adaptive parsons problems. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 269--270.
[9]
Barbara J Ericson, Mark J Guzdial, and Briana B Morrison. 2015. Analysis of interactive features designed to enhance learning in an ebook. In Proceedings of the Eleventh Annual International Conference on International Computing Education Research. 169--178.
[10]
Barbara J Ericson, Lauren E Margulieux, and Jochen Rick. 2017. Solving parsons problems versus fixing and writing code. In Proceedings of the 17th koli calling international conference on computing education research. 20--29.
[11]
Barbara J Ericson, Kantwon Rogers, Miranda Parker, Briana Morrison, and Mark Guzdial. 2016. Identifying design principles for CS teacher Ebooks through design-based research. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 191--200.
[12]
Max Fowler, David H Smith IV, Mohammed Hassan, Seth Poulsen, Matthew West, and Craig Zilles. 2022. Reevaluating the relationship between explaining, tracing, and writing skills in CS1 in a replication study. Computer Science Education (2022), 1--29.
[13]
Stuart Garner. 2007. An Exploration of How a Technology-Facilitated Part-Complete Solution Method Supports the Learning of Computer Programming. Issues in Informing Science & Information Technology 4 (2007).
[14]
Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017. Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research 87, 6 (2017), 1082--1116.
[15]
Ibrahim Abou Halloun and David Hestenes. 1985. The initial knowledge state of college physics students. American journal of Physics 53, 11 (1985), 1043--1055.
[16]
Kyle James Harms, Jason Chen, and Caitlin L Kelleher. 2016. Distractors in Parsons problems decrease learning efficiency for young novice programmers. In Proceedings of the 2016 ACM Conference on International Computing Education Research. 241--250.
[17]
Carl C Haynes and Barbara J Ericson. 2021. Problem-Solving Efficiency and Cognitive Load for Adaptive Parsons Problems vs. Writing the Equivalent Code. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.
[18]
Petri Ihantola and Ville Karavirta. 2010. Open source widget for parson's puzzles. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education. 302--302.
[19]
Petri Ihantola and Ville Karavirta. 2011. Two-dimensional parson's puzzles: The concept, tools, and first observations. Journal of Information Technology Education 10, 2 (2011), 119--132.
[20]
Chizuko Izawa. 1966. Reinforcement-test sequences in paired-associate learning. Psychological Reports 18, 3 (1966), 879--919.
[21]
Raymond Lister, Tony Clear, Dennis J Bouvier, Paul Carter, Anna Eckerdal, Jana Jacková, Mike Lopez, Robert McCartney, Phil Robbins, Otto Seppälä, et al. 2010. Naturally occurring data as research instrument: analyzing examination re- sponses to study the novice programmer. ACM SIGCSE Bulletin 41, 4 (2010), 156--173.
[22]
Jeri L Little and Elizabeth Ligon Bjork. 2015. Optimizing multiple-choice tests as tools for learning. Memory & Cognition 43, 1 (2015), 14--26.
[23]
Mike Lopez, Jacqueline Whalley, Phil Robbins, and Raymond Lister. 2008. Rela- tionships between reading, tracing and writing skills in introductory programming. In Proceedings of the fourth international workshop on computing education research. 101--112.
[24]
Susan Matlock-Hetzel. 1997. Basic Concepts in Item and Test Analysis. (1997).
[25]
Thayer W McGahee and Julia Ball. 2009. How to read and really use an item analysis. Nurse educator 34, 4 (2009), 166--171.
[26]
Briana B Morrison, Brian Dorn, and Mark Guzdial. 2014. Measuring cognitive load in introductory CS: adaptation of an instrument. In Proceedings of the tenth annual conference on International computing education research. 131--138.
[27]
Mitchell J Nathan, Kenneth R Koedinger, Martha W Alibali, et al. 2001. Expert blind spot: When content knowledge eclipses pedagogical content knowledge. In Proceedings of the third international conference on cognitive science, Vol. 644648.
[28]
Dale Parsons and Patricia Haden. 2006. Parson's programming puzzles: a fun and effective learning tool for first programming courses. In Proceedings of the 8th Australasian Conference on Computing Education-Volume 52. 157--163.
[29]
Christopher A Rowland. 2014. The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychological bulletin 140, 6 (2014), 1432.
[30]
Travis Scheponik, Enis Golaszewski, Geoffrey Herman, Spencer Offenberger, Linda Oliva, Peter AH Peterson, and Alan T Sherman. 2019. Investigating crowd-sourcing to generate distractors for multiple-choice assessments. In National Cyber Summit. Springer, 185--201.
[31]
D. H. Smith, IV. 2022. DistractingParsonsGen - A Parsons Problem Distractor Gen- erator. https://github.com/CoffeePoweredComputers/DistractingParsonsGen/
[32]
Anne Venables, Grace Tan, and Raymond Lister. 2009. A closer look at tracing, explaining and code writing skills in the novice programmer. In Proceedings of the fifth international workshop on Computing education research workshop. 117--128.
[33]
Lev Semenovich Vygotsky and Michael Cole. 1978. Mind in society: Development of higher psychological processes. Harvard university press.
[34]
Nathaniel Weinman, Armando Fox, and Marti A Hearst. 2021. Improving instruction of programming patterns with faded parsons problems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--4.
[35]
Matthew West, Geoffrey L Herman, and Craig Zilles. 2015. Prairielearn: Mastery- based online problem solving with adaptive scoring and recommendations driven by machine learning. In 2015 ASEE Annual Conference & Exposition. 26--1238.
[36]
J Whalley, Tony Clear, and RF Lister. 2007. The many ways of the BRACElet project. Bulletin of Applied Computing and Information Technology (2007).
[37]
Craig B Zilles, Matthew West, Geoffrey L Herman, and Timothy Bretl. 2019. Every University Should Have a Computer-Based Testing Facility. In CSEDU (1). 414--420.

Cited By

View all
  • (2024)Evaluating Micro Parsons Problems as Exam QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653583(674-680)Online publication date: 3-Jul-2024
  • (2024)Neurodiverse Programmers and the Accessibility of Parsons Problems: An Exploratory Multiple-Case StudyProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630898(491-497)Online publication date: 7-Mar-2024
  • (2024)Jigsaw: A Tool for Decomposing and Planning Programming Problems2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00034(236-247)Online publication date: 2-Sep-2024
  • Show More Cited By

Index Terms

  1. Discovering, Autogenerating, and Evaluating Distractors for Python Parsons Problems in CS1

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGCSE 2023: Proceedings of the 54th ACM Technical Symposium on Computer Science Education V. 1
    March 2023
    1481 pages
    ISBN:9781450394314
    DOI:10.1145/3545945
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 03 March 2023

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. cs1
    2. distractors
    3. item discrimination
    4. parsons problems
    5. tools

    Qualifiers

    • Research-article

    Conference

    SIGCSE 2023
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

    Upcoming Conference

    SIGCSE TS 2025
    The 56th ACM Technical Symposium on Computer Science Education
    February 26 - March 1, 2025
    Pittsburgh , PA , USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)44
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Evaluating Micro Parsons Problems as Exam QuestionsProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653583(674-680)Online publication date: 3-Jul-2024
    • (2024)Neurodiverse Programmers and the Accessibility of Parsons Problems: An Exploratory Multiple-Case StudyProceedings of the 55th ACM Technical Symposium on Computer Science Education V. 110.1145/3626252.3630898(491-497)Online publication date: 7-Mar-2024
    • (2024)Jigsaw: A Tool for Decomposing and Planning Programming Problems2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC)10.1109/VL/HCC60511.2024.00034(236-247)Online publication date: 2-Sep-2024
    • (2023)Multi-Institutional Multi-National Studies of Parsons ProblemsProceedings of the 2023 Working Group Reports on Innovation and Technology in Computer Science Education10.1145/3623762.3633498(57-107)Online publication date: 22-Dec-2023
    • (2023)Generating Multiple Choice Questions for Computing Courses Using Large Language Models2023 IEEE Frontiers in Education Conference (FIE)10.1109/FIE58773.2023.10342898(1-8)Online publication date: 18-Oct-2023

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media