skip to main content
10.1145/1047344.1047471acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
Article

Intention-based scoring: an approach to measuring success at solving the composition problem

Published: 23 February 2005 Publication History

Abstract

Traditional methods of evaluating student programs are not always appropriate for assessment of different instructional interventions. They tend to focus on the final product rather than on the process that led to it. This paper presents intention-based scoring (IBS), an approach to measuring programming ability that looks at intermediate programs produced over the course of an implementation rather than just the one at the end. The intent is to assess a student's ability to produce algorithmically correct code on the first attempt at achieving each program goal. In other words, the goal is to answer question "How close was the student to being initially correct?" but not to speak to a student's debugging skills or ability to ultimately produce a working program. To produce an IBS, it is necessary to inspect a student's online protocol, which is simply the collection of all programs submitted to a compiler. IBS involves a three-phase process of (1) identification of the subset of all programs in a protocol that represent the initial attempts at achieving programming goals, (2) bug identification, and (3) rubric-based scoring. We conclude with an example application of IBS in the evaluation of a tutoring system for beginning programmers and also show how an IBS can be broken down by the underlying bug categories to reveal more subtle differences.

References

[1]
C. Daly and J. Waldron. Assessing the assessment of programming ability. In Proceedings of the 35th Technical Symposium on Computer Science Education (SIGCSE), pages 210--213, Norfolk, VA, 2004. ACM Press.
[2]
W. D. Gray and J. R. Anderson. Change-episodes in coding: When and how do programmers change their code? In G. M. Olson, S. Sheppard, and E. Soloway, editors, Empirical Studies of Programmers: Second Workshop, pages 185--197. Ablex, Norwood, NJ, 1987.
[3]
M. Guzdial, L. Hohmann, M. Konneman, C. Walton, and E. Soloway. Supporting programming and learning-to-program with an integrated cad and scaffolding workbench. Interactive Learning Environments, 6(1&2):143--179, 1998.
[4]
M. C. Jadud. A first look at novice compilation behavior using bluej. In 16th Annual Workshop of the Psychology of Programming Interest Group (PPIG 2004), Institute of Technology, Carlow, Ireland, April 2004.
[5]
W. L. Johnson. Understanding and debugging novice programs. Artificial Intelligence, 42:51--97, 1990.
[6]
H. C. Lane and K. VanLehn. A dialogue-based tutoring system for beginning programming. In Proceedings of the Seventeenth International Florida Artificial Intelligence Research Society Conference (FLAIRS), pages 449--454, Miami Beach, FL, 2004. AAAI Press.
[7]
H. C. Lane and K. VanLehn. Teaching program planning skills to novices with natural language tutoring. In S. Fitzgerald and M. Guzdial, editors, Computer Science Education. Swets and Zeitlinger, September 2005. Special issue on doctoral research in CS Education.
[8]
M. McCracken, V. Almstrum, D. Diaz, M. Guzdial, D. Hagan, Y. B.-D. Kolikant, C. Laxer, L. Thomas, I. Utting, and T. Wilusz. A multi-national, multi-institutional study of assessment of programming skills of first-year cs students. ACM SIGCSE Bulliten, 33(4):125--140, 2001. Report by the ITiCSE 2001 Working Group on Assessment of Programming Skills of First-year CS.
[9]
R. S. Rist. Program Structure and Design. Cognitive Science, 19:507--562, 1995.
[10]
E. Soloway and K. Ehrlich. Empirical studies of programming knowledge. IEEE Transactions on Software and Engineering, SE-10(5):595--609, September 1984.
[11]
E. Soloway and J. C. Spohrer, editors. Studying the Novice Programmer. Ablex Corp., Norwood, New Jersey, 1989.
[12]
J. C. Spohrer and E. Soloway. Putting it all together is hard for novice programmers. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, Tucson, Arizona, November 12-15 1985.
[13]
J. C. Spohrer, E. Soloway, and E. Pope. A goal/plan analysis of buggy pascal programs. In Soloway and Spohrer {11}, pages 355--399.

Cited By

View all
  • (2023)An Empirical Evaluation of Live Coding in CS1Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600122(476-494)Online publication date: 7-Aug-2023
  • (2021)Development of the metric of determination probabilistic distance to solution in difficult problem areasМОДЕЛИРОВАНИЕ, ОПТИМИЗАЦИЯ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИ10.26102/2310-6018/2021.32.1.0069:1(32)(6-7)Online publication date: 31-Jan-2021
  • (2020)A review on process-oriented approaches for analyzing novice solutions to programming problemsResearch and Practice in Technology Enhanced Learning10.1186/s41039-020-00130-y15:1Online publication date: 7-Apr-2020
  • Show More Cited By

Index Terms

  1. Intention-based scoring: an approach to measuring success at solving the composition problem

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGCSE '05: Proceedings of the 36th SIGCSE technical symposium on Computer science education
    February 2005
    610 pages
    ISBN:1581139977
    DOI:10.1145/1047344
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 23 February 2005

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. intelligent tutoring systems
    2. intention-based scoring
    3. novice programming
    4. online protocols
    5. structured programming

    Qualifiers

    • Article

    Conference

    SIGCSE05
    Sponsor:
    SIGCSE05: Technical Symposium on Computer Science Education
    February 23 - 27, 2005
    Missouri, St. Louis, USA

    Acceptance Rates

    Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

    Upcoming Conference

    SIGCSE TS 2025
    The 56th ACM Technical Symposium on Computer Science Education
    February 26 - March 1, 2025
    Pittsburgh , PA , USA

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)3
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 08 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2023)An Empirical Evaluation of Live Coding in CS1Proceedings of the 2023 ACM Conference on International Computing Education Research - Volume 110.1145/3568813.3600122(476-494)Online publication date: 7-Aug-2023
    • (2021)Development of the metric of determination probabilistic distance to solution in difficult problem areasМОДЕЛИРОВАНИЕ, ОПТИМИЗАЦИЯ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИ10.26102/2310-6018/2021.32.1.0069:1(32)(6-7)Online publication date: 31-Jan-2021
    • (2020)A review on process-oriented approaches for analyzing novice solutions to programming problemsResearch and Practice in Technology Enhanced Learning10.1186/s41039-020-00130-y15:1Online publication date: 7-Apr-2020
    • (2016)Analyzing novice programmers' response to compiler error messagesJournal of Computing Sciences in Colleges10.5555/2835377.283538631:3(53-61)Online publication date: 1-Jan-2016
    • (2014)An Analysis of Java Programming Behaviors, Affect, Perceptions, and Syntax Errors among Low-Achieving, Average, and High-Achieving Novice ProgrammersJournal of Educational Computing Research10.2190/EC.49.3.b49:3(293-325)Online publication date: 4-Apr-2014
    • (2011)Mind your languageProceedings of the 10th SIGPLAN symposium on New ideas, new paradigms, and reflections on programming and software10.1145/2048237.2048241(3-18)Online publication date: 22-Oct-2011
    • (2009)Affective and behavioral predictors of novice programmer achievementACM SIGCSE Bulletin10.1145/1595496.156292941:3(156-160)Online publication date: 6-Jul-2009
    • (2009)Coarse-grained detection of student frustration in an introductory programming courseProceedings of the fifth international workshop on Computing education research workshop10.1145/1584322.1584332(75-80)Online publication date: 10-Aug-2009
    • (2009)Affective and behavioral predictors of novice programmer achievementProceedings of the 14th annual ACM SIGCSE conference on Innovation and technology in computer science education10.1145/1562877.1562929(156-160)Online publication date: 6-Jul-2009
    • (2024)Debugging in Computational Thinking: A Meta-analysis on the Effects of Interventions on Debugging SkillsJournal of Educational Computing Research10.1177/0735633124122779362:4(1087-1121)Online publication date: 20-Jan-2024
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media