skip to main content
10.1145/2157136.2157202acmconferencesArticle/Chapter ViewAbstractPublication PagessigcseConference Proceedingsconference-collections
research-article

Running students' software tests against each others' code: new life for an old "gimmick"

Published: 29 February 2012 Publication History

Abstract

At SIGCSE 2002, Michael Goldwasser suggested a strategy for adding software testing practices to programming courses by requiring students to turn in tests along with their solutions, and then running every student's tests against every other student's program. This approach provides a much more robust environment for assessing the quality of student-written tests, and also provides more thorough testing of student solutions. Although software testing is included as a regular part of many more programming courses today, the all-pairs model of executing tests is still a rarity. This is because student-written tests, such as JUnit tests written for Java programs, are now more commonly written in the form of program code themselves, and they may depend on virtually any aspect of their author's own solution. These dependencies may keep one student's tests from even compiling against another student's program. This paper discusses the problem and presents a novel solution for Java that uses bytecode rewriting to transform a student's tests into a form that uses reflection to run against any other solution, regardless of any compile-time dependencies that may have been present in the original tests. Results of applying this technique to two assignments, encompassing 147 student programs and 240,158 individual test case runs, shows the feasibility of the approach and provides some insight into the quality of both student tests and student programs. An analysis of these results is presented.

References

[1]
J. Spacco and W. Pugh, "Helping students appreciate test-driven development (TDD)," presented at the Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications, Portland, Oregon, USA, 2006.
[2]
M. H. Goldwasser, "A gimmick to integrate software testing throughout the curriculum," presented at the Proceedings of the 33rd SIGCSE Technical Symposium on Computer Science Education, ACM, New York, NY, pp. 271--275, 2002.
[3]
M. Hauswirth, et al., "The JavaFest: a collaborative learning technique for Java programming courses," presented at the Proceedings of the 6th international symposium on Principles and practice of programming in Java, Modena, Italy, 2008.
[4]
W. Marrero and A. Settle, "Testing first: emphasizing testing in early programming courses," SIGCSE Bull., vol. 37, pp. 4--8, 2005.
[5]
S. H. Edwards, "Rethinking computer science education from a test-first perspective," presented at the Companion of the 18th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, Anaheim, CA, USA, 2003.
[6]
D. Jackson and M. Usher, "Grading student programs using ASSYST," SIGCSE Bull., vol. 29, pp. 335--339, 1997.
[7]
B. Cole, et al., "Improving your software using static analysis to find bugs," presented at the Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications, Portland, Oregon, USA, 2006.
[8]
K. Aaltonen, et al., "Mutation analysis vs. code coverage in automated assessment of students' testing skills," presented at the Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion, Reno/Tahoe, Nevada, USA, 2010.
[9]
S. Elbaum, et al., "Bug Hunt: Making Early Software Testing Lessons Engaging and Affordable," presented at the 29th International Conference on Software Engineering (ICSE), pp. 688--697, 2007.
[10]
Beck, K. Aim, fire (test-first coding). IEEE Software, 18(5): 87--89, Sept./Oct. 2001.
[11]
Beck, K. Test-Driven Development: By Example. Addison-Wesley, Boston, MA, 2003.
[12]
S. Chiba and M. Nishizawa, "An easy-to-use toolkit for efficient Java bytecode translators," presented at the Proceedings of the 2nd international conference on Generative programming and component engineering (GPCE '03), Springer-Verlag New York, Inc., New York, NY, USA, pp. 364--376, 2003.

Cited By

View all
  • (2024)Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication StudyProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671102(55-64)Online publication date: 12-Aug-2024
  • (2023)A Model of How Students Engineer Test Cases With FeedbackACM Transactions on Computing Education10.1145/362860424:1(1-31)Online publication date: 20-Oct-2023
  • (2023)Investigating the Potential of GPT-3 in Providing Feedback for Programming AssessmentsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588852(292-298)Online publication date: 29-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SIGCSE '12: Proceedings of the 43rd ACM technical symposium on Computer Science Education
February 2012
734 pages
ISBN:9781450310987
DOI:10.1145/2157136
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 February 2012

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. automated assessment
  2. bytecode transformation
  3. programming assignments
  4. reflection
  5. software testing
  6. test coverage
  7. test-driven development

Qualifiers

  • Research-article

Conference

SIGCSE '12
Sponsor:
SIGCSE '12: The 43rd ACM Technical Symposium on Computer Science Education
February 29 - March 3, 2012
North Carolina, Raleigh, USA

Acceptance Rates

SIGCSE '12 Paper Acceptance Rate 100 of 289 submissions, 35%;
Overall Acceptance Rate 1,595 of 4,542 submissions, 35%

Upcoming Conference

SIGCSE TS 2025
The 56th ACM Technical Symposium on Computer Science Education
February 26 - March 1, 2025
Pittsburgh , PA , USA

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)12
  • Downloads (Last 6 weeks)3
Reflects downloads up to 14 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication StudyProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671102(55-64)Online publication date: 12-Aug-2024
  • (2023)A Model of How Students Engineer Test Cases With FeedbackACM Transactions on Computing Education10.1145/362860424:1(1-31)Online publication date: 20-Oct-2023
  • (2023)Investigating the Potential of GPT-3 in Providing Feedback for Programming AssessmentsProceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 110.1145/3587102.3588852(292-298)Online publication date: 29-Jun-2023
  • (2023)Automated Evaluation of the Structure of Student-Written Unit TestsProceedings of the 2023 ACM Southeast Conference10.1145/3564746.3587002(56-63)Online publication date: 12-Apr-2023
  • (2023)Do the Test Smells Assertion Roulette and Eager Test Impact Students' Troubleshooting and Debugging Capabilities?Proceedings of the 45th International Conference on Software Engineering: Software Engineering Education and Training10.1109/ICSE-SEET58685.2023.00009(29-39)Online publication date: 17-May-2023
  • (2023)Cognitive Reflection in Software Verification and TestingProceedings of the 45th International Conference on Software Engineering: Software Engineering Education and Training10.1109/ICSE-SEET58685.2023.00006(1-10)Online publication date: 17-May-2023
  • (2022)Programming Pedagogy and Assessment in the Era of AI/ML: A Position PaperProceedings of the 15th Annual ACM India Compute Conference10.1145/3561833.3561843(29-34)Online publication date: 9-Nov-2022
  • (2021)Automating Testing of Visual Observed Concurrency2021 IEEE/ACM Ninth Workshop on Education for High Performance Computing (EduHPC)10.1109/EduHPC54835.2021.00010(32-42)Online publication date: Nov-2021
  • (2020)Software Testing as Medium for Peer FeedbackUnited Kingdom & Ireland Computing Education Research conference.10.1145/3416465.3416474(66-72)Online publication date: 3-Sep-2020
  • (2019)Assessing Incremental Testing Practices and Their Impact on Project OutcomesProceedings of the 50th ACM Technical Symposium on Computer Science Education10.1145/3287324.3287366(407-413)Online publication date: 22-Feb-2019
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media