skip to main content
10.1145/3502718.3524799acmconferencesArticle/Chapter ViewAbstractPublication PagesiticseConference Proceedingsconference-collections
research-article

Check It Off: Exploring the Impact of a Checklist Intervention on the Quality of Student-authored Unit Tests

Published: 07 July 2022 Publication History

Abstract

Software testing is an essential skill for computer science students. Prior work reports that students desire support in determining what code to test and which scenarios should be tested. In response to this, we present a lightweight testing checklist that contains both tutorial information and testing strategies to guide students in what and how to test. To assess the impact of the testing checklist, we conducted an experimental, controlled A/B study with 32 undergraduate and graduate students. The study task was writing a test suite for an existing program. Students were given either the testing checklist (the experimental group) or a tutorial on a standard coverage tool with which they were already familiar (the control group). By analyzing the combination of student-written tests and survey responses, we found students with the checklist performed as well as or better than the coverage tool group, suggesting a potential positive impact of the checklist (or at minimum, a non-negative impact). This is particularly noteworthy given the control condition of the coverage tool is the state of the practice. These findings suggest that the testing tool support does not need to be sophisticated to be effective.

References

[1]
Mahtab Alam. 2010. Software security requirements checklist. International Journal of Software Engineering, IJSE, Vol. 3, 1 (2010), 53--62.
[2]
Tiago L. Alves and Joost Visser. 2009. Static Estimation of Test Coverage. In International Working Conf. on Source Code Analysis and Manipulation. 55--64.
[3]
Maur'icio Aniche, Felienne Hermans, and Arie van Deursen. 2019. Pragmatic Software Testing Education. In ACM Technical Symposium on Computer Science Education (SIGCSE '19). 414--420.
[4]
Andrea Arcuri. 2020. Teaching Software Testing in an Algorithms and Data Structures Course. In Conference on Software Testing, Verification and Validation Workshops (ICSTW) . 419--424.
[5]
Gina R. Bai. 2022. ginaBai/TestingChecklistStudy: TestingChecklistStudyMaterials. https://doi.org/10.5281/zenodo.6466776
[6]
Gina R. Bai, Justin Smith, and Kathryn T. Stolee. 2021. How Students Unit Test: Perceptions, Practices, and Pitfalls. In ITiCSE 2021: 26th ACM Conference on Innovation and Technology in Computer Science Education . ACM, 248--254.
[7]
Lex Bijlsma, Niels Doorn, Harrie Passier, Harold Pootjes, and Sylvia Stuurman. 2021. How do Students Test Software Units?. In Int'l Conf. on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 189--198.
[8]
Michael K. Bradshaw. 2015. Ante Up: A Framework to Strengthen Student-Based Testing of Assignments. In Technical Symposium on CS Education. 488--493.
[9]
Bill Brykczynski. 1999. A Survey of Software Inspection Checklists. SIGSOFT Softw. Eng. Notes, Vol. 24, 1 (Jan. 1999), 82.
[10]
Ingrid A. Buckley and Winston S. Buckley. 2017. Teaching Software Testing using Data Structures. International Journal of Advanced Computer Science and Applications, Vol. 8, 4 (2017).
[11]
Kevin Buffardi and Juan Aguirre-Ayala. 2021. Unit Test Smells and Accuracy of Software Engineering Student Test Suites. In Conference on Innovation and Technology in Computer Science Education. 7.
[12]
Chun Yong Chong, Patanamon Thongtanunam, and Chakkrit Tantithamthavorn. 2021. Assessing the Students' Understanding and their Mistakes in Code Review Checklists: An Experience Report of 1,791 Code Review Checklist Questions from 394 Students. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 20--29.
[13]
Lucas Cordova, Jeffrey Carver, Noah Gershmel, and Gursimran Walia. 2021. A Comparison of Inquiry-Based Conceptual Feedback vs. Traditional Detailed Feedback Mechanisms in Software Testing Education: An Empirical Investigation . 87--93.
[14]
Bruno Pedracca de Souza, Rebeca Campos Motta, and Guilherme Horta Travassos. 2019. The First Version of SCENARIotCHECK: A Checklist for IoT Based Scenarios. In Brazilian Symposium on Software Engineering (Salvador, Brazil) (SBES 2019). Association for Computing Machinery, 219--223.
[15]
Simone C. dos Santos, Maria da Conceição Moraes Batista, Ana Paula C. Cavalcanti, Jones O. Albuquerque, and Silvio R.L. Meira. 2009. Applying PBL in Software Engineering Education. In 2009 22nd Conference on Software Engineering Education and Training. 182--189.
[16]
Stephen H. Edwards and Zalia Shams. 2014a. Comparing Test Quality Measures for Assessing Student-Written Tests. In 36th International Conference on Software Engineering (ICSE Companion 2014). 354--363.
[17]
Stephen H. Edwards and Zalia Shams. 2014b. Do Student Programmers All Tend to Write the Same Software Tests?. In Conference on Innovation and Technology in Computer Science Education (ITiCSE '14). 171--176.
[18]
Stephen H. Edwards, Zalia Shams, Michael Cogswell, and Robert C. Senkbeil. 2012. Running Students' Software Tests Against Each Others' Code: New Life for an Old "Gimmick". In Technical Symposium on CS Education . 221--226.
[19]
Davide Fucci and Burak Turhan. 2013. A Replicated Experiment on the Effectiveness of Test-First Development. In 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement . 103--112.
[20]
Alessio Gaspar, Sarah Langevin, Naomi Boyer, and Ralph Tindell. 2013. A Preliminary Review of Undergraduate Programming Students' Perspectives on Writing Tests, Working with Others, & Using Peer Testing. In ACM SIGITE Conference on Information Technology Education. 109--114.
[21]
Michael H. Goldwasser. 2002. A Gimmick to Integrate Software Testing Throughout the Curriculum. In Technical Symposium on CS Education. 271--275.
[22]
Orlena CZ Gotel and CW Finkelstein. 1994. An analysis of the requirements traceability problem. In Int'l Conf. on Requirements Engineering. 94--101.
[23]
Giovanni Grano, Fabio Palomba, and Harald C. Gall. 2019. Lightweight Assessment of Test-Case Effectiveness using Source-Code-Quality Indicators. IEEE Transactions on Software Engineering (2019), 1--1.
[24]
Spencer E. Harpe. 2015. How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning, Vol. 7, 6 (2015), 836--850.
[25]
Sarah Heckman, Jessica Young Schmidt, and Jason King. 2020. Integrating Testing Throughout the CS Curriculum. In Conference on Software Testing, Verification and Validation Workshops (ICSTW). 441--444.
[26]
Laura Inozemtseva and Reid Holmes. 2014. Coverage is Not Strongly Correlated with Test Suite Effectiveness. In 36th Intl. Conf. on Software Eng. (ICSE 2014). 435--445.
[27]
Ursula Jackson, Bill Z. Manaris, and Renée A. McCauley. 1997. Strategies for Effective Integration of Software Engineering Concepts and Techniques into the Undergraduate Computer Science Curriculum. In Technical Symposium on Computer Science Education (SIGCSE '97). 360--364.
[28]
Brittany Johnson, Rahul Pandita, Emerson Murphy-Hill, and Sarah Heckman. 2015. Bespoke Tools: Adapted to the Concepts Developers Know. In Foundations of Software Engineering (ESEC/FSE 2015). 878--881.
[29]
Brittany Johnson, Rahul Pandita, Justin Smith, Denae Ford, Sarah Elder, Emerson Murphy-Hill, Sarah Heckman, and Caitlin Sadowski. 2016. A Cross-Tool Communication Study on Program Analysis Tool Notifications. In ACM SIGSOFT International Symposium on Foundations of Software Engineering . 73--84.
[30]
Brittany Johnson, Yoonki Song, Emerson Murphy-Hill, and Robert Bowdidge. 2013. Why don't software developers use static analysis tools to find bugs?. In 2013 35th International Conference on Software Engineering (ICSE). 672--681.
[31]
Edward L. Jones. 2001. Integrating Testing into the Curriculum - Arsenic in Small Doses. SIGCSE Bull., Vol. 33, 1 (Feb. 2001), 337--341.
[32]
William H. Kruskal and W. Allen Wallis. 1952. Use of Ranks in One-Criterion Variance Analysis. J. Amer. Statist. Assoc., Vol. 47, 260 (1952), 583--621.
[33]
Otávio Augusto Lazzarini Lemos, Fábio Fagundes Silveira, Fabiano Cutigi Ferrari, and Alessandro Garcia. 2018. The impact of Software Testing education on code reliability: An empirical assessment. J. of Sys. and Software, Vol. 137 (2018), 497--511.
[34]
Samiha Marwan, Yang Shi, Ian Menezes, Min Chi, Tiffany Barnes, and Thomas W Price. 2021. Just a Few Expert Constraints Can Help : Humanizing Data-Driven Subgoal Detection for Novice Programming. In International Conference on Educational Data Mining. 1--13.
[35]
David A. McMeekin, Brian R. von Konsky, Elizabeth Chang, and David J.A. Cooper. 2008. Checklist Based Reading's Influence on a Developer's Understanding. In 19th Australian Conference on Software Engineering (aswec 2008). 489--496.
[36]
Fabio Palomba, Annibale Panichella, Andy Zaidman, Rocco Oliveto, and Andrea De Lucia. 2016. Automatic Test Case Generation: What if Test Code Quality Matters?. In International Symposium on Software Testing and Analysis. 130--141.
[37]
Kai Petersen and Jefferson Seide Mollé ri. 2021. Preliminary Evaluation of a Survey Checklist in the Context of Evidence-based Software Engineering Education. In International Conference on Evaluation of Novel Approaches to Software Engineering, Raian Ali, Hermann Kaindl, and Leszek A. Maciaszek (Eds.). 437--444.
[38]
Raphael Pham, Stephan Kiesling, Olga Liskin, Leif Singer, and Kurt Schneider. 2014. Enablers, Inhibitors, and Perceptions of Testing in Novice Software Teams. In International Symposium on Foundations of Software Engineering. 30--40.
[39]
Leandro Sales Pinto, Saurabh Sinha, and Alessandro Orso. 2012. Understanding Myths and Realities of Test-suite Evolution. In Foundations of Software Engineering (FSE '12). Article 33, bibinfonumpages11 pages.
[40]
Upsorn Praphamontripong, Mark Floryan, and Ryan Ritzo. 2020. A Preliminary Report on Hands-On and Cross-Course Activities in a College Software Testing Course. In Conference on Software Testing, Verification and Validation Workshops (ICSTW). 445--451.
[41]
Guoping Rong, Jingyi Li, Mingjuan Xie, and Tao Zheng. 2012. The Effect of Checklist in Code Review for Inexperienced Students: An Empirical Study. In Conference on Software Engineering Education and Training. 120--124.
[42]
Lilian Passos Scatalon, Jeffrey C. Carver, Rogério Eduardo Garcia, and Ellen Francine Barbosa. 2019. Software Testing in Introductory Programming Courses: A Systematic Mapping Study. In Technical Symposium on Computer Science Education (SIGCSE '19). 421--427.
[43]
Lilian P. Scatalon, Jorge M. Prates, Draylson M. de Souza, Ellen F. Barbosa, and Rogério E. Garcia. 2017. Towards the Role of Test Design in Programming Assignments. In 2017 IEEE 30th Conference on Software Engineering Education and Training (CSEE&T) . 170--179.
[44]
Zalia Shams and Stephen H. Edwards. 2013. Toward Practical Mutation Analysis for Evaluating the Quality of Student-Written Software Tests. In ACM Conference on International Computing Education Research (ICER '13). 53--58.
[45]
Jaime Spacco and William Pugh. 2006. Helping Students Appreciate Test-driven Development (TDD). In Symposium on Object-oriented Programming Systems, Languages, and Applications (OOPSLA '06). 907--913.
[46]
Davide Spadini, Martin Schvarcbacher, Ana-Maria Oprescu, Magiel Bruntink, and Alberto Bacchelli. 2020. Investigating Severity Thresholds for Test Smells. In Mining Software Repositories (MSR '20). 311--321.
[47]
Matthew S. Thiese, Brenden Ronna, and Ulrike Ott. 2016. P value interpretations and considerations. Journal of Thoracic Disease, Vol. 8, 9 (2016).
[48]
Jeffrey Voas. 1997. How assertions can increase test effectiveness. IEEE Software, Vol. 14, 2 (Mar 1997), 118--119.
[49]
Wick, Michael and Stevenson, Daniel and Wagner, Paul. 2005. Using Testing and JUnit Across the Curriculum . In 36th SIGCSE Technical Symposium on Computer Science Education (SIGCSE '05). 236--240.
[50]
Laurie Williams, E. Michael Maximilien, and Mladen Vouk. 2003. Test-driven development as a defect-reduction practice. In 14th International Symposium on Software Reliability Engineering, 2003. ISSRE 2003. 34--45.

Cited By

View all
  • (2025)Instructor-Written Hints as Automated Test Suite Quality FeedbackProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701866(910-916)Online publication date: 12-Feb-2025
  • (2024)Insights from the Field: Exploring Students' Perspectives on Bad Unit Testing PracticesProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653643(101-107)Online publication date: 3-Jul-2024
  • (2024)Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication StudyProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671102(55-64)Online publication date: 12-Aug-2024
  • Show More Cited By

Index Terms

  1. Check It Off: Exploring the Impact of a Checklist Intervention on the Quality of Student-authored Unit Tests

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ITiCSE '22: Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 1
      July 2022
      686 pages
      ISBN:9781450392013
      DOI:10.1145/3502718
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 07 July 2022

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. checklist
      2. testing education
      3. unit testing

      Qualifiers

      • Research-article

      Funding Sources

      • NSF SHF

      Conference

      ITiCSE 2022
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 552 of 1,613 submissions, 34%

      Upcoming Conference

      ITiCSE '25
      Innovation and Technology in Computer Science Education
      June 27 - July 2, 2025
      Nijmegen , Netherlands

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)37
      • Downloads (Last 6 weeks)2
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Instructor-Written Hints as Automated Test Suite Quality FeedbackProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701866(910-916)Online publication date: 12-Feb-2025
      • (2024)Insights from the Field: Exploring Students' Perspectives on Bad Unit Testing PracticesProceedings of the 2024 on Innovation and Technology in Computer Science Education V. 110.1145/3649217.3653643(101-107)Online publication date: 3-Jul-2024
      • (2024)Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication StudyProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671102(55-64)Online publication date: 12-Aug-2024
      • (2023)Do the Test Smells Assertion Roulette and Eager Test Impact Students' Troubleshooting and Debugging Capabilities?Proceedings of the 45th International Conference on Software Engineering: Software Engineering Education and Training10.1109/ICSE-SEET58685.2023.00009(29-39)Online publication date: 17-May-2023

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media