skip to main content
10.1145/3587102.3588781acmconferencesArticle/Chapter ViewAbstractPublication PagesiticseConference Proceedingsconference-collections
research-article

An Experience Report on Introducing Explicit Strategies into Testing Checklists for Advanced Beginners

Published: 30 June 2023 Publication History

Abstract

Software testing is a critical skill for computing students, but learning and practicing testing can be challenging, particularly for beginners. A recent study suggests that a lightweight testing checklist that contains testing strategies and tutorial information could assist students in writing quality tests. However, students expressed a desire for more support in knowing how to test the code/scenario. Moreover, the potential costs and benefits of the testing checklist are not yet examined in a classroom setting. To that end, we improved the checklist by integrating explicit testing strategies to it (ETS Checklist), which provide step-by-step guidance on how to transfer semantic information from instructions to the possible testing scenarios. In this paper, we report our experiences in designing explicit strategies in unit testing, as well as adapting the ETS Checklist as optional tool support in a CS1.5 course. With the quantitative and qualitative analysis of the survey responses and lab assignment submissions generated by students, we discuss students' engagement with the ETS Checklists. Our results suggest that students who used the checklist intervention had significantly higher quality in their student-authored test code, in terms of code coverage, compared to those who did not, especially for assignments earlier in the course. We also observed students' unawareness of their need for help in writing high-quality tests.

References

[1]
[n.d.]. EclEmma: Coverage Counters. https://www.eclemma.org/jacoco/trunk/doc/counters.html. Accessed: 2022-08--19.
[2]
[n.d.]. PITest: Mutation Operators. http://pitest.org/quickstart/mutators/. Accessed: 2022-08--19.
[3]
Paul Ammann and Jeff Offutt. 2017. Introduction to Software Testing (2 ed.). Cambridge University Press, USA.
[4]
Michael Andersson. 2017. An Experimental Evaluation of PIT's Mutation Operators., 27 pages.
[5]
Maurício Aniche, Felienne Hermans, and Arie van Deursen. 2019. Pragmatic Software Testing Education. In ACM Technical Symposium on Computer Science Education (SIGCSE '19). 414--420.
[6]
Gina R. Bai. 2023. ginaBai/ExplicitTestingStrategiesStudy:StudyMaterials. https://doi.org/10.5281/zenodo.7702953
[7]
Gina R. Bai, Kai Presler-Marshall, Thomas Price, and Kathryn T. Stolee. 2022. Check It Off: Exploring the Impact of a Checklist Intervention on the Quality of Student-written Unit Tests. In 27th ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE '22).
[8]
Gina R. Bai, Justin Smith, and Kathryn T. Stolee. 2021. How Students Unit Test: Perceptions, Practices, and Pitfalls. In ITiCSE 2021: 26th ACM Conference on Innovation and Technology in Computer Science Education. ACM, 248--254.
[9]
Lex Bijlsma, Niels Doorn, Harrie Passier, Harold Pootjes, and Sylvia Stuurman. 2021. How do Students Test Software Units?. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 189--198.
[10]
Michael K. Bradshaw. 2015. Ante Up: A Framework to Strengthen Student-Based Testing of Assignments. In Technical Symposium on CS Education. 488--493.
[11]
Ingrid A. Buckley and Winston S. Buckley. 2017. Teaching Software Testing using Data Structures. International Journal of Advanced Computer Science and Applications 8, 4 (2017).
[12]
Kevin Buffardi and Juan Aguirre-Ayala. 2021. Unit Test Smells and Accuracy of Software Engineering Student Test Suites. ACM, New York, NY, USA, 234--240.
[13]
Chun Yong Chong, Patanamon Thongtanunam, and Chakkrit Tantithamthavorn. 2021. Assessing the Students' Understanding and their Mistakes in Code Review Checklists: An Experience Report of 1,791 Code Review Checklist Questions from 394 Students. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering Education and Training (ICSE-SEET). 20--29.
[14]
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: A Practical Mutation Testing Tool for Java (Demo). In Proceedings of the 25th International Symposium on Software Testing and Analysis (Saarbrücken, Germany) (ISSTA 2016). Association for Computing Machinery, New York, NY, USA, 449--452.
[15]
Lucas Cordova, Jeffrey Carver, Noah Gershmel, and Gursimran Walia. 2021. A Comparison of Inquiry-Based Conceptual Feedback vs. Traditional Detailed Feedback Mechanisms in Software Testing Education: An Empirical Investigation. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (Virtual Event, USA) (SIGCSE '21). Association for Computing Machinery, New York, NY, USA, 87--93.
[16]
Hans Van der Meij. 1988. Constraints on Question Asking in Classrooms. Journal of Educational Psychology 80, 3 (1988), 401--405. (1988).
[17]
Sharon Nelson-Le Gall. 1981. Help-seeking: An understudied problem-solving skill in children. 1, 3 (1981), 224--246.
[18]
Vahid Garousi, Austen Rainer, Per Lauvås, and Andrea Arcuri. 2020. Software-testing education: A systematic literature mapping. Journal of Systems and Software 165 (2020), 110570.
[19]
Spencer E. Harpe. 2015. How to analyze Likert and other rating scale data. Currents in Pharmacy Teaching and Learning 7, 6 (2015), 836--850.
[20]
Sarah Heckman, Jessica Young Schmidt, and Jason King. 2020. Integrating Testing Throughout the CS Curriculum. In Conference on Software Testing, Verification and Validation Workshops (ICSTW). 441--444.
[21]
Brittany Johnson, Rahul Pandita, Emerson Murphy-Hill, and Sarah Heckman. 2015. Bespoke Tools: Adapted to the Concepts Developers Know. In Foundations of Software Engineering (ESEC/FSE 2015). 878--881.
[22]
Edward L. Jones. 2001. Integrating Testing into the Curriculum - Arsenic in Small Doses. In Proceedings of the 32nd SIGCSE Technical Symposium on Computer Science Education (SIGCSE '01). ACM, New York, NY, USA, 337--341.
[23]
Amy J Ko, Thomas D LaToza, Stephen Hull, Ellen A Ko, William Kwok, Jane Quichocho, Harshitha Akkaraju, and Rishin Pandit. 2019. Teaching explicit programming strategies to adolescents. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 469--475.
[24]
William H. Kruskal and W. Allen Wallis. 1952. Use of Ranks in One-Criterion Variance Analysis. J. Amer. Statist. Assoc. 47, 260 (1952), 583--621.
[25]
Thomas D LaToza, Maryam Arab, Dastyni Loksa, and Amy J Ko. 2020. Explicit programming strategies. Empirical Software Engineering 25, 4 (2020), 2416--2449.
[26]
Samiha Marwan, Yang Shi, Ian Menezes, Min Chi, Tiffany Barnes, and Thomas W. Price. 2021. Just a Few Expert Constraints Can Help: Humanizing Data-Driven Subgoal Detection for Novice Programming. In International Conference on Educational Data Mining.
[27]
Raphael Pham, Stephan Kiesling, Olga Liskin, Leif Singer, and Kurt Schneider. 2014. Enablers, Inhibitors, and Perceptions of Testing in Novice Software Teams. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). 30--40.
[28]
Upsorn Praphamontripong, Mark Floryan, and Ryan Ritzo. 2020. A Preliminary Report on Hands-On and Cross-Course Activities in a College Software Testing Course. In Conference on Software Testing, Verification and Validation Workshops (ICSTW). 445--451.
[29]
Thomas Price, Zhongxiu Peddycord-Liu, Veronica Catete, and Tiffany Barnes. 2017. Factors Influencing Students' Help-Seeking Behavior while Programming with Human and Computer Tutors.
[30]
Lilian Passos Scatalon, Jeffrey C. Carver, Rogério Eduardo Garcia, and Ellen Francine Barbosa. 2019. Software Testing in Introductory Programming Courses: A Systematic Mapping Study. In ACM Technical Symposium on Computer Science Education (SIGCSE '19). 421--427.
[31]
Jaime Spacco and William Pugh. 2006. Helping Students Appreciate Test-driven Development (TDD). In Symposium on Object-oriented Programming Systems, Languages, and Applications (OOPSLA '06). 907--913.
[32]
Joseph Timoney, Stephen Brown, and Deshi Ye. 2008. Experiences in Software Testing Education: Some Observations from an International Cooperation. In 2008 The 9th International Conference for Young Computer Scientists. 2686--2691.

Cited By

View all
  • (2025)Instructor-Written Hints as Automated Test Suite Quality FeedbackProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701866(910-916)Online publication date: 12-Feb-2025
  • (2024)Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication StudyProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671102(55-64)Online publication date: 12-Aug-2024

Index Terms

  1. An Experience Report on Introducing Explicit Strategies into Testing Checklists for Advanced Beginners

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      ITiCSE 2023: Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1
      June 2023
      694 pages
      ISBN:9798400701382
      DOI:10.1145/3587102
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 30 June 2023

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. checklist
      2. testing education
      3. unit testing

      Qualifiers

      • Research-article

      Funding Sources

      • NSF IUSE
      • NSF SHF

      Conference

      ITiCSE 2023
      Sponsor:

      Acceptance Rates

      Overall Acceptance Rate 552 of 1,613 submissions, 34%

      Upcoming Conference

      ITiCSE '25
      Innovation and Technology in Computer Science Education
      June 27 - July 2, 2025
      Nijmegen , Netherlands

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)38
      • Downloads (Last 6 weeks)7
      Reflects downloads up to 20 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Instructor-Written Hints as Automated Test Suite Quality FeedbackProceedings of the 56th ACM Technical Symposium on Computer Science Education V. 110.1145/3641554.3701866(910-916)Online publication date: 12-Feb-2025
      • (2024)Evaluating the Effectiveness of a Testing Checklist Intervention in CS2: An Quasi-experimental Replication StudyProceedings of the 2024 ACM Conference on International Computing Education Research - Volume 110.1145/3632620.3671102(55-64)Online publication date: 12-Aug-2024

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media