Mutating Matters: Analyzing the Influence of Mutation Testing in Programming Courses
Pages 151 - 157
Abstract
Mutation testing is used to gauge the quality of software test suites by introducing small faults, called "mutations'', into code to assess if a test suite can detect them. Although it has been applied extensively in the software industry, mutation testing's use in programming courses faces both computational and pedagogical barriers. This study examines the impact of mutation testing on student performance in a post-CS2 Data Structures and Algorithms course with 3-4 week life-cycle programming projects. We collected a semester of data with projects using only code coverage (control group) and another semester that used mutation testing (experimental group). We investigated three aspects of mutation testing impact: the quality of student-written test suites, the correctness and complexity of students' solution code, and the degree of incremental test writing. Our findings suggest that students using mutation testing, as a group, demonstrated higher quality test suites and wrote better solution code compared to students using traditional code coverage methods. Students using mutation testing were more likely to exhibit incremental testing practices.
References
[1]
Kalle Aaltonen, Petri Ihantola, and Otto Seppäl"a. 2010. Mutation analysis vs. code coverage in automated assessment of students' testing skills. In Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion. 153--160.
[2]
James H Andrews, Lionel C Briand, and Yvan Labiche. 2005. Is mutation an appropriate tool for testing experiments?. In Proceedings of the 27th international conference on Software engineering. 402--411.
[3]
Maurício Aniche, Felienne Hermans, and Arie Van Deursen. 2019. Pragmatic software testing education. In Proceedings of the 50th ACM Technical Symposium on Computer Science Education. 414--420.
[4]
Moritz Beller, Chu-Pan Wong, Johannes Bader, Andrew Scott, Mateusz Machalica, Satish Chandra, and Erik Meijer. 2021. What it would take to use mutation testing in industry?a study at facebook. In 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE, 268--277.
[5]
Yiqun T Chen, Rahul Gopinath, Anita Tadakamalla, Michael D Ernst, Reid Holmes, Gordon Fraser, Paul Ammann, and René Just. 2020. Revisiting the relationship between fault detection, test adequacy criteria, and test set size. In Proceedings of the 35th IEEE/ACM international conference on automated software engineering. 237--249.
[6]
Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. Pit: a practical mutation testing tool for java. In Proceedings of the 25th international symposium on software testing and analysis. 449--452.
[7]
Tom Copeland. 2005. PMD applied. Vol. 10. Centennial Books San Francisco.
[8]
Murial Daran and Pascale Thévenod-Fosse. 1996. Software error analysis: A real case study involving real faults and mutations. ACM SIGSOFT Software Engineering Notes, Vol. 21, 3 (1996), 158--171.
[9]
Mickaël Delahaye and Lydie Du Bousquet. 2015. Selecting a software engineering tool: lessons learnt from mutation analysis. Software: Practice and Experience, Vol. 45, 7 (2015), 875--891.
[10]
Richard A DeMillo, Richard J Lipton, and Frederick G Sayward. 1978. Hints on test data selection: Help for the practicing programmer. Computer, Vol. 11, 4 (1978), 34--41.
[11]
Stephen H Edwards. 2004. Using software testing to move students from trial-and-error to reflection-in-action. In Proceedings of the 35th SIGCSE technical symposium on Computer science education. 26--30.
[12]
Stephen H Edwards and Manuel A Perez-Quinones. 2008. Web-CAT: automatically grading programming assignments. In Proceedings of the 13th annual conference on Innovation and technology in computer science education. 328--328.
[13]
Geoffrey K Gill and Chris F Kemerer. 1991. Cyclomatic complexity density and software maintenance productivity. IEEE transactions on software engineering, Vol. 17, 12 (1991), 1284--1288.
[14]
Michael H Goldwasser. 2002. A gimmick to integrate software testing throughout the curriculum. ACM SIGCSE Bulletin, Vol. 34, 1 (2002), 271--275.
[15]
Rahul Gopinath, Carlos Jensen, and Alex Groce. 2014. Code coverage for suite evaluation by developers. In Proceedings of the 36th international conference on software engineering. 72--82.
[16]
Braxton Hall and Elisa Baniassad. 2022. Evaluating the quality of student-written software tests with curated mutation analysis. In Proceedings of the 2022 ACM SIGPLAN International Symposium on SPLASH-E. 24--34.
[17]
Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th international conference on software engineering. 435--445.
[18]
Edward L Jones. 2000. Software testing in the computer science curriculum--a holistic approach. In Proceedings of the Australasian conference on Computing education. 153--157.
[19]
René Just. 2014. The Major mutation framework: Efficient and scalable mutation analysis for Java. In Proceedings of the 2014 international symposium on software testing and analysis. 433--436.
[20]
René Just, Darioush Jalali, Laura Inozemtseva, Michael D Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. In Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering. 654--665.
[21]
Ayaan M Kazerouni, James C Davis, Arinjoy Basak, Clifford A Shaffer, Francisco Servant, and Stephen H Edwards. 2021. Fast and accurate incremental feedback for students? software tests using selective mutation analysis. Journal of Systems and Software, Vol. 175 (2021), 110905.
[22]
Ayaan M Kazerouni, Stephen H Edwards, T Simin Hall, and Clifford A Shaffer. 2017. DevEventTracker: Tracking development events to assess incremental development and procrastination. In Proceedings of the 2017 ACM Conference on Innovation and Technology in Computer Science Education. 104--109.
[23]
Yu-Seung Ma, Jeff Offutt, and Yong-Rae Kwon. 2006. MuJava: a mutation system for Java. In Proceedings of the 28th international conference on Software engineering. 827--830.
[24]
Rifat Sabbir Mansur, Ayaan M Kazerouni, Stephen H Edwards, and Clifford A Shaffer. 2020. Exploring the Bug Investigation Techniques of Intermediate Student Programmers. In Koli Calling'20: Proceedings of the 20th Koli Calling International Conference on Computing Education Research. 1--10.
[25]
Thomas J McCabe. 1976. A complexity measure. IEEE Transactions on software Engineering 4 (1976), 308--320.
[26]
Akbar Siami Namin and James H Andrews. 2009. The influence of size and coverage on test suite effectiveness. In Proceedings of the eighteenth international symposium on Software testing and analysis. 57--68.
[27]
Akbar Siami Namin and Sahitya Kakarla. 2011. The use of mutation in testing experiments and its sensitivity to external threats. In Proceedings of the 2011 International Symposium on Software Testing and Analysis. 342--352.
[28]
A Jefferson Offutt and Jeffrey M Voas. 1996. Subsumption of condition coverage techniques by mutation testing. Department of Information and Software Systems Engineering, George Mason University, Tech. Rep. ISSE-TR-96--100 (1996).
[29]
Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, YL Traon, and Mark Harman. 2019. Chapter six-mutation testing advances: An analysis and survey. volume 112 of Advances in Computers.
[30]
Mike Papadakis, Donghwan Shin, Shin Yoo, and Doo-Hwan Bae. 2018. Are mutation scores correlated with real fault detection? a large scale empirical study on the relationship between mutants and real faults. In Proceedings of the 40th international conference on software engineering. 537--548.
[31]
Goran Petrović, Marko Ivanković, Gordon Fraser, and René Just. 2021. Does mutation testing improve testing practices?. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). IEEE, 910--921.
[32]
Ana B Sánchez, Pedro Delgado-Pérez, Inmaculada Medina-Bulo, and Sergio Segura. 2022. Mutation testing in the wild: findings from GitHub. Empirical Software Engineering, Vol. 27, 6 (2022), 132.
[33]
Zalia Shams. 2015. Automated Assessment of Student-written Tests Based on Defect-detection Capability. Ph.,D. Dissertation. Virginia Tech.
[34]
Jaime Spacco, David Hovemeyer, William Pugh, Fawzi Emad, Jeffrey K Hollingsworth, and Nelson Padua-Perez. 2006. Experiences with marmoset: designing and using an advanced submission and testing system for programming courses. ACM Sigcse Bulletin, Vol. 38, 3 (2006), 13--17.
[35]
Jaime Spacco and William Pugh. 2006. Helping students appreciate test-driven development (TDD). In Companion to the 21st ACM SIGPLAN symposium on Object-oriented programming systems, languages, and applications. 907--913.
[36]
Filippos I Vokolos and Phyllis G Frankl. 1998. Empirical evaluation of the textual differencing regression testing technique. In Proceedings. International Conference on Software Maintenance (Cat. No. 98CB36272). IEEE, 44--53.
[37]
Arthur Henry Watson, Dolores R Wallace, and Thomas J McCabe. 1996. Structured testing: A testing methodology using the cyclomatic complexity metric. Vol. 500. US Department of Commerce, Technology Administration, National Institute of Standards and Technology.
[38]
W Eric Wong and Aditya P Mathur. 1995. Fault detection effectiveness of mutation and data flow testing. Software Quality Journal, Vol. 4, 1 (1995), 69--83.
Index Terms
- Mutating Matters: Analyzing the Influence of Mutation Testing in Programming Courses
Recommendations
Contextual Predictive Mutation Testing
ESEC/FSE 2023: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software EngineeringMutation testing is a powerful technique for assessing and improving test suite quality that artificially introduces bugs and checks whether the test suites catch them. However, it is also computationally expensive and thus does not scale to large ...
Are mutants a valid substitute for real faults in software testing?
FSE 2014: Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software EngineeringA good test suite is one that detects real faults. Because the set of faults in a program is usually unknowable, this definition is not useful to practitioners who are creating test suites, nor to researchers who are creating and evaluating tools that ...
The effect of code coverage on fault detection under different testing profiles
A-MOST '05: Proceedings of the 1st international workshop on Advances in model-based testingSoftware testing is a key procedure to ensure high quality and reliability of software programs. The key issue in software testing is the selection and evaluation of different test cases. Code coverage has been proposed to be an estimator for testing ...
Comments
Information & Contributors
Information
Published In
December 2024
292 pages
ISBN:9798400705984
DOI:10.1145/3649165
- General Chair:
- Mohsen Dorodchi,
- Program Chairs:
- Ming Zhange,
- Stephen Cooper
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 05 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
SIGCSE Virtual 2024
Sponsor:
SIGCSE Virtual 2024: 1st ACM Virtual Global Computing Education Conference
December 5 - 8, 2024
NC, Virtual Event, USA
Acceptance Rates
Overall Acceptance Rate 1,787 of 5,146 submissions, 35%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 103Total Downloads
- Downloads (Last 12 months)103
- Downloads (Last 6 weeks)44
Reflects downloads up to 02 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in