Exploratory studies of the software testing methods used by novice programmers

Abstract

Computer science courses, including those at the introductory level, require students to be capable of demonstrating that the programs they write satisfy the problem specifications. This demonstration of correctness is nearly always done via testing. Yet these courses tend to offer little or no guidance to the students in selecting appropriate test data. This paper analyzes the characteristics of test data used by novice programmers in the absence of such guidance.

Test data generated by over 100 students in introductory courses at two large public institutions were analyzed for feature, statement, and validity coverage as described in the IEEE standard for unit testing. The student test data were often found inadequate to cover all but the most high-level features described in the problem specifications. Statement coverage was frequently not achieved, and most of the students did not include invalid inputs in their test suites, even if they wrote their programs to check for such invalid inputs. Students tended to do a better job in achieving these levels of coverage when testing someone else's program than they did when testing their own program.

About 2/3 of the defective programs found in the study were tested well enough by the student to reveal a defect, but in many cases the defects were not recognized even when revealed. Simple systematic testing strategies, based only on information contained in the specifications, were found capable of revealing defects in most of the defective programs, including most of those for which the student's own data were inadequate to reveal a defect.

Dr. Zweben's research is supported in part by grant CCR-8802312 from the National Science Foundation.