Let a thousand flowers bloom: on the uses of diversity in software testing

Software testing is hard, and a testing problem is composed of many sub-problems with different, often conflicting, solutions. Like many real-world problems, it admits no single optimal solution, but requires dexterity, and the opportunistic combination of many partial solutions. Exploration and experiment, even by practitioners, are important in real-world critical testing efforts. An important set of research results in the field endorse and codify the value of diversity in test generation. However, our current approaches to evaluating research results arguably cut against this fundamental reality: while effective testing may need true diversity, combining many partial answers, the iron logic of the research results section often imposes a totalizing vision where authors must at least pretend to present a monolithic, unitary solution, a new “king of the hill.”