The Role of Controlled Experiments Working Group Results

Abstract

The purpose of this working group was to identify which role controlled experiments play in the field of empirical Software Engineering. The discussions resulted in a list of motivational factors, challenges, and improvements suggestions. The main outcome is that, although the empirical Software Engineering community, over the last 14 years, has matured with regard to doing controlled experiments, there is still room for improvement. By now the community has understood under which conditions it is possible to empirically evaluate Software Engineering methods, techniques, and tools, but the way controlled experiments are designed, performed, and reported still lacks the level of quality of other disciplines such as social sciences or medicine. The generalizability of the results of controlled experiments is one major concern. Furthermore, more emphasis should be put on the role of empirical Software Engineering education.