A Controlled Experiment to Evaluate On-Line Process Guidance

Abstract

Process-centered software engineering environments are expected to improve an individual's comprehension of work activities, as well as improve communication and reduce conflicts among teams of software developers. We chose to investigate individual responses when using such an environment before looking for a group response. A controlled experiment compared off-line and on-line implementations of measurement-based process guidance to test three hypotheses: first, individuals accomplish their work more efficiently when using on-line process guidance as compared to off-line guidance; second, individuals are willing to use an on-line system for guidance; and third, individuals adjust their behavior based on quantitative quality models. The subjects worked alone on two testing exercises. Subjects used either an off-line or an on-line guidance technique during the first exercise, and the other technique during the second exercise. The results did not confirm the hypotheses. On average, subjects worked more efficiently when using off-line guidance, preferred off-line guidance, and ignored quantitative quality models. Post hoc analysis identified a strong correlation between subject experience level and preference for the type of guidance, a result that deserves further investigation.