Abstract
Replications play a key role in Empirical Software Engineering by allowing the community to build knowledge about which results or observations hold under which conditions. Therefore, not only can a replication that produces similar results as the original experiment be viewed as successful, but a replication that produce results different from those of the original experiment can also be viewed as successful. In this paper we identify two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replications, in which the same research question is evaluated by using a different experimental procedure. The focus of this paper is on exact replications. We further explore them to identify two sub-categories: dependent replications, where researchers attempt to keep all the conditions of the experiment the same or very similar and independent replications, where researchers deliberately vary one or more major aspects of the conditions of the experiment. We then discuss the role played by each type of replication in terms of its goals, benefits, and limitations. Finally, we highlight the importance of producing adequate documentation for an experiment (original or replication) to allow for replication. A properly documented replication provides the details necessary to gain a sufficient understanding of the study being replicated without requiring the replicator to slavishly follow the given procedures.
Similar content being viewed by others
References
Basili V, Shull F et al (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25(4):456–473
Brooks A, Roper M et al (2007) Replication's role in software engineering. In: Shull F, Singer J, Sjoberg DIK (eds) Guide to advanced empirical software engineering. Springer, London, pp 365–379
Cozby (2007) Methods in behavioral research. McGraw-Hill, New York
Lindvall M, Rus I et al (2005) An evolutionary testbed for software technology evolution. Innovations in Systems and Software Engineering—A NASA Journal 1(1):3–11
Miller J (2000) Applying meta-analytical procedures to software engineering experiments. J Syst Softw 54(1):29–39
Shull F, Mendonca M et al (2004) Knowledge-sharing issues in experimental software engineering. Empirical Software Engineering—An International Journal 9(1):111–137
Vegas S, Juristo N et al (2006) Analysis of the influence of communication between researchers on an experiment replication. Proceedings of 5th International Symposium on Empirical Software Engineering, Rio de Janeiro, Brazil, pp 28–37
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor: Claes Wohlin
Rights and permissions
About this article
Cite this article
Shull, F.J., Carver, J.C., Vegas, S. et al. The role of replications in Empirical Software Engineering. Empir Software Eng 13, 211–218 (2008). https://doi.org/10.1007/s10664-008-9060-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10664-008-9060-1