A Preliminary Comparison of Using Variability Modeling Approaches to Represent Experiment Families

Background: Replication is essential to build knowledge in empirical science. Experiment replications reported in the software engineering context present variabilities on their design elements, e.g., variables, materials. The understanding of these variabilities is required to plan experimental replications within a research program. However, the lack of an explicit representation of experiments' variabilities and commonalities is likely to hamper their understanding and replication planning. Aims: The goal of this paper is to explore the use of Variability Modeling Approaches (VMAs) to represent experiment families (i.e., an original study and its replications) and to investigate the feasibility of using VMAs to support experiment replication planning. Method: We selected two experiment families, analyzed their commonalities and variabilities, and represented them using a set of well-known VMAs: Feature Model, Decision Model, and Orthogonal Variability Model. Based on the resulting models, we conducted a preliminary comparison of using such alternative VMAs to support replication planning. Results: Subjects were able to plan consistent experiment replications with the VMAs as support. Additionally, through a qualitative analysis, we identified and discuss advantages and limitations of using the VMAs. Conclusions: It is feasible to represent experiment families and to plan replications using VMAs. Based on our emerging results, we conclude that the Feature Model VMA provides the most suitable representation. Furthermore, we identified benefits in a potential merge between the Feature Model and Decision Model VMAs to provide more details to support replication planning.