Skip to main content
Springer Nature Link
Log in

Controlled Experimentation of Software Product Lines

  • Chapter
  • First Online:
UML-Based Software Product Line Engineering with SMarty

  • 510 Accesses

Abstract

The process of experimentation is one of several scientific methods that can provide evidence for a proof of a theory. This process is counterpoint to the real world observation method, thus providing a reliable body of knowledge. However, in the experimentation for emerging areas and in the consolidation process in scientific and industrial communities, such as the software product line (SPL), there has been a constant lack of adequate documentation of experiments that makes it difficult to repeat, replicate, and reproduce studies in SPL. Therefore, this chapter presents a set of guidelines for the quality assessment of SPL experiments with its conceptual model to support the understanding of the proposed guidelines, as well as an ontology for SPL experiments, called OntoExper-SPL, in addition to support the teaching experimentation in SPL. Thus, these points aim to improve the planning, conduction, analysis, sharing, and documentation of SPL experiments, supporting the construction of a reliable and reference body of knowledge in such a context in addition to enabling improvement in the teaching of SPL experiments.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

eBook
USD 12.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Almeida, M.B., Bax, M.P.: An overview about ontologies: research about definitions, types, applications, evaluation and construction methods. information science. Ciência da informação 32(3), 7–20 (2003). In Portuguese

    Google Scholar 

  2. Arcuri, A., Briand, L.: A hitchhiker’s guide to statistical tests for assessing randomized algorithms in software engineering. Software Testing, Verification and Reliability, 24(3), 219–250 (2014)

    Article  Google Scholar 

  3. Asadi, M., Soltani, S., Gašević, D., Hatala, M.: The effects of visualization and interaction techniques on feature model configuration. Empir. Softw. Eng., 1–38 (2016)

    Google Scholar 

  4. Basili, V.R., Rombach, H.D.: The TAME project: towards improvement-oriented software environments. IEEE Trans. Softw. Eng. 14(6), 758–773 (1988)

    Article  Google Scholar 

  5. Calvanese, D., De Giacomo, G., Lembo, D., Lenzerini, M., Rosati, R.: Dl-lite: tractable description logics for ontologies. In: Proceedings of the 20th National Conference on Artificial Intelligence (AAAI 2005), vol. 5, pp. 602–607 (2005)

    Google Scholar 

  6. Cohen, J.: Statistical power analysis. Curr. Dir. Psychol. Sci. 1(3), 98–101 (1992)

    Article  Google Scholar 

  7. Dieste, O., Juristo, N.: Challenges of evaluating the quality of software engineering experiments. In: Perspectives on the Future of Software Engineering, pp. 159–177. Springer, Berlin (2013)

    Google Scholar 

  8. Dieste, O., Grim, A., Juristo, N., Saxena, H.: Quantitative determination of the relationship between internal validity and bias in software engineering experiments: consequences for systematic literature reviews. In: 5th International Symposium on Empirical Software Engineering and Measurement (ESEM) pp. 285–294 (2011)

    Google Scholar 

  9. Eyal-Salman, H., Seriai, A.D., Dony, C.: Feature location in a collection of product variants: combining information retrieval and hierarchical clustering. In: SEKE: Software Engineering and Knowledge Engineering, pp. 426–430 (2014)

    Google Scholar 

  10. Furtado, V.R., Vignando, H., França, V., OliveiraJr, E.: Comparing approaches for quality evaluation of software engineering experiments: an empirical study on software product line experiments. J. Comput. Sci., 1396–1429 (2019)

    Google Scholar 

  11. Furtado, V., OliveiraJr, E., Kalinowski, M.: Guidelines for promoting software product line experiments. In: Brazilian Conference on Software Components, Architecture, and Reuse, pp. 31–40. ACM, New York (2021)

    Google Scholar 

  12. Gruber, T.R.: A translation approach to portable ontology specifications. Knowl. Acquis. 5(2), 199–220 (1993)

    Article  Google Scholar 

  13. Isotani, S., Bittencourt, I.I., Barbosa, E.F., Dermeval, D., Paiva, R.O.A.: Ontology driven software engineering: a review of challenges and opportunities. IEEE Lat. Am. Trans. 13(3), 863–869 (2015)

    Article  Google Scholar 

  14. Jedlitschka, A., Ciolkowski, M., Pfahl, D.: Reporting experiments in software engineering. In: Shull, F., Singer, J., Sjøberg, D.I.K. (eds.) Guide to Advanced Empirical Software Engineering, pp. 201–228. Springer, London (2008). https://doi.org/10.1007/978-1-84800-044-5_8

    Chapter  Google Scholar 

  15. Jena, A.: Semantic web framework for Java (2007). https://jena.apache.org/

  16. Jolliffe, I.: Principal component analysis. In: International Encyclopedia of Statistical Science, pp. 1094–1096. Springer, Berlin (2011)

    Google Scholar 

  17. Kampenes, V.: Quality of design, analysis and reporting of software engineering experiments: a systematic review. Ph.D. Thesis, Department of Informatics, Faculty of Mathematics and Natural Sciences, University of Oslo (2007)

    Google Scholar 

  18. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., Rosenberg, J.: Preliminary guidelines for empirical research in software engineering. IEEE Trans. Softw. Eng. 28(8), 721–734 (2002)

    Article  Google Scholar 

  19. Kitchenham, B., Al-Khilidar, H., Babar, M.A., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., Zhu, L.: Evaluating guidelines for reporting empirical software engineering studies. Empir. Softw. Eng. 13(1), 97–121 (2008)

    Article  Google Scholar 

  20. Kitchenham, B., Sjøberg, D.I.K., Brereton, O.P., Budgen, D., Dybå, T., Höst, M., Pfahl, D., Runeson, P.: Can we evaluate the quality of software engineering experiments? In: International Symposium on Empirical Software Engineering and Measurement, pp. 1–8 (2010)

    Google Scholar 

  21. Kitchenham, B.A., Budgen, D., Brereton, P.: Evidence-Based Software Engineering and Systematic Reviews, vol. 4. CRC Press, Boca Raton (2016)

    Google Scholar 

  22. Lamy, J.B.: Owlready: ontology-oriented programming in python with automatic classification and high level constructs for biomedical ontologies. Artif. Intell. Med. 80, 11–28 (2017)

    Article  Google Scholar 

  23. LiZhang, X.L.: An evolutionary methodology for optimized feature selection in software product lines. In: International Conference on Software Engineering and Knowledge Engineering, SEKE (2014)

    Google Scholar 

  24. Lopez-Herrejon, R.E., Linsbauer, L., Galindo, J.A., Parejo, J.A., Benavides, D., Segura, S., Egyed, A.: An assessment of search-based techniques for reverse engineering feature models. J. Syst. Softw. 103, 353–369 (2015)

    Article  Google Scholar 

  25. Machado, I.d.C., Silveira Neto, P.A.d.M., Almeida, E.S.d., Meira, S.R.d.L.: Riple-te: a process for testing software product lines. In: SEKE, pp. 711–716 (2011)

    Google Scholar 

  26. McKinney, W.: Data structures for statistical computing in python. In: Proceedings of the 9th Python in Science Conference, pp. 51–56 (2010)

    Google Scholar 

  27. Mendonca, F.M.: OntoForInfoScience: methodology for building ontologies by information scientists – a practical application in the development of the ontology about components of human blood (Hemonto) (2015)

    Google Scholar 

  28. Monteiro, F.: Modelagem conceitual: a construção de uma ontologia sobre avaliação do ciclo de vida (acv) para fomentar a disseminação de seus conceitos (2007)

    Google Scholar 

  29. Petersen, K., Vakkalanka, S., Kuzniarz, L.: Guidelines for conducting systematic mapping studies in software engineering: an update. Inf. Softw. Technol. 64, 1–18 (2015)

    Article  Google Scholar 

  30. Prud’hommeaux, E., Seaborne, A.: SPARQL Query Language for RDF. W3C Recommendation (2008). http://www.w3.org/TR/rdf-sparql-query/

  31. Pucci, J.N.: Supporting the execution of controlled experiments using an ontology for em packaging: the tool OntoExpTool. Master’s Thesis, Paulista State University (UNESP), São José do Rio Preto (2015). 105 p. In Portuguese

    Google Scholar 

  32. Reinhartz-Berger, I., Figl, K., Haugen, Ø.: Comprehending feature models expressed in CVL. In: International Conference on Model Driven Engineering Languages and Systems, pp. 501–517 (2014)

    Google Scholar 

  33. Ribeiro, M., Borba, P., Kästner, C.: Feature maintenance with emergent interfaces. In: 36th International Conference on Software Engineering, pp. 989–1000 (2014)

    Google Scholar 

  34. Rodrigues, I.P., Bacelo, A.P.T., Silveira, M.S., Campos, M.d.B., Rodrigues, E.M.: Evaluating the representation of user interface elements in feature models: an empirical study. In: SEKE, pp. 628–633 (2016)

    Google Scholar 

  35. Santos, W.B., Almeida, E.S.d., Meira, S.R.d.L.: Tirt: A traceability information retrieval tool for software product lines projects. In: Euromicro Conference on Software Engineering and Advanced Applications, pp. 93–100 (2012)

    Google Scholar 

  36. Santos, A.R., Machado, I.d.C., Almeida, E.S.d.: Riple-hc: Javascript systems meets spl composition. In: International Systems and Software Product Line Conference, pp. 154–163 (2016)

    Google Scholar 

  37. Silveira Neto, P.A.d.M., Machado, I.d.C., Cavalcanti, Y.C., Almeida, E.S.d., Garcia, V.C., Meira, S.R.d.L.: A regression testing approach for software product lines architectures. In: Software Components, Architectures and Reuse (SBCARS), pp. 41–50 (2010)

    Google Scholar 

  38. Silveira Neto, P.A.d.M., Machado, I.d.C., Cavalcanti, Y.C., Almeida, E.S.d., Garcia, V.C., Meira, S.R.d.L.: An experimental study to evaluate a spl architecture regression testing approach. In: Information Reuse and Integration (IRI), pp. 608–615 (2012)

    Google Scholar 

  39. Sjøberg, D.I.K., Anda, B., Arisholm, E., Dyba, T., Jorgensen, M., Karahasanovic, A., Koren, E.F., Vokác, M.: Conducting realistic experiments in software engineering. Empir. Softw. Eng., 17–26 (2002)

    Google Scholar 

  40. Teixeira, E.O.: Quality analysis of controlled experiments in the context of empirical software engineering. Master’s Thesis, Federal University of Pernambuco, Recife (2014). 109 p. In Portuguese

    Google Scholar 

  41. Van Rossum, G., Drake Jr, F.L.: Python tutorial. In: Centrum voor Wiskunde en Informatica Amsterdam (1995)

    Google Scholar 

  42. Vignando, H.: Ontoexper-spl: an ontology to support software product line experiments experiments. Master’s Thesis, State University of Maringá, Maringá-PR (2020). In Portuguese

    Google Scholar 

  43. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in Software Engineering. Springer Science & Business Media, Berlin (2012)

    Book  MATH  Google Scholar 

Download references

Acknowledgements

Dr. Igor Steinmacher receives a CNPq PQ2 Research Productivity Fellowship (process #313067/2020-1). Viviane R. Furtado would like to thank CAPES/Brazil (code 001) for supporting this work. Edson Oliveira Jr. would like to thank CAPES (PROCAD) for supporting this work.

Author information

Authors and Affiliations

  1. State University of Maringá, Informatics Department, Maringá, Paraná, Brazil

    Viviane R. Furtado, Henrique Vignando & Carlos D. Luz

  2. Northern Arizona University, Flagstaff, AZ, USA

    Igor F. Steinmacher

  3. Federal University of Technology, Campo Mourão, Paraná, Brazil

    Igor F. Steinmacher

  4. Pontifical Catholic University of Rio de Janeiro, Informatics Department, Rio de Janeiro, Brazil

    Marcos Kalinowski

  5. State University of Maringá, Informatics Department, Maringá, Paraná, Brazil

    Edson OliveiraJr

Authors
  1. Viviane R. Furtado
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henrique Vignando
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Carlos D. Luz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Igor F. Steinmacher
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Marcos Kalinowski
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Edson OliveiraJr
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Informatics Department, State University of Maringá, Maringá, Paraná, Brazil

    Edson OliveiraJr

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Furtado, V.R., Vignando, H., Luz, C.D., Steinmacher, I.F., Kalinowski, M., OliveiraJr, E. (2023). Controlled Experimentation of Software Product Lines. In: OliveiraJr, E. (eds) UML-Based Software Product Line Engineering with SMarty. Springer, Cham. https://doi.org/10.1007/978-3-031-18556-4_19

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-18556-4_19

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-18555-7

  • Online ISBN: 978-3-031-18556-4

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation