skip to main content
10.1145/3530019.3535348acmotherconferencesArticle/Chapter ViewAbstractPublication PageseaseConference Proceedingsconference-collections
research-article

An Open Science-Based Framework for Managing Experimental Data in Software Engineering

Published: 13 June 2022 Publication History

Abstract

The Experimental Software Engineering (ESE) area investigates improvements that can be considered in carried out Software Engineering (SE) experiments. During the life cycle of an experiment, different artifacts are developed, such as, processes, scripts and data. These artifacts straightforwardly contribute to increase the capability of experiments reproducibility, thus, consequently, to SE evolution. Organizing such artifacts into proper experimental/laboratory packages facilitates their comprehension for prospective findings and reproducibility. The lack of reproducibility for SE results and experiments has been explained in the current literature. However, such artifacts treatment usually fails, thus jeopardizing findings and external reproductions to confirm and audit SE experiments. In this context, this paper describes a research work in progress for the development of an open science-based framework that aids in the management of SE experimental data. Specific data and metadata groups will be formed and integrated to formally describe SE experiments, in terms of open science common practices as preservation, provenance, curation, management, and data/metadata storage. The framework will support the elaboration of experimental packages, thus it intends to promote the reproducibility of SE controlled experiments and quasi-experiments.

References

[1]
Carlos E Anchundia 2020. Resources for reproducibility of experiments in empirical software engineering: Topics derived from a secondary study. IEEE Access 8(2020), 8992–9004.
[2]
Monya Baker. 2016. 1,500 scientists lift the lid on reproducibility.
[3]
André FR Cordeiro and Edson OliveiraJr. 2021. Open Science Practices for Software Engineering Controlled Experiments and Quasi-Experiments. In OpenScienSE. SBC, Brazil, 19–21.
[4]
Carlos Damasceno, Isotilia Melo, and Daniel Strüber. 2021. Towards Multi-Criteria Prioritization of Best Practices in Research Artifact Sharing. In OpenScienSE. SBC, Brazil, 1–6.
[5]
Foster. 2022. Foster Open Science Taxonomy. https://www.fosteropenscience.eu/foster-taxonomy/open-science?page=6.
[6]
Jesús M González-Barahona and Gregorio Robles. 2012. On the reproducibility of empirical software engineering studies based on data retrieved from development repositories. Empirical Software Engineering 17, 1 (2012), 75–89.
[7]
Bhisham C Gupta and Irwin Guttman. 2014. Statistics and probability with applications for engineers and scientists. John Wiley & Sons, New Jersey.
[8]
Matthew Hutson. 2018. Artificial intelligence faces reproducibility crisis.
[9]
Andreas Jedlitschka, Marcus Ciolkowski, and Dietmar Pfahl. 2008. Reporting experiments in software engineering. In Guide to advanced empirical software engineering. Springer, New York, 201–228.
[10]
Natalia Juristo and Ana M Moreno. 2013. Basics of software engineering experimentation. Springer Science & Business Media, Germany.
[11]
Ruchika Malhotra. 2016. Empirical research in software engineering: concepts, analysis, and applications. CRC press, London.
[12]
Daniel Mendez, Daniel Graziotin, Stefan Wagner, and Heidi Seibold. 2020. Open science in software engineering. In Contemporary empirical methods in software engineering. Springer, New York, 477–501.
[13]
Daniel Méndez Fernández, Martin Monperrus, Robert Feldt, and Thomas Zimmermann. 2019. The open science initiative of the Empirical Software Engineering journal. Empirical Software Engineering 24, 3 (2019), 1057–1060.
[14]
Foster Open Science. 2022. Foster Open Science. https://www.fosteropenscience.eu.
[15]
Ken Peffers, Marcus Rothenberger, Tuure Tuunanen, and Reza Vaezi. 2012. Design science research evaluation. In International Conference on Design Science Research in Information Systems. Springer, Springer, The United States, 398–410.
[16]
Ken Peffers, Tuure Tuunanen, Marcus A Rothenberger, and Samir Chatterjee. 2007. A design science research methodology for information systems research. Journal of management information systems 24, 3 (2007), 45–77.
[17]
Nancy Pontika, Petr Knoth, Matteo Cancellieri, and Samuel Pearce. 2015. Fostering open science to research using a taxonomy and an eLearning portal. In i-KNOW. ACM, New York, 1–8.
[18]
Roy Schmidt, Kalle Lyytinen, Mark Keil, and Paul Cule. 2001. Identifying software project risks: An international Delphi study. Journal of management information systems 17, 4 (2001), 5–36.
[19]
Forrest Shull, Janice Singer, and Dag IK Sjøberg. 2007. Guide to advanced empirical software engineering. Springer, New York.
[20]
Anselm Strauss and Juliet Corbin. 2014. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory. Thousand oaks, CA: Sage publications, New York.
[21]
Christopher S Timperley, Lauren Herckis, Claire Le Goues, and Michael Hilton. 2021. Understanding and improving artifact sharing in software engineering research. Empirical Software Engineering 26, 4 (2021), 1–41.
[22]
Scientific United Nations Educational and Cultural Organization. 2022. UNESCO Recommendation on Open Science. https://en.unesco.org/science-sustainable-future/open-science/recommendation.
[23]
Claes Wohlin, Per Runeson, Martin Höst, Magnus C Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in software engineering. Springer Science & Business Media, Germany.
[24]
Robert K Yin. 2015. Qualitative research from start to finish. Guilford publications, New York.

Cited By

View all
  • (2024)A Vision on Open Science for the Evolution of Software Engineering Research and PracticeCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663788(512-516)Online publication date: 10-Jul-2024

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences
EASE '22: Proceedings of the 26th International Conference on Evaluation and Assessment in Software Engineering
June 2022
466 pages
ISBN:9781450396134
DOI:10.1145/3530019
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2022

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Controlled Experiments and Quasi-Experiments
  2. Open Data
  3. Open Science
  4. Reproducibility.
  5. Software Engineering

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • Capes/Brazil

Conference

EASE 2022

Acceptance Rates

Overall Acceptance Rate 71 of 232 submissions, 31%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)15
  • Downloads (Last 6 weeks)2
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)A Vision on Open Science for the Evolution of Software Engineering Research and PracticeCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering10.1145/3663529.3663788(512-516)Online publication date: 10-Jul-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media