skip to main content
10.1145/3528231.3528359acmconferencesArticle/Chapter ViewAbstractPublication PagesicseConference Proceedingsconference-collections
short-paper

Beyond the badge: reproducibility engineering as a lifetime skill

Published: 07 February 2023 Publication History

Abstract

Ascertaining reproducibility of scientific experiments is receiving increased attention across disciplines. We argue that the necessary skills are important beyond pure scientific utility, and that they should be taught as part of software engineering (SWE) education. They serve a dual purpose: Apart from acquiring the coveted badges assigned to reproducible research, reproducibility engineering is a lifetime skill for a professional industrial career in computer science.
SWE curricula seem an ideal fit for conveying such capabilities, yet they require some extensions, especially given that even at flagship conferences like ICSE, only slightly more than one-third of the technical papers (at the 2021 edition) receive recognition for artefact reusability. Knowledge and capabilities in setting up engineering environments that allow for reproducing artefacts and results over decades (a standard requirement in many traditional engineering disciplines), writing semi-literate commit messages that document crucial steps of a decision-making process and that are tightly coupled with code, or sustainably taming dynamic, quickly changing software dependencies, to name a few: They all contribute to solving the scientific reproducibility crisis, and enable software engineers to build sustainable, long-term maintainable, software-intensive, industrial systems. We propose to teach these skills at the undergraduate level, on par with traditional SWE topics.

References

[1]
ACM 2020. Review and Badging Artifact. https://www.ieee.org/publications/research-reproducibility.html [Online].
[2]
Carl Boettiger. 2015. An Introduction to Docker for Reproducible Research. SIGOPS Oper. Syst. Rev. 49, 1 (Jan. 2015), 71--79.
[3]
Dimitri Braininger, Wolfgang Mauerer, and Stefanie Scherzinger. 2020. Replicability and Reproducibility of a Schema Evolution Study in Embedded Databases. In Proc. EmpER 2020. 210--219.
[4]
Steven P. Callahan, Juliana Freire, Emanuele Santos, Carlos E. Scheidegger, Cláudio T. Silva, and Huy T. Vo. 2006. VisTrails: Visualization Meets Data Management. In Proc. ACM SIGMOD International Conference on Management of Data (Chicago, IL, USA) (SIGMOD '06).
[5]
Jon F. Claerbout and Martin Karrenbach. 2005. Electronic documents give reproducible research a new meaning. In SEG Technical Program Expanded Abstracts 1992. 601--604.
[6]
Sergey Fomel. 2015. Reproducible Research as a Community Effort: Lessons from the Madagascar Project. Computing in Science and Engg. 17, 1 (Jan. 2015), 20--26.
[7]
Benjamin J. Heil, Michael M. Hoffman, Florian Markowetz, Su-In Lee, Casey S. Greene, and Stephanie C. Hicks. 2021. Reproducibility standards for machine learning in the life sciences. Nature Methods 18, 10 (Aug. 2021), 1132--1135.
[8]
IEEE 2016. Report on the First IEEE Workshop on The Future of Research Curation and Research Reproducibility. https://www.ieee.org/publications/research-reproducibility.html [Online].
[9]
Bertrand Kerautret, Adrien Krähenbühl, Pascal Monasse, Miguel Colom, Daniel Lopresti, and Hugues Talbot. 2021. Reproducible Research in Pattern Recognition. Springer.
[10]
Donald E. Knuth. 1984. Literate Programming. Comput. J. 27, 2 (1984), 97--111. http://dblp.uni-trier.de/db/journals/cj/cj27.html#Knuth84
[11]
Wolfgang Mauerer, Ralf Ramsauer, Edson Ramiro Lucas Filho, and Stefanie Scherzinger. 2021. Silentium! Run-Analyse-Eradicate the Noise out of the DB/OS Stack. In Proc. Fachtagung für Datenbanksysteme für Business, Technologie und Web (BTW) 2021.
[12]
Wolfgang Mauerer and Stefanie Scherzinger. 2020. Educating Future Software Architects in the Art and Science of Analysing Software Data. In Proc. "Software Engineering im Unterricht der Hochschulen" 2020 (CEUR Workshop Proceedings, Vol. 2531). CEUR-WS.org, 56--60.
[13]
Wolfgang Mauerer and Stefanie Scherzinger. 2021. Nullius in Verba: Reproducibility for Database Systems Research, Revisited. In Proc. International Conference on Data Engineering. 2377--2380.
[14]
Wolfgang Mauerer and Stefanie Scherzinger. 2022. 1-2-3 Reproducibility for Quantum Software Experiments. In In Proc. 1st International Workshop on Quantum Software Analysis, Evolution and Reengineering (Q-SANER@SANER 2022).
[15]
Erwin B Montgomery Jr. 2019. Reproducibility in Biomedical Research: Epistemological and Statistical Problems. Academic Press.
[16]
José M. Sánchez Morgado and Aurora Brønstad (Eds.). 2021. Experimental Design and Reproducibility in Preclinical Animal Studies. Springer.
[17]
National Academies of Sciences, Engineering, and Medicine and others. 2019. Reproducibility and replicability in science. National Academies Press.
[18]
Ralf Ramsauer, Daniel Lohmann, and Wolfgang Mauerer. 2019. The list is the process: reliable pre-integration tracking of commits on mailing lists. In Proc. ICSE 2019. 807--818.
[19]
Manuel Schönberger, Maja Franz, Stefanie Scherzinger, and Wolfgang Mauerer. 2022. Peel | Pile? Cross-Framework Portability of Quantum Software. QSA@ICSA 2022.
[20]
Victoria Stodden, Friedrich Leisch, and Roger D. Peng (Eds.). 2014. Implementing Reproducible Research. CRC Press.
[21]
Michael Williams, Michael Curtis, and Kevin Mullane. 2017. Research in the biomedical sciences: Transparent and reproducible. Academic Press.
[22]
Jianjun Zhao. 2020. Quantum software engineering: Landscapes and horizons. (2020).
[23]
Thomas Zimmermann, Andreas Zeller, Peter Weissgerber, and Stephan Diehl. 2005. Mining version histories to guide software changes. Software Engineering, IEEE Transactions on 31, 6 (2005), 429--445.

Cited By

View all
  • (2025)Reproducible research policies and software/data management in scientific computing journals: a survey, discussion, and perspectivesFrontiers in Computer Science10.3389/fcomp.2024.14918236Online publication date: 15-Jan-2025
  • (2024)Skills Required for Quantum Computing: A Comprehensive Review of Recent StudiesProgramming and Computing Software10.1134/S036176882470080450:8(844-874)Online publication date: 1-Dec-2024
  • (2023)We Need More Reproducibility Content Across the Computer Science CurriculumProceedings of the 2023 ACM Conference on Reproducibility and Replicability10.1145/3589806.3600033(97-101)Online publication date: 27-Jun-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SEENG '22: Proceedings of the 4th International Workshop on Software Engineering Education for the Next Generation
May 2022
70 pages
ISBN:9781450393362
DOI:10.1145/3528231
This work is licensed under a Creative Commons Attribution-NoDerivatives International 4.0 License.

Sponsors

In-Cooperation

  • IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 February 2023

Check for updates

Author Tags

  1. reproducibility engineering
  2. teaching software engineering

Qualifiers

  • Short-paper

Funding Sources

Conference

ICSE '22
Sponsor:

Upcoming Conference

ICSE 2025

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)26
  • Downloads (Last 6 weeks)9
Reflects downloads up to 15 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Reproducible research policies and software/data management in scientific computing journals: a survey, discussion, and perspectivesFrontiers in Computer Science10.3389/fcomp.2024.14918236Online publication date: 15-Jan-2025
  • (2024)Skills Required for Quantum Computing: A Comprehensive Review of Recent StudiesProgramming and Computing Software10.1134/S036176882470080450:8(844-874)Online publication date: 1-Dec-2024
  • (2023)We Need More Reproducibility Content Across the Computer Science CurriculumProceedings of the 2023 ACM Conference on Reproducibility and Replicability10.1145/3589806.3600033(97-101)Online publication date: 27-Jun-2023
  • (2023)Software Engineering Education for the Next Generation SEENG 2022 Workshop ReportACM SIGSOFT Software Engineering Notes10.1145/3573074.357310448:1(112-116)Online publication date: 17-Jan-2023
  • (2023)RETSINA: Reproducibility and Experimentation Testbed for Signal-Strength Indoor Near Analysis2023 13th International Conference on Indoor Positioning and Indoor Navigation (IPIN)10.1109/IPIN57070.2023.10332500(1-6)Online publication date: 25-Sep-2023

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media