short-paper

Free access

Taming the Complexity of Artifact Reproducibility

Authors:

Matthias Flittner,

Stefan Geißler,

Martina ZitterbartAuthors Info & Claims

Reproducibility '17: Proceedings of the Reproducibility Workshop

Pages 14 - 16

https://doi.org/10.1145/3097766.3097770

Published: 11 August 2017 Publication History

Abstract

Reproducing research results, as it is required for peer review, can be a time-consuming and difficult task. In this work, we propose three approaches to improve the way of how research results can be substantiated and discuss their applicability. Our proposals are based on a brief study on evaluation methods (for SDN research) and insights from a comprehensive discussion on reproducibility.

Supplementary Material

WEBM File (tamingthecomplexityofartifactreproducibility.webm)

Download
130.25 MB

References

[1]

C. Collberg and T. A. Proebsting, "Repeatability in computer systems research," Commun. ACM, vol. 59, no. 3, pp. 62--69, Feb. 2016.

Digital Library

[2]

S. Krishnamurthi and J. Vitek, "The real software crisis: Repeatability as a core value," Commun. ACM, vol. 58, no. 3, pp. 34--36, Feb. 2015.

Digital Library

[3]

D. Delling, C. Demetrescu, D. S. Johnson, and J. Vitek, "Rethinking Experimental Methods in Computing," Dagstuhl Reports, vol. 6, no. 3, pp. 24--43, 2016.

[4]

B. A. Nosek, "An open, large-scale, collaborative effort to estimate the reproducibility of psychological science," Perspectives on Psychological Science, vol. 7, no. 6, pp. 657--660, 2012.

[5]

O. Bonaventure, "The january 2017 issue," SIGCOMM Comput. Commun. Rev., vol. 47, no. 1, pp. 1--3, Jan. 2017.

Digital Library

[6]

Result and Artifact Review and Badging. https://www.acm.org/publications/policies/artifact-review-badging.

[7]

Experimental Evaluation of Software and Systems in Computer Science. http://evaluate.inf.usi.ch/artifacts.

[8]

Artifact Evaluation for Software Conferences. http://www.artifact-eval.org/.

[9]

S. Krishnamurthi, "Artifact evaluation for software conferences," SIGPLAN Not., vol. 48, no. 4S, pp. 17--21, Jul. 2013.

Digital Library

[10]

B. R. Childers, G. Fursin, S. Krishnamurthi, and A. Zeller, "Artifact Evaluation for Publications," Dagstuhl Reports, vol. 5, no. 11, pp. 29--35, 2016.

[11]

Secure Networking for a Data Center Cloud in Europe. http://www.sendate.eu/sendate-planets/.

[12]

J. Vitek and T. Kalibera, "R3: Repeatability, reproducibility and rigor," SIGPLAN Not., vol. 47, no. 4a, pp. 30--36, Mar. 2012.

Digital Library

[13]

Collective Knowledge Technology. http://cknowledge.org/.

[14]

Pantheon of Congestion Control. https://github.com/StanfordLPNG/pantheon.

[15]

N. Handigol, B. Heller, V. Jeyakumar, B. Lantz, and N. McKeown, "Reproducible network experiments using container-based emulation," in Proceedings of the 8th International Conference on Emerging Networking Experiments and Technologies, ser. CoNEXT '12, 2012, pp. 253--264.

Digital Library

[16]

recomputation -- If we can compute your experiment now, anyone can recompute it 20 years from now. http://recomputation.org/.

[17]

M. A. Edwards and S. Roy, "Academic research in the 21st century: Maintaining scientific integrity in a climate of perverse incentives and hypercompetition," Environmental Engineering Science, vol. 34, no. 1, pp. 51--61, 2017.

[18]

L. Yan and N. McKeown, "Learning networking by reproducing research results," SIGCOMM Comput. Commun. Rev., vol. 47, no. 2, pp. 19--26, May 2017.

Digital Library

[19]

Research in Practice: Towards a Systematic Evaluation Environment for Software-Defined-Network Applications. https://telematics.tm.kit.edu/english/projects_3553.php.

[20]

Project: Software-based Networks. https://telematics.tm.kit.edu/english/ss2017_ppsdn.php.

Cited By

Raghupathi WRaghupathi VRen J(2022)Reproducibility in Computing Research: An Empirical StudyIEEE Access10.1109/ACCESS.2022.315867510(29207-29223)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3158675
Timperley CHerckis LLe Goues CHilton M(2021)Understanding and improving artifact sharing in software engineering researchEmpirical Software Engineering10.1007/s10664-021-09973-526:4Online publication date: 11-May-2021
https://doi.org/10.1007/s10664-021-09973-5
Saucez DIannone L(2018)Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility WorkshopACM SIGCOMM Computer Communication Review10.1145/3211852.321186348:1(70-74)Online publication date: 27-Apr-2018
https://dl.acm.org/doi/10.1145/3211852.3211863

Index Terms

Taming the Complexity of Artifact Reproducibility
1. Networks
  1. Network performance evaluation
  2. Network services
    1. Programmable networks

Recommendations

Towards reproducibility in recommender-systems research

Numerous recommendation approaches are in use today. However, comparing their effectiveness is a challenging task because evaluation results are rarely reproducible. In this article, we examine the challenge of reproducibility in recommender-system ...
An in-depth investigation on the behavior of measures to quantify reproducibility
Abstract
Science is facing a so-called reproducibility crisis, where researchers struggle to repeat experiments and to get the same or comparable results. This represents a fundamental problem in any scientific discipline because ...
Highlights
- Critical analysis of Information Retrieval measures to quantify reproducibility.
Workshop on reproducibility and replication in recommender systems evaluation: RepSys
RecSys '13: Proceedings of the 7th ACM conference on Recommender systems

Experiment replication and reproduction are key requirements for empirical research methodology, and an important open issue in the field of Recommender Systems. When an experiment is repeated by a different researcher and exactly the same result is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Reproducibility '17: Proceedings of the Reproducibility Workshop

August 2017

31 pages

ISBN:9781450350600

DOI:10.1145/3097766

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCOMM: ACM Special Interest Group on Data Communication

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper
Research
Refereed limited

Funding Sources

CELTIC EUREKA
Bundesministerium für Bildung und Forschung

Conference

SIGCOMM '17

Sponsor:

SIGCOMM

SIGCOMM '17: ACM SIGCOMM 2017 Conference

August 25, 2017

CA, Los Angeles, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
337
Total Downloads

Downloads (Last 12 months)54
Downloads (Last 6 weeks)10

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Raghupathi WRaghupathi VRen J(2022)Reproducibility in Computing Research: An Empirical StudyIEEE Access10.1109/ACCESS.2022.315867510(29207-29223)Online publication date: 2022
https://doi.org/10.1109/ACCESS.2022.3158675
Timperley CHerckis LLe Goues CHilton M(2021)Understanding and improving artifact sharing in software engineering researchEmpirical Software Engineering10.1007/s10664-021-09973-526:4Online publication date: 11-May-2021
https://doi.org/10.1007/s10664-021-09973-5
Saucez DIannone L(2018)Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility WorkshopACM SIGCOMM Computer Communication Review10.1145/3211852.321186348:1(70-74)Online publication date: 27-Apr-2018
https://dl.acm.org/doi/10.1145/3211852.3211863

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten