Quirin Scheitle
Matthias Wählisch
Oliver Gasser
Thomas C. Schmidt
ABSTRACT
Reproducibility is key to rigorous scientific progress. However, many publications in the computer networks community lack support for reproducibility. In this paper, we argue that the lack is mainly rooted in the additional effort that authors need to spend, without expecting sufficient benefits. Based on our experience in both authoring reproducible research and reproducing publications, we propose an ecosystem that incentivizes authors and reproducers to invest additional effort. This ecosystem consists of various building blocks, which can be combined into venue-specific profiles. A key building block is the Reproducibility Challenge, which we suggest to co-locate with the annual SIGCOMM conference to leverage reproducibility research in practice.
Supplemental Material
- http://www.artifact-eval.org/, 2/2017.Google Scholar
- https://www.cs.ox.ac.uk/conferences/hscc2016/re.html, 3/2017.Google Scholar
- ACM. Result and Artifact Review and Badging. http://acm.org/publications/policies/artifact-review-badging, 1/2017.Google Scholar
- ACM. Workshop on Models, Methods and Tools for Reproducible Network Research. http://conferences.sigcomm.org/sigcomm/2003/workshop/mometools/, 2/2017.Google Scholar
- V. Bajpai, M. Kühlewind, J. Ott, J. Schönwälder, A. Sperotto, and B. Trammel. Challenges with Reproducibility. In Proc. of ACM SIGCOMM Reproducibility Workshop. ACM, 2017. Google ScholarDigital Library
- M. Canini and J. Crowcroft. Learning Reproducibility with a Yearly Networking Contest. In Proc. of ACM SIGCOMM Reproducibility Workshop. ACM, 2017. Google ScholarDigital Library
- M. P. Grosvenor, M. Schwarzkopf, I. Gog, R. N. M. Watson, A. W. Moore, S. Hand, and J. Crowcroft. Queues Don't Matter When You Can JUMP Them! In Proc. of USENIX NSDI, 2015. Google ScholarDigital Library
- M. Wachs, Q. Scheitle, and G. Carle. Push Away Your Privacy: Precise User Tracking Based on TLS Client Certificate Authentication. In Proc. of Network Traffic Measurement and Analysis Conference (TMA), 2017.Google ScholarCross Ref
- L. Yan and N. McKeown. Learning Networking by Reproducing Research Results. ACM SIGCOMM Computer Communication Review, 2017. Google ScholarDigital Library
Index Terms
- Towards an Ecosystem for Reproducible Research in Computer Networking
Recommendations
Reproducible Online Search Experiments
Advances in Information RetrievalAbstractIn the empirical sciences, the evidence is commonly manifested by experimental results. However, very often, these findings are not reproducible, hindering scientific progress. Innovations in the field of information retrieval (IR) are mainly ...
Software in reproducible research: advice and best practice collected from experiences at the collaborations workshop
TRUST '14: Proceedings of the 1st ACM SIGPLAN Workshop on Reproducible Research Methodologies and New Publication Models in Computer EngineeringThe Collaborations Workshop 2014 (CW14) brought together representatives from across the research community to discuss the issues around software's role in reproducible research. In this paper we summarise the themes, practices and ideas raised at the ...
Reproducible Research for Scientific Computing: Tools and Strategies for Changing the Culture
This article considers the obstacles involved in creating reproducible computational research as well as some efforts and approaches to overcome them.
Comments