skip to main content
research-article

Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility Workshop

Published: 27 April 2018 Publication History

Abstract

Ensuring the reproducibility of results is an essential part of experimental sciences, including computer networking. Unfortunately, as highlighted recently, a large portion of research results are hardly, if not at all, reproducible, raising reasonable lack of conviction on the research carried out around the world.
Recent years have shown an increasing awareness about reproducibility of results as an essential part of research carried out by members of the ACM SIGCOMM community. To address this important issue, ACM has introduced a new policy on results and artifacts review and badging. The policy defines the terminology to be used to assess results and artifacts but does not specify the review process or how to make research reproducible.
During SIGCOMM'17 a side workshop has been organized with the specific purpose to tackle this issue. The objective being to trigger discussion and activity in order to craft recommendations on how to introduce incentives for authors to share their artifacts, and the details on how to use them, as well as defining the process to be used.
This editorial overviews the workshop activity and summarizes the main discussions and outcomes.

References

[1]
A Catalog of Research Artifacts for Computer Science. http://www.findresearch.org. Accessed: 2017-09-11.
[2]
Artifact Review and Badging. https://www.acm.org/publications/policies/artifact-review-badging. Accessed: 2017-09-11.
[3]
ISO 26324:2012, Information and documentation – Digital object identifier system. https://www.iso.org/standard/43506.html. Accessed: 2017-09-13.
[4]
Bajpai, V., Kühlewind, M., Ott, J., Schönwälder, J., Sperotto, A., and Trammell, B. Challenges with reproducibility. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 1–4.
[5]
Canini, M., and Crowcroft, J. Learning reproducibility with a yearly networking contest. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 9–13.
[6]
Ferreira, D. C., Vázquez, F. I., Vormayr, G., Bachl, M., and Zseby, T. A meta-analysis approach for feature selection in network traffic research. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 17–20.
[7]
Flittner, M., Bauer, R., Rizk, A., Geißler, S., Zinner, T., and Zitterbart, M. Taming the complexity of artifact reproducibility. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 14–16.
[8]
Kumar, S., Cifuentes, D., Gollakota, S., and Katabi, D. Bringing cross-layer mimo to today's wireless lans. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (New York, NY, USA, 2013), SIGCOMM '13, ACM, pp. 387–398.
[9]
Mahfoudi, M. N., Turletti, T., Parmentelat, T., and Dabbous, W. Lessons learned while trying to reproduce the openrf experiment. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 21–23.
[10]
Nussbaum, L. Testbeds support for reproducible research. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 24–26.
[11]
Scheitle, Q., Wählisch, M., Gasser, O., Schmidt, T. C., and Carle, G. Towards an ecosystem for reproducible research in computer networking. In Proceedings of the Reproducibility Workshop (New York, NY, USA, 2017), Reproducibility '17, ACM, pp. 5–8.
[12]
Yan, L., and McKeown, N. Learning networking by reproducing research results. SIGCOMM Comput. Commun. Rev. 47, 2 (May 2017), 19–26.

Cited By

View all
  • (2023)Analyzing Cyber Security Research Practices through a Meta-Research FrameworkProceedings of the 16th Cyber Security Experimentation and Test Workshop10.1145/3607505.3607523(64-74)Online publication date: 7-Aug-2023
  • (2022)Challenges and recent advances in the design of real-time wireless Cyber-Physical SystemsBenchCouncil Transactions on Benchmarks, Standards and Evaluations10.1016/j.tbench.2022.1000362:1(100036)Online publication date: Mar-2022
  • (2020)Validating the Sharing Behavior and Latency Characteristics of the L4S ArchitectureACM SIGCOMM Computer Communication Review10.1145/3402413.340241950:2(37-44)Online publication date: 23-May-2020
  • Show More Cited By

Index Terms

  1. Thoughts and Recommendations from the ACM SIGCOMM 2017 Reproducibility Workshop

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM SIGCOMM Computer Communication Review
        ACM SIGCOMM Computer Communication Review  Volume 48, Issue 1
        January 2018
        80 pages
        ISSN:0146-4833
        DOI:10.1145/3211852
        Issue’s Table of Contents

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 27 April 2018
        Published in SIGCOMM-CCR Volume 48, Issue 1

        Check for updates

        Author Tags

        1. Artifacts
        2. Reproducibility
        3. SIGCOMM

        Qualifiers

        • Research-article

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)15
        • Downloads (Last 6 weeks)2
        Reflects downloads up to 17 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2023)Analyzing Cyber Security Research Practices through a Meta-Research FrameworkProceedings of the 16th Cyber Security Experimentation and Test Workshop10.1145/3607505.3607523(64-74)Online publication date: 7-Aug-2023
        • (2022)Challenges and recent advances in the design of real-time wireless Cyber-Physical SystemsBenchCouncil Transactions on Benchmarks, Standards and Evaluations10.1016/j.tbench.2022.1000362:1(100036)Online publication date: Mar-2022
        • (2020)Validating the Sharing Behavior and Latency Characteristics of the L4S ArchitectureACM SIGCOMM Computer Communication Review10.1145/3402413.340241950:2(37-44)Online publication date: 23-May-2020
        • (2018)A Long Way to the TopProceedings of the Internet Measurement Conference 201810.1145/3278532.3278574(478-493)Online publication date: 31-Oct-2018

        View Options

        Login options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media