skip to main content
short-paper

Evaluating the artifacts of SIGCOMM papers

Published: 21 May 2019 Publication History

Abstract

A growing fraction of the papers published by CCR and at SIGCOMM-sponsored conferences include artifacts such as software or datasets. Besides CCR, these artifacts were rarely evaluated. During the last months of 2018, we organised two different Artifacts Evaluation Committees to which authors could submit the artifacts of their papers for evaluation. The first one evaluated the papers accepted by Conext’18 shortly after the TPC decision. It assigned ACM reproducibility badges to 12 different papers. The second one evaluated papers accepted by CCR and any SIGCOMM-sponsored conference. 28 papers received ACM reproducibility badges. We report on the results of a short survey among artifacts authors and reviewers and provide some suggestions for future artifacts evaluations.

References

[1]
<scp>Aubry, F., Vissicchio, S., Bonaventure, O., and Deville, Y.</scp> Robustly disjoint paths with segment routing. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 204–216.
[2]
<scp>Bano, S., Richter, P., Javed, M., Sundaresan, S., Durumeric, Z., Murdoch, S. J., Mortier, R., and Paxson, V.</scp> Scanning the internet for liveness. SIGCOMM Comput. Commun. Rev. 48, 2 (May 2018), 2–9.
[3]
<scp>Barth, L., Hagenmeyer, V., Ludwig, N., and Wagner, D.</scp> How much demand side flexibility do we need?: Analyzing where to exploit flexibility in industrial processes. In Proceedings of the Ninth International Conference on Future Energy Systems (New York, NY, USA, 2018), e-Energy '18, ACM, pp. 43–62.
[4]
<scp>Basat, R. B., Einziger, G., Feibish, S. L., Moraney, J., and Raz, D.</scp> Network-wide routing-oblivious heavy hitters. In Proceedings of the 2018 Symposium on Architectures for Networking and Communications Systems (New York, NY, USA, 2018), ANCS '18, ACM, pp. 66–73.
[5]
<scp>Basat, R. B., Einziger, G., Keslassy, I., Orda, A., Vargaftik, S., and Waisbard, E.</scp> Memento: Making sliding windows efficient for heavy hitters. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 254–266.
[6]
<scp>Beysens, J., Galisteo, A., Wang, Q., Juara, D., Giustiniano, D., and Pollin, S.</scp> Densevlc: A cell-free massive mimo system with distributed leds. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 320–332.
[7]
<scp>Bischof, S., Trittenbach, H., Vollmer, M., Werle, D., Blank, T., and Böhm, K.</scp> Hipe: An energy-status-data set from industrial production. In Proceedings of the Ninth International Conference on Future Energy Systems (New York, NY, USA, 2018), e-Energy '18, ACM, pp. 599–603.
[8]
<scp>Cao, L., Fahmy, S., Sharma, P., and Zhe, S.</scp> Data-driven resource flexing for network functions visualization. In Proceedings of the 2018 Symposium on Architectures for Networking and Communications Systems (New York, NY, USA, 2018), ANCS '18, ACM, pp. 111–124.
[9]
<scp>Deng, H., Peng, C., Fida, A., Meng, J., and Hu, Y. C.</scp> Mobility support in cellular networks: A measurement study on its configurations and implications. In Proceedings of the Internet Measurement Conference 2018 (New York, NY, USA, 2018), IMC '18, ACM, pp. 147–160.
[10]
<scp>Dhamdhere, A., Clark, D. D., Gamero-Garrido, A., Luckie, M., Mok, R. K. P., Akiwate, G., Gogia, K., Bajpai, V., Snoeren, A. C., and Claffy, K.</scp> Inferring persistent interdomain congestion. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 1–15.
[11]
<scp>Gvozdiev, N., Vissicchio, S., Karp, B., and Handley, M.</scp> On low-latency-capable topologies, and their impact on the design of intra-domain routing. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 88–102.
[12]
<scp>Heorhiadi, V., Chandrasekaran, S., Reiter, M. K., and Sekar, V.</scp> Intent-driven composition of resource-management sdn applications. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 86–97.
[13]
<scp>Høiland-Jørgensen, T., Brouer, J. D., Borkmann, D., Fastabend, J., Herbert, T., Ahern, D., and Miller, D.</scp> The express data path: Fast programmable packet processing in the operating system kernel. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 54–66.
[14]
<scp>Huang, Q., Lee, P. P. C., and Bao, Y.</scp> Sketchlearn: Relieving user burdens in approximate measurement with automated statistical inference. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 576–590.
[15]
<scp>Jensen, J. S., Krøgh, T. B., Madsen, J. S., Schmid, S., Srba, J., and Thorgersen, M. T.</scp> P-rex: Fast verification of mpls networks with multiple link failures. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 217–227.
[16]
<scp>Kulkarni, S. G., Liu, G., Ramakrishnan, K., Arumaithurai, M., Wood, T., and Fu, X.</scp> Reinforce: Achieving efficient failure resiliency for network function virtualization based services. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 41–53.
[17]
<scp>Kumar, P., Yu, C., Yuan, Y., Foster, N., Kleinberg, R., and Soulé, R.</scp> Yates: Rapid prototyping for traffic engineering systems. In Proceedings of the Symposium on SDN Research (New York, NY, USA, 2018), SOSR '18, ACM, pp. 11:1–11:7.
[18]
<scp>Mammen, P. M., Kumar, H., Ramamritham, K., and Rashid, H.</scp> Want to reduce energy consumption, whom should we call? In Proceedings of the Ninth International Conference on Future Energy Systems (New York, NY, USA, 2018), e-Energy '18, ACM, pp. 12–20.
[19]
<scp>Mani, A., Wilson-Brown, T., Jansen, R., Johnson, A., and Sherr, M.</scp> Understanding tor usage with privacy-preserving measurement. In Proceedings of the Internet Measurement Conference 2018 (New York, NY, USA, 2018), IMC '18, ACM, pp. 175–187.
[20]
<scp>Marcos, P., Chiesa, M., Müller, L., Kathiravelu, P., Dietzel, C., Canini, M., and Barcellos, M.</scp> Dynam-ix: A dynamic interconnection exchange. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 228–240.
[21]
<scp>Melis, L., Pyrgelis, A., and De Cristofaro, E.</scp> On collaborative predictive blacklisting. SIGCOMM Comput. Commun. Rev. 48, 5 (Jan. 2019), 9–20.
[22]
<scp>Montazeri, B., Li, Y., Alizadeh, M., and Ousterhout, J.</scp> Homa: A receiver-driven low-latency transport protocol using network priorities. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 221–235.
[23]
<scp>Narayan, A., Cangialosi, F., Raghavan, D., Goyal, P., Narayana, S., Mittal, R., Alizadeh, M., and Balakrishnan, H.</scp> Restructuring endpoint congestion control. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 30–43.
[24]
<scp>Neugebauer, R., Antichi, G., Zazo, J. F., Audzevich, Y., López-Buedo, S., and Moore, A. W.</scp> Understanding pcie performance for end host networking. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 327–341.
[25]
<scp>Neves, M., Freire, L., Schaeffer-Filho, A., and Barcellos, M.</scp> Verification of p4 programs in feasible time using assertions. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 73–85.
[26]
<scp>Niu, K., Zhang, F., Xiong, J., Li, X., Yi, E., and Zhang, D.</scp> Boosting fine-grained activity sensing by embracing wireless multipath effects. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 139–151.
[27]
<scp>Pedrosa, L., Iyer, R., Zaostrovnykh, A., Fietz, J., and Argyraki, K.</scp> Automated synthesis of adversarial workloads for network functions. In Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication (New York, NY, USA, 2018), SIGCOMM '18, ACM, pp. 372–385.
[28]
<scp>Pirelli, S., Zaostrovnykh, A., and Candea, G.</scp> A formally verified nat stack. SIGCOMM Comput. Commun. Rev. 48, 5 (Jan. 2019), 77–83.
[29]
<scp>Razaghpanah, A., Niaki, A. A., Vallina-Rodriguez, N., Sundaresan, S., Amann, J., and Gill, P.</scp> Studying tls usage in android apps. In Proceedings of the Applied Networking Research Workshop (New York, NY, USA, 2018), ANRW '18, ACM, pp. 5–5.
[30]
<scp>Reuter, A., Bush, R., Cunha, I., Katz-Bassett, E., Schmidt, T. C., and Wählisch, M.</scp> Towards a rigorous methodology for measuring adoption of rpki route validation and filtering. SIGCOMM Comput. Commun. Rev. 48, 1 (Apr. 2018), 19–27.
[31]
<scp>Scheitle, Q., Chung, T., Hiller, J., Gasser, O., Naab, J., van Rijswijk-Deij, R., Hohlfeld, O., Holz, R., Choffnes, D., Mislove, A., and Carle, G.</scp> A first look at certification authority authorization (caa). SIGCOMM Comput. Commun. Rev. 48, 2 (May 2018), 10–23.
[32]
<scp>Scheitle, Q., Gasser, O., Nolte, T., Amann, J., Brent, L., Carle, G., Holz, R., Schmidt, T. C., and Wählisch, M.</scp> The rise of certificate transparency and its implications on the internet ecosystem. In Proceedings of the Internet Measurement Conference 2018 (New York, NY, USA, 2018), IMC '18, ACM, pp. 343–349.
[33]
<scp>Scheitle, Q., Hohlfeld, O., Gamba, J., Jelten, J., Zimmermann, T., Strowes, S. D., and Vallina-Rodriguez, N.</scp> A long way to the top: Significance, structure, and stability of internet top lists. In Proceedings of the Internet Measurement Conference 2018 (New York, NY, USA, 2018), IMC '18, ACM, pp. 478–493.
[34]
<scp>Scouarnec, N. L.</scp> Cuckoo++ hash tables: High-performance hash tables for networking applications. In Proceedings of the 2018 Symposium on Architectures for Networking and Communications Systems (New York, NY, USA, 2018), ANCS '18, ACM, pp. 41–54.
[35]
<scp>Vermeulen, K., Strowes, S. D., Fourmaux, O., and Friedman, T.</scp> Multilevel mda-lite paris traceroute. In Proceedings of the Internet Measurement Conference 2018 (New York, NY, USA, 2018), IMC '18, ACM, pp. 29–42.
[36]
<scp>Xhonneux, M., Duchene, F., and Bonaventure, O.</scp> Leveraging ebpf for programmable network functions with ipv6 segment routing. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 67–72.
[37]
<scp>Zannettou, S., Caulfield, T., Blackburn, J., De Cristofaro, E., Sirivianos, M., Stringhini, G., and Suarez-Tangil, G.</scp> On the origins of memes by means of fringe web communities. In Proceedings of the Internet Measurement Conference 2018 (New York, NY, USA, 2018), IMC '18, ACM, pp. 188–202.
[38]
<scp>Zheng, P., Benson, T., and Hu, C.</scp> P4visor: Lightweight virtualization and composition primitives for building and testing modular programs. In Proceedings of the 14th International Conference on Emerging Networking EXperiments and Technologies (New York, NY, USA, 2018), CoNEXT '18, ACM, pp. 98–111.
[39]
<scp>Zhou, Y., Alipourfard, O., Yu, M., and Yang, T.</scp> Accelerating network measurement in software. SIGCOMM Comput. Commun. Rev. 48, 3 (Sept. 2018), 2–12.

Cited By

View all
  • (2024)The Pos Experiment Controller: Reproducible & Portable Network Experiments2024 19th Wireless On-Demand Network Systems and Services Conference (WONS)10.23919/WONS60642.2024.10449532(85-92)Online publication date: 29-Jan-2024
  • (2024)Research artifacts in software engineering publications: Status and trendsJournal of Systems and Software10.1016/j.jss.2024.112032213(112032)Online publication date: Jul-2024
  • (2024)Deep variability modeling to enhance reproducibility of database performance testingCluster Computing10.1007/s10586-024-04533-027:8(11683-11708)Online publication date: 1-Nov-2024
  • Show More Cited By

Index Terms

  1. Evaluating the artifacts of SIGCOMM papers

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM SIGCOMM Computer Communication Review
    ACM SIGCOMM Computer Communication Review  Volume 49, Issue 2
    April 2019
    47 pages
    ISSN:0146-4833
    DOI:10.1145/3336937
    Issue’s Table of Contents

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 21 May 2019
    Published in SIGCOMM-CCR Volume 49, Issue 2

    Check for updates

    Author Tags

    1. Artifacts
    2. Reproducibility

    Qualifiers

    • Short-paper

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)28
    • Downloads (Last 6 weeks)4
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)The Pos Experiment Controller: Reproducible & Portable Network Experiments2024 19th Wireless On-Demand Network Systems and Services Conference (WONS)10.23919/WONS60642.2024.10449532(85-92)Online publication date: 29-Jan-2024
    • (2024)Research artifacts in software engineering publications: Status and trendsJournal of Systems and Software10.1016/j.jss.2024.112032213(112032)Online publication date: Jul-2024
    • (2024)Deep variability modeling to enhance reproducibility of database performance testingCluster Computing10.1007/s10586-024-04533-027:8(11683-11708)Online publication date: 1-Nov-2024
    • (2023)Experimental Research Reproducibility and Experiment Workflow Management2023 15th International Conference on COMmunication Systems & NETworkS (COMSNETS)10.1109/COMSNETS56262.2023.10041378(835-840)Online publication date: 3-Jan-2023
    • (2022)Research artifacts and citations in computer systems papersPeerJ Computer Science10.7717/peerj-cs.8878(e887)Online publication date: 7-Feb-2022
    • (2022)On the Conciliation of Traditional and Computer-Assisted ProofsPhilosophy of Computing10.1007/978-3-030-75267-5_3(73-112)Online publication date: 5-May-2022
    • (2021)The pos frameworkProceedings of the 17th International Conference on emerging Networking EXperiments and Technologies10.1145/3485983.3494841(259-266)Online publication date: 2-Dec-2021
    • (2021)BibliographyComputing in Communication Networks10.1016/B978-0-12-820488-7.00045-1(457-471)Online publication date: 2021
    • (2021)Mininet: an instant virtual network on your computerComputing in Communication Networks10.1016/B978-0-12-820488-7.00025-6(219-230)Online publication date: 2021
    • (2020)Thoughts about Artifact BadgingACM SIGCOMM Computer Communication Review10.1145/3402413.340242250:2(60-63)Online publication date: 23-May-2020
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media