skip to main content
review-article

Novel tools to streamline the conference review process: experiences from SIGKDD'09

Published: 27 May 2010 Publication History

Abstract

The SIGKDD'09 Research Track received 537 paper submissions, which were reviewed by a Program Committee of 199 members, and a Senior Program Committee of 22 members. We used techniques from artificial intelligence and data mining to streamline and support this complicated process at three crucial stages: bidding by PC members on papers, assigning papers to reviewers, and calibrating scores obtained from the reviews. In this paper we report on the approaches taken, evaluate how well they worked, and describe some further work done after the conference.

References

[1]
G. Smolka and J. Walser. Review Assignment for CP97, 1997. http://www.ps.uni-sb.de/~walser/reviewing.ps.
[2]
M. Berkelaar, K. Eikland and P. Notebaert. lp solve : Open source (Mixed-Integer) Linear Programming system. http://lpsolve.sourceforge.net/.
[3]
T. Minka. Expectation propagation for approximate bayesian inference. In J. S. Breese and D. Koller, editors, Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, pages 362--369. Morgan Kaufmann, 2001.
[4]
T. Minka. Divergence measures and message passing. Technical report, Microsoft Research, 2005.
[5]
T. Minka, J. Winn, J. Guiver, and A. Kannan. Infer. NET 2.3, 2009. Microsoft Research Cambridge. http://research.microsoft.com/infernet.
[6]
G. Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Commun. ACM, 18(11):613--620, 1975.
[7]
K. Spärck-Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28(1):11--21, 1972.
[8]
S. Spiegler. Comparative study of clustering algorithms on textual databases - clustering of curricula vitae into comptency-based groups to support knowledge management. Master's thesis, Technical University Ilmenau, Ilmenau, Germany, February 2007.
[9]
J. Walser. Integer Optimization by Local Search, volume 1637 of Lecture Notes in Computer Science, pages 81--93. Springer, 1999.

Cited By

View all
  • (2024)Matching papers and reviewers at large conferencesArtificial Intelligence10.1016/j.artint.2024.104119331(104119)Online publication date: Jun-2024
  • (2023)Counterfactual evaluation of peer-review assignment policiesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668685(58765-58786)Online publication date: 10-Dec-2023
  • (2023)Community review: a robust and scalable selection system for resource allocation within open science and innovation communitiesF1000Research10.12688/f1000research.125886.211(1440)Online publication date: 18-Apr-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM SIGKDD Explorations Newsletter
ACM SIGKDD Explorations Newsletter  Volume 11, Issue 2
December 2009
128 pages
ISSN:1931-0145
EISSN:1931-0153
DOI:10.1145/1809400
Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2010
Published in SIGKDD Volume 11, Issue 2

Check for updates

Qualifiers

  • Review-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)23
  • Downloads (Last 6 weeks)1
Reflects downloads up to 13 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)Matching papers and reviewers at large conferencesArtificial Intelligence10.1016/j.artint.2024.104119331(104119)Online publication date: Jun-2024
  • (2023)Counterfactual evaluation of peer-review assignment policiesProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3668685(58765-58786)Online publication date: 10-Dec-2023
  • (2023)Community review: a robust and scalable selection system for resource allocation within open science and innovation communitiesF1000Research10.12688/f1000research.125886.211(1440)Online publication date: 18-Apr-2023
  • (2023)Batching of Tasks by Users of Pseudonymous Forums: Anonymity Compromise and ProtectionProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35793357:1(1-41)Online publication date: 2-Mar-2023
  • (2023)A Dataset on Malicious Paper Bidding in Peer ReviewProceedings of the ACM Web Conference 202310.1145/3543507.3583424(3816-3826)Online publication date: 30-Apr-2023
  • (2023)An Algorithm for Peer Reviewer Recommendation Based on Scholarly Activity AssessmentIEEE Access10.1109/ACCESS.2023.326385711(39609-39620)Online publication date: 2023
  • (2023)Recommending Reviewers and ExpertsKnowledge Recommendation Systems with Machine Intelligence Algorithms10.1007/978-3-031-32696-7_3(29-50)Online publication date: 1-Oct-2023
  • (2023)Literature ReviewKnowledge Recommendation Systems with Machine Intelligence Algorithms10.1007/978-3-031-32696-7_2(9-27)Online publication date: 1-Oct-2023
  • (2022)Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment DesignProceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems10.5555/3535850.3536062(1642-1644)Online publication date: 9-May-2022
  • (2022)Challenges, experiments, and computational solutions in peer reviewCommunications of the ACM10.1145/352808665:6(76-87)Online publication date: 20-May-2022
  • Show More Cited By

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media