An algorithm for automatic assignment of reviewers to papers

Abstract

The assignment of reviewers to papers is one of the most important and challenging tasks in organizing scientific conferences and a peer review process in general. It is a typical example of an optimization task where limited resources (reviewers) should be assigned to a number of consumers (papers), so that every paper should be evaluated by highly competent, in its subject domain, reviewers while maintaining a workload balancing of the reviewers. This article suggests a heuristic algorithm for automatic assignment of reviewers to papers that achieves accuracy of about 98–99% in comparison to the maximum-weighted matching (the most accurate) algorithms, but has better time complexity of Θ(n²). The algorithm provides an uniform distribution of papers to reviewers (i.e. all reviewers evaluate roughly the same number of papers); guarantees that if there is at least one reviewer competent to evaluate a paper, then the paper will have a reviewer assigned to it; and allows iterative and interactive execution that could further increase accuracy and enables subsequent reassignments. Both accuracy and time complexity are experimentally confirmed by performing a large number of experiments and proper statistical analyses. Although it is initially designed to assign reviewers to papers, the algorithm is universal and could be successfully implemented in other subject domains, where assignment or matching is necessary. For example: assigning resources to consumers, tasks to persons, matching men and women on dating web sites, grouping documents in digital libraries and others.

Appendices

Appendix 1: An example

Think of a hypothetical conference. Let’s assume there are 5 submitted papers and 5 registered reviewers. Each paper should be evaluated by 2 reviewers, so every reviewer should evaluate exactly (5 * 2)/5 = 2.0 papers.

Let the similarity matrix is as follows. Rows represent papers and columns represent reviewers.

At the beginning the algorithm sorts each row by the similarity factor in descending order. As a result, the first column suggests the most competent reviewer to every single paper.

The first column suggests that the reviewer r₁ should be assigned to 4 papers (p₁, p₂, p₄ and p₅). However, the maximum allowed number of papers per reviewer is 2, i.e. nobody should review more than two papers. So the algorithm has to decide which 2 of these 4 papers to assign to r₁. At a glance it seems logical that r₁ should be assigned to p₁ and p₂, as they have the highest similarity factors with him/her. On the other hand, there are fewer reviewers competent to evaluate p₄ and p₅ so they should be processed with priority. If r₁ has to be assigned just to p₄ or p₅ which one is more suitable? One may say p₄ as it has higher similarity factor. However, the second-suggested reviewer of p₄ is almost as competent as r₁, while the second-suggested reviewer of p₅ is much less competent in it than r₁ is. In this case it is better to assign r₁ to p₅ rather than to p₄. If r₁ is assigned to p₄, then p₅ will be evaluated by less competent reviewers only, a situation that is highly undesirable. So when deciding which papers to assign to a specific reviewer, the algorithm should take into account both the number of competent reviewers for each paper as well as the rate of decrease in the competence of the next-suggested reviewers for those papers. To automate the process the algorithm modifies the similarity factors from the first column by adding two corrections—C₁ and C₂. They are calculated by formulas 3 and 4. C₁ takes into account the number of non-zero similarity factors with p_i (i.e. the number of reviewers competent to evaluate p_i), while C₂ depends on the rate of decrease in the competence of the next-suggested reviewers for p_i.

The specific values of C₁ и C₂ are as follows:

C1(p1, r1) = 0.0625	C2(p1, r1) = 2 * 0.05 = 0.1
C1(p2, r1) = 0.0625	C2(p2, r1) = 2 * 0.09 = 0.18
C1(p3, r5) = 0.0625	C2(p3, r5) = 2 * 0.03 = 0.06
C1(p4, r1) = 0.25	C2(p4, r1) = 2 * 0.03 = 0.06
C1(p5, r1) = 0.25	C2(p5, r1) = 2 * 0.17 = 0.34

To preserve the real weight of matching, similarity factors should be modified in an auxiliary data structure (an ordinary array) rather than the matrix itself. Here is the first column stored in a single-dimension array.

$$ \left[ {\begin{array}{*{20}c} p_{1} & p_{2} & p_{3} & p_{4} & p_{5} \\ r_{1} = > 0.60 & r_{1} = > 0.89 &r_{5} = > 0.60 &r_{1} = > 0.53 &r_{1} = > 0.50 \end{array} } \right] $$

After adding C₁ и C₂ the first column of the matrix will look like:

$$ \left[ {\begin{array}{*{20}c} p_{1} & p_{2} & p_{3} & p_{4} & p_{5} \\ r_{1} = > 0.76 & r_{1} = > 1.13 & r_{5} = > 0.72 & r_{1} = > 0.84 & r_{1} = > 1.09 \end{array} } \right] $$

As the number of papers per reviewer is 2 then r₁ is assigned to those 2 papers which have the highest similarity factors with him/her after modification. These are p₂ and p₅.

Rows corresponding to p₁ and p₄ in the similarity matrix are shifted one position to the left so that the next-competent reviewers are suggested to these papers. As reviewer r₁ has already got the maximum allowed number of papers to review, he/she is considered to be busy and no more papers should be assigned to him/her in future. Thus all similarity factors, outside the first column, between r₁ and all papers are deleted from the matrix. Deletion guarantees that he/she will not be assign to any more papers.

After shifting p₁ and p₄ one position left and deleting all occurrences of r₁ outside the first column of the matrix, it will look like:

Now r₁ is suggested not to 4 but just to 2 papers. However, after all operations performed above, r₅ is now suggested to 3 papers. To decide which 2 of these 3 papers to assign to r₅, the algorithm again modifies the similarity factors taken from the newly-formed first column of the matrix. As in the previous step, this is done by using formulas 3 and 4. After modification the first column will look like:

$$ \left[ {\begin{array}{*{20}c} p_{1} & p_{2} & p_{3} & p_{4} & p_{5} \\ r_{5} = > 0.70 & r_{1} = > 1.13 &r_{5} = > 0.77 &r_{5} = > 1.70 & r_{1} = > 1.09 \end{array} } \right] $$

It should be assigned to those two papers which have the highest similarity factors with him/her. These are p₃ and p₄. The row corresponding to p₁ is shifted one position to the left again, so that the next-competent reviewer is suggested to that paper. As r₅ already has 2 papers to review, all similarity factors outside the first column that are associated with him/her are deleted from the matrix.

Therefore, the matrix looks like:

As seen in the first column, no reviewer is suggested to evaluate more than 2 papers. So it is now possible to assign all reviewers from the first column directly to the papers which they are suggested to. If the last matrix is compared to the initial one, it could be spotted that 3 of 5 papers (p₂, p₃ and p₅) are assigned to their most competent reviewers. One (p₄) has got its second-competent reviewer and another one (p₁) its third-competent reviewer. However, the levels of competence of these reviewers in respect to p₄ and p₁ are very close to the levels of the most competent reviewers for these papers. r₅ is assigned to p₄ with a similarity factor of 0.50 while the most competent reviewer for p₄ has a similarity factor of 0.53.

“Appendix 2”: Detailed pseudo code

The detailed pseudo code here could be directly translated in any high-level imperative programming language as each operation in the pseudo corresponds to an operator or a built-in function in the chosen programming language. Here is the meaning of the complex data structures (mostly arrays) used within the code:

• SM[i,j]—similarity matrix—bi-dimensional array of arrays, where rows (i) represent papers and columns (j) represent reviewers. Each element contains an associative array of two elements—revUser (the user id of the reviewer who is suggested to evaluate paper i) and weight (the similarity factor between paper i and reviewer revUser).

• papersOfReviewer[]—an associative array, whose keys corresponds to the usernames (revUser) of the reviewers who appear in the first column of the similarity matrix; and values containing arrays of two elements—id of the paper, being suggested to this reviewer; and the similarity factor between the paper and the reviewer.

For example:

• rowsToShift[]—an array containing the row ids (these are actually the paper ids) that should be shifted one position to the left, so that the next-competent reviewer is suggested to this paper.

• signifficantSF[paperId]—an array holding the number of significant, non-zero, similarity factors for every paper, identified by its paperId.

• reviewersToRemove[]—an array holding the identifiers (revUser) of the reviewers who are already busy (i.e. have enough papers to review) and should be reviewed from the similarity matrix (except its first row) on the next pass through the outermost do-while cycle, so that they are not assigned to any more papers.

• busyReviewers[]—an array holding the identifiers of the reviewers who are already busy (i.e. have enough papers to review). This is similar to reviewersToRemove with one major difference—busyReviewers keeps identifiers all the time, while reviewersToRemove is cleared on each pass after deleting the respective reviewers (similarity factors) from the similarity matrix.

• maxPapersToAssign[j]—an array holding the maximum number of papers that could be assigned to every reviewer j, identified by its revUser.