Skip to main content
SpringerLink
Log in

How to allocate review tasks for robust ranking

  • Original Article
  • Published:
Acta Informatica Aims and scope Submit manuscript

Abstract

In the process of reviewing and ranking projects by a group of reviewers, the allocation of the subset of projects to each reviewer has major impact on the robustness of the outcome ranking. We address here this problem where each reviewer is assigned, out of the list of all projects, a subset of up to k projects. Each individual reviewer then ranks and compares all pairs of k projects. The k-allocation problem is to determine an allocation of up to k projects to each reviewer, that lie within the expertise set of the reviewer, so that the resulting union of reviewed projects has certain desirable properties. The k-complete problem is a k-allocation with the property that all pairs of projects have been compared by at least one reviewer. A k-complete allocation is desirable as otherwise there may be projects that were not compared by any reviewer, leading to possible adverse properties in the outcome ranking. When a k-complete allocation cannot be achieved, one might settle for other properties. One basic requirement is that each pair of projects is comparable via a ranking path which is a sequence of pairwise rankings of projects implying a comparison of all pairs on the path. A k-allocation with a ranking path between each pair is the connectivity-k-aloc. Since the robustness of relative comparisons deteriorates with increased length of the ranking path, another goal is that between each pair of projects there will be at least one ranking path that has at most two hops or q hops for fixed values of q. An alternative means for increasing the robustness of the ranking is to use a k-allocation with at least p disjoint ranking paths between each pair. We model all these problems as graph problems. We demonstrate that the connectivity-k-aloc problem is polynomially solvable, using matroid intersection; we prove that the k-complete problem is NP-hard unless k = 2; and we provide approximation algorithms for a related optimization problem. All other variants are shown to be NP-complete for all values of k ≥ 2.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ageev A.A., Sviridenko M.I.: Pipage rounding: a new method of constructing algorithms with proven performance guarantee. J. Comb. Optim. 8, 307–328 (2004)

    Article  MATH  MathSciNet  Google Scholar 

  2. Asahiro Y., Iwama K., Tamaki H., Tokuyama T.: Greedily finding a dense subgraph. J. Algorithms 34, 203–221 (2000)

    Article  MATH  MathSciNet  Google Scholar 

  3. Bartholdi J.J., Tovey C.A., Trick M.A.: The computational difficulty of manipulating an election. Soc. Choice Welfare 6, 227–241 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  4. Brans J.P., Vincke P.H.: A preference ranking organization method. Manage. Sci. 31, 647–656 (1985)

    Article  MATH  MathSciNet  Google Scholar 

  5. Brezovec C., Cornuejols G., Glover F.: Two algorithms for weighted matroid intersection. Math. Programming 36, 39–53 (1986)

    Article  MATH  MathSciNet  Google Scholar 

  6. Brezovec C., Cornuejols G., Glover F.: A matroid algorithm and its application to the efficient solution of two optimization problems on graphs. Math. Programming 42, 471–487 (1988)

    Article  MATH  MathSciNet  Google Scholar 

  7. Chekuri, C., Kumar, A.: Maximum coverage problem with group budget constraints and applications. In: Proceedings of APPROX 2004, pp. 72–83 (2004)

  8. Cook W.D., Golany B., Kress M., Penn M., Raviv T.: Optimal allocation of proposals to reviewers to facilitate effective ranking. Manage. Sci. 51, 655–661 (2005)

    Article  Google Scholar 

  9. Dor D., Tarsi M.: Graph decomposition is NP-complete: a complete proof of Holyer’s conjecture. SIAM J. Comput. 26, 1166–1187 (1997)

    Article  MATH  MathSciNet  Google Scholar 

  10. Eswaran K.P., Tarjan R.E.: Augmentation problems. SIAM J. Comput. 5, 653–665 (1976)

    Article  MATH  MathSciNet  Google Scholar 

  11. Feige U., Kortsarz G., Peleg D.: The dense k-subgraph problem. Algorithmica 29, 410–421 (2001)

    Article  MATH  MathSciNet  Google Scholar 

  12. Fernandez E., Olemdo R.: An agent model based on ideas of concordance and discordance for group ranking problems. Decis. Support Syst. 39, 429–443 (2005)

    Article  Google Scholar 

  13. Frederickson G.N., Srinivas M.A.: Algorithms and data structures for an expanded family of matroid intersection problems. SIAM J. Comput. 18, 112–138 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  14. Fuller R., Carlsson C.H.: Fuzzy multiple criteria decision making: recent developments. Fuzzy sets syst. 78, 139–153 (1996)

    Article  MATH  MathSciNet  Google Scholar 

  15. Gabow, H.N., Tarjan, R.E.: Almost optimum speed-ups of algorithms for bipartite matching and related problems. In: Proceedings of STOC 1988, pp. 514–527 (1988)

  16. Gabow H.N., Tarjan R.E.: Faster scaling algorithms for network problems. SIAM J. Comput. 18, 1013–1036 (1989)

    Article  MATH  MathSciNet  Google Scholar 

  17. Garey M.R., Johnson D.S.: Computers and Intractability. W.H. Freeman, New York (1979)

    MATH  Google Scholar 

  18. Hochbaum, D.S., Chandran, B.: Further Below the Flow Decomposition Barrier of Maximum Flow for Bipartite Matching and Maximum Closure. Manuscript, UC Berkeley, April (2004)

  19. Hochbaum D.S., Levin A.: Methodologies for the group rankings decision. Manage. Sci. 52, 1394–1408 (2006)

    Article  Google Scholar 

  20. Holyer I.: The NP-completeness of some edge-partition problems. SIAM J. Comput. 10, 713–717 (1981)

    Article  MATH  MathSciNet  Google Scholar 

  21. Hsu T., Ramachandran V.: On finding smallest augmentation to biconnect a graph. SIAM J. Comput. 22, 889–912 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  22. Keener J.P.: The perron-frobenius theorem and the rating of football teams. SIAM Rev. 35, 80–93 (1993)

    Article  MATH  MathSciNet  Google Scholar 

  23. Kemeny, J.G., Snell, J.L.: Preference ranking: an axiomatic approach. In: Mathematical Models in the Social Sciences, Ginn, Boston, pp. 9–23 (1962)

  24. Park, J., Newman, M.E.J.: A network-based ranking system for US college football. J. Stat. Mech.: Theory Exp. (Oct. 31, 2005). Abstract available at http://www.iop.org/EJ/abstract/1742-5468/2005/10/P10014

  25. Rosenthal A., Goldner A.: Smallest augmentations to biconnect a graph. SIAM J. Comput. 6, 55–66 (1977)

    Article  MATH  MathSciNet  Google Scholar 

  26. Schrijver A.: Combinatorial Optimization Polyhedra and Efficiency. Springer, Berlin (2003)

    MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering and Operations Research and Walter A. Haas School of Business, University of California, Berkeley, USA

    Dorit S. Hochbaum

  2. Faculty of Industrial Engineering and Management, The Technion, 32000, Haifa, Israel

    Asaf Levin

Authors
  1. Dorit S. Hochbaum
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Asaf Levin
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Dorit S. Hochbaum.

Additional information

An extended abstract version of this paper appeared with the title “The k-allocation problem and its variants” in Proceedings of the Fourth Workshop on Approximation and Online Algorithms (WAOA 2006).

Dorit S. Hochbaum—Research supported in part by NSF award No. DMI-0620677 and CBET-0736232.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hochbaum, D.S., Levin, A. How to allocate review tasks for robust ranking. Acta Informatica 47, 325–345 (2010). https://doi.org/10.1007/s00236-010-0120-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00236-010-0120-9

Keywords

Search

Navigation