Maryam Karimzadehgan
ABSTRACT
Review assignment is a common task that many people such as conference organizers, journal editors, and grant administrators would have to do routinely. As a computational problem, it involves matching a set of candidate reviewers with a paper or proposal to be reviewed. A common deficiency of all existing work on solving this problem is that they do not consider the multiple aspects of topics or expertise and all match the entire document to be reviewed with the overall expertise of a reviewer. As a result, if a document contains multiple subtopics, which often happens, existing methods would not attempt to assign reviewers to cover all the subtopics; instead, it is quite possible that all the assigned reviewers would cover the major subtopic quite well, but not covering any other subtopic. In this paper, we study how to model multiple aspects of expertise and assign reviewers so that they together can cover all subtopics in the document well. We propose three general strategies for solving this problem and propose new evaluation measures for this task. We also create a multi-aspect review assignment test set using ACM SIGIR publications. Experiment results on this data set show that the proposed methods are effective for assigning reviewers to cover all topical aspects of a document.
- K. Balog, L. Azzopardi, and M. de Rijke. Formal models for expert finding in enterprise corpora. In Proceedings of ACM SIGIR'06, pages 43--50, 2006. Google Scholar
Digital Library
- C. Basu, H. Hirsh, W. Cohen, and C. Nevill-Manning. Recommending papers by mining the web. Proceedings of IJCAI Workshops on Learning About Users and Machine Learning for Information Filtering, 1999.Google Scholar
- D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. J. Mach. Learning Res., 3:993--1022, 2003. Google ScholarDigital Library
- J. Carbonell and J. Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proceedings of SIGIR 1998, pages 335--336, 1998. Google ScholarDigital Library
- T. Cover and J. Thomas. Elements of Information Theory. John Wiley and Sons, New York, USA, 1991. Google ScholarDigital Library
- S. Dumais and J. Nielsen. Automating the assignments of submitted manuscripts to reviewers. In Proceedings of SIGIR 1992, pages 233--244, 1992. Google ScholarDigital Library
- H. Fang and C. Zhai. Probabilistic models for expert finding. In Proceedings of the 29th European Conference on Information Retrieval, pages 418--430, 2007. Google ScholarDigital Library
- M. A. Hearst. Texttiling: Segmenting text into multi-paragraph subtopic passages. Computational Linguistics, 23(1):33--64, 1997. Google ScholarDigital Library
- S. Hettich and M. J. Pazzani. Mining for proposal reviewers: Lessons learned at the national science foundation. In Proceedings of the 12th ACM SIGKDD, pages 862--871, 2006. Google ScholarDigital Library
- T. Hofmann. Probabilistic latent sematic indexing. In Proceedings of ACM SIGIR'99, pages 50--57, 1999. Google ScholarDigital Library
- K. Jarvelin and J. Kekalainen. Ir evaluation methods for retrieving highly relevant documents. In Proceedings of ACM SIGIR 2000, pages 41--48, 2000. Google ScholarDigital Library
- B. Larson and A. Chinatsu. Fast and effective text mining using linear-time document clustering. In Proceedings of ACM SIGKDD'99, pages 16--22, 1999. Google ScholarDigital Library
- D. Mimno and A. McCallum. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD'07, pages 500--509, 2007. Google ScholarDigital Library
- D. Petkova and W. B. Croft. Hierarchical language models for expert finding in enterprise corpora. In Proceedings of the 18th IEEE on Tools with Artificial Intelligence, pages 599--608, 2006. Google ScholarDigital Library
- M. Rosen-Zvi, T. Griffiths, P. Smyth, and M. Steyvers. The author-topic model for authors and documents. In Proceedings of the 20th conference on Uncertainty in Artificial Intelligence, pages 487--494, 2004. Google ScholarDigital Library
- C. Zhai, W. Cohen, and J. Lafferty. Beyond independent relevance: Methods and evaluation metrics for subtopic retrieval. Proceedings of the 26th annual international ACM SIGIR, pages 10--17, 2003. Google ScholarDigital Library
- C. Zhai and J. Lafferty. Model-based feedback in the language modeling approach to information retrieval. In Proceedings of CIKM'01, pages 403--410, 2001. Google ScholarDigital Library
- C. Zhai and J. Lafferty. A study of smoothing methods for language models applied to ad hoc information retrieval. In Proceedings of ACM Transactions on Information Systems, pages 179--214, 2004. Google ScholarDigital Library
Index Terms
- Multi-aspect expertise matching for review assignment
Recommendations
Expertise modeling for matching papers with reviewers
KDD '07: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data miningAn essential part of an expert-finding task, such as matching reviewers to submitted papers, is the ability to model the expertise of a person based on documents. We evaluate several measures of the association between an author in an existing ...
Constrained multi-aspect expertise matching for committee review assignment
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge managementAutomatic review assignment can significantly improve the productivity of many people such as conference organizers, journal editors and grant administrators. Most previous works have set the problem up as using a paper as a query to independently "...
A robust model for paper reviewer assignment
RecSys '14: Proceedings of the 8th ACM Conference on Recommender systemsAutomatic expert assignment is a common problem encountered in both industry and academia. For example, for conference program chairs and journal editors, in order to collect "good" judgments for a paper, it is necessary for them to assign the paper to ...
Comments