skip to main content
10.1145/2508497.2508501acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Benchmarking domain-specific expert search using workshop program committees

Published: 28 October 2013 Publication History

Abstract

Traditionally, relevance assessments for expert search have been gathered through self-assessment or based on the opinions of co-workers. We introduce three benchmark datasets for expert search that use conference workshops for relevance assessment. Our data sets cover entire research domains as opposed to single institutions. In addition, they provide a larger number of topic-person associations and allow a more objective and fine-grained evaluation of expertise than existing data sets do. We present and discuss baseline results for a language modelling and a topic-centric approach to expert search. We find that the topic-centric approach achieves the best results on domain-specific datasets.

References

[1]
P. Bailey, A. P. de Vries, N. Craswell, and I. Soboroff. Overview of the TREC 2007 Enterprise Track. In Proceedings of the Fourteenth Text REtrieval Conference (TREC), 2007.
[2]
K. Balog, T. Bogers, L. Azzopardi, M. de Rijke, and A. van den Bosch. Broad Expertise Retrieval in Sparse Data Environments. In SIGIR '07: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 551--558, New York, NY, July 2007. ACM.
[3]
R. Berendsen, K. Balog, T. Bogers, A. Van den Bosch, and M. De Rijke. On the Assessment of Expertise Profiles. Journal of the American Society for Information Science, July 2013.
[4]
H. Biswas and M. Hasan. Using Publications and Domain Knowledge to Build Research Profiles: An Application in Automatic Reviewer Assignment. In Proceedings of the 2007 International Conference on Information and Communication Technology (ICICT'07), pages 82--86, 2007.
[5]
G. Bordea, S. Kirrane, P. Buitelaar, and B. O. Pereira. Expertise Mining for Enterprise Content Management. In LREC, pages 3495--3498, 2012.
[6]
C. S. Campbell, P. P. Maglio, A. Cozzi, and B. Dom. Expertise identification using email communications. In CIKM '03: Proceedings of the Twelfth International Conference on Information and Knowledge Management, pages 528--531, New Orleans, LA, 2003.
[7]
H. Deng, I. King, and M. R. Lyu. Formal Models for Expert Finding on DBLP Bibliography Data. In ICDM '08: Proceedings of the Eighth IEEE International Conference on Data Mining, pages 163--172. IEEE, 2008.
[8]
S. T. Dumais and J. Nielsen. Automating the Assignment of Submitted Manuscripts to Reviewers. In SIGIR '92: Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 233--244, New York, NY, USA, 1992. ACM.
[9]
S. Ferilli, N. Di Mauro, T. Basile, F. Esposito, and M. Biba. Automatic Topics Identification for Reviewer Assignment. Advances in Applied Artificial Intelligence, pages 721--730, 2006.
[10]
K. Hofmann, K. Balog, T. Bogers, and M. de Rijke. Contextual Factors for Finding Similar Experts. Journal of the American Society for Information Science, 61(5):994--1014, 2010.
[11]
M. Maybury. Expert finding systems. Technical Report MTR 06B000040, MITRE Corporation, 2006.
[12]
D. Mimno and A. McCallum. Expertise Modeling for Matching Papers with Reviewers. In SIGKDD '07: Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 500--509, 2007.
[13]
M. A. Rodriguez and J. Bollen. An Algorithm to Determine Peer-Reviewers. In '08: Proceedings of the Seventeenth International Conference on Information and Knowledge Management, pages 319--328. ACM, 2008.
[14]
E. Smirnova and K. Balog. A User-oriented Model for Expert Finding. In Proceedings of the 33rd European conference on Advances in information retrieval, ECIR'11, pages 580--592, Berlin, Heidelberg, 2011. Springer-Verlag.
[15]
I. Soboroff, A. P. de Vries, and N. Craswell. Overview of the TREC 2006 Enterprise Track. In In The Fifteenth Text Retrieval Conference (TREC 2006). NIST, 2006.
[16]
M. Stankovic, J. Jovanovic, and P. Laublet. Linked Data Metrics for Flexible Expert Search on the Open Web. In Proceedings of the 8th extended semantic web conference on The semantic web: research and applications - Volume Part I, ESWC'11, pages 108--123, Berlin, Heidelberg, 2011. Springer-Verlag.
[17]
J. Tang, J. Zhang, L. Yao, J. Li, L. Zhang, and Z. Su. ArnetMiner: Extraction and Mining of Academic Social Networks. In KDD '08: Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 990--998, New York, NY, USA, 2008. ACM.
[18]
D. Yarowsky and R. Florian. Taking the Load off the Conference Chairs: Towards a Digital Paper-Routing Assistant. In Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in NLP and Very Large Corpora, pages 220--230, 1999.

Cited By

View all
  • (2024)Reinforcement Learning for Expert Finding from Web Search ResultsAdvances in Knowledge Discovery and Management10.1007/978-3-031-40403-0_6(113-128)Online publication date: 1-Feb-2024
  • (2023)ExpFinder: A hybrid model for expert finding from text-based expertise dataExpert Systems with Applications10.1016/j.eswa.2022.118691211(118691)Online publication date: Jan-2023
  • (2018)Semantic representation and enrichment of information retrieval experimental dataInternational Journal on Digital Libraries10.1007/s00799-016-0172-818:2(145-172)Online publication date: 19-Dec-2018
  • Show More Cited By

Index Terms

  1. Benchmarking domain-specific expert search using workshop program committees

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CompSci '13: Proceedings of the 2013 workshop on Computational scientometrics: theory & applications
    October 2013
    44 pages
    ISBN:9781450324144
    DOI:10.1145/2508497
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 28 October 2013

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. expert finding
    2. expertise search
    3. workshop dataset

    Qualifiers

    • Research-article

    Conference

    CIKM'13
    Sponsor:

    Acceptance Rates

    CompSci '13 Paper Acceptance Rate 6 of 7 submissions, 86%;
    Overall Acceptance Rate 6 of 7 submissions, 86%

    Upcoming Conference

    CIKM '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)Reinforcement Learning for Expert Finding from Web Search ResultsAdvances in Knowledge Discovery and Management10.1007/978-3-031-40403-0_6(113-128)Online publication date: 1-Feb-2024
    • (2023)ExpFinder: A hybrid model for expert finding from text-based expertise dataExpert Systems with Applications10.1016/j.eswa.2022.118691211(118691)Online publication date: Jan-2023
    • (2018)Semantic representation and enrichment of information retrieval experimental dataInternational Journal on Digital Libraries10.1007/s00799-016-0172-818:2(145-172)Online publication date: 19-Dec-2018
    • (2016)Exploiting heterogeneous scientific literature networks to combat ranking biasJournal of the Association for Information Science and Technology10.1002/asi.2346367:7(1679-1702)Online publication date: 1-Jul-2016

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media