skip to main content
10.1145/2600428.2609486acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
poster

Analyzing bias in CQA-based expert finding test sets

Published: 03 July 2014 Publication History

Abstract

Data retrieved from community question answering (CQA) sites, such as content and users' assessments of content, is commonly used for expertise estimation related tasks. One such task, in which the received votes are directly used as graded relevance assessment values, is ranking replies of a question. Even though these available assessments values are very practical for evaluation purposes, they may not always reflect the correct assessment value of the content, due to the possible temporal or presentation bias introduced by the CQA system during voting process. This paper analyzes a very commonly used CQA data collection in terms of these introduced biases and their effects on the experimental evaluation of approaches. A more bias free test set construction approach, which has correlated results with the manual assessments, is also proposed in this paper.

References

[1]
K. Balog, Y. Fang, M. de Rijke, P. Serdyukov, and L. Si. Expertise retrieval. Foundations and Trends in Information Retrieval, 6(2--3), 2012.
[2]
M. Bouguessa, B. Dumoulin, and S. Wang. Identifying authoritative actors in question-answering forums: The case of Yahoo! answers. In Proceedings of KDD, 2008.
[3]
X. Liu, W. B. Croft, and M. Koll. Finding experts in community-based question-answering services. In Proceedings of CIKM, 2005.
[4]
J. Zhang, M. S. Ackerman, and L. Adamic. Expertise networks in online communities: structure and algorithms. In Proceedings of WWW, 2007.

Cited By

View all
  • (2021)An Analysis on User Behaviors in Online Question and Answering CommunitiesComputer Supported Cooperative Work and Social Computing10.1007/978-981-16-2540-4_34(469-483)Online publication date: 7-May-2021
  • (2016)Estimating Domain-Specific User Expertise for Answer Retrieval in Community Question-Answering PlatformsProceedings of the 21st Australasian Document Computing Symposium10.1145/3015022.3015032(33-40)Online publication date: 5-Dec-2016
  • (2016)A Comprehensive Survey and Classification of Approaches for Community Question AnsweringACM Transactions on the Web10.1145/293468710:3(1-63)Online publication date: 16-Aug-2016
  • Show More Cited By

Index Terms

  1. Analyzing bias in CQA-based expert finding test sets

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval
    July 2014
    1330 pages
    ISBN:9781450322577
    DOI:10.1145/2600428
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 03 July 2014

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. cqa
    2. expert retrieval
    3. test set construction

    Qualifiers

    • Poster

    Conference

    SIGIR '14
    Sponsor:

    Acceptance Rates

    SIGIR '14 Paper Acceptance Rate 82 of 387 submissions, 21%;
    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)2
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 16 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)An Analysis on User Behaviors in Online Question and Answering CommunitiesComputer Supported Cooperative Work and Social Computing10.1007/978-981-16-2540-4_34(469-483)Online publication date: 7-May-2021
    • (2016)Estimating Domain-Specific User Expertise for Answer Retrieval in Community Question-Answering PlatformsProceedings of the 21st Australasian Document Computing Symposium10.1145/3015022.3015032(33-40)Online publication date: 5-Dec-2016
    • (2016)A Comprehensive Survey and Classification of Approaches for Community Question AnsweringACM Transactions on the Web10.1145/293468710:3(1-63)Online publication date: 16-Aug-2016
    • (2016)Learning to Find Topic Experts in Twitter via Different RelationsIEEE Transactions on Knowledge and Data Engineering10.1109/TKDE.2016.253916628:7(1764-1778)Online publication date: 1-Jul-2016
    • (2016)Modeling Temporal Behavior to Identify Potential Experts in Question Answering CommunitiesCooperative Design, Visualization, and Engineering10.1007/978-3-319-46771-9_7(51-58)Online publication date: 28-Sep-2016
    • (2016)Incorporating Distinct Opinions in Content Recommender SystemInformation Retrieval Technology10.1007/978-3-319-28940-3_9(109-120)Online publication date: 22-Jan-2016

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media