skip to main content
10.1145/1458082.1458160acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

How evaluator domain expertise affects search result relevance judgments

Published: 26 October 2008 Publication History

Abstract

Traditional search evaluation approaches have often relied on domain experts to evaluate results for each query. Unfortunately, the range of topics present in any representative sample of web queries makes it impractical to have expert evaluators for every topic. In this paper, we investigate the effect of using "generalist" evaluators instead of experts in the domain of queries being evaluated. Empirically, we ind that for queries drawn from domains requiring high expertise, (1) generalists tend to give shallow, inaccurate ratings as compared to experts. (2) Further experiments show that generalists disagree on the underlying meaning of these queries significantly more often than experts, and often appear to "give up'' and fall back on surface features such as keyword matching. (3) Finally, by estimating the percentage of "expertise requiring'' queries in a web query sample, we estimate the impact of using generalists, versus the ideal of having domain experts for every "expertise requiring'' query.

References

[1]
A. Aula and K. Nordhausen. Modeling successful performance in web searching. J. Am. Soc. Inf. Sci. Technol., 57(12):1678--1693, 2006.
[2]
R. E. Downing, J. L. Moore, and S. W. Brown. The effects and interaction of spatial visualization and domain expertise on information seeking. Computers in Human Behavior, 21(2):195--209, 2004.
[3]
G. B. Duggan and S. J. Payne. Knowledge in the head and on the web: using topic expertise to aid search. In CHI '08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems, pages 39--48, New York, NY, USA, 2008. ACM.
[4]
D. Harman. Overview of the first trec conference. In SIGIR '93: Proceedings of the 16th annual international ACM SIGIR conference on Research and development in information retrieval, pages 36--47, New York, NY, USA, 1993. ACM.
[5]
W. Hersh, C. Buckley, T. J. Leone, and D. Hickam. Ohsumed: an interactive retrieval evaluation and new large test collection for research. In SIGIR '94: Proceedings of the 17th annual international ACM SIGIR conference on Research and development in information retrieval, pages 192--201, New York, NY, USA, 1994. Springer-Verlag New York, Inc.
[6]
C. Holscher and G. Strube. Web search behavior of internet experts and newbies. Computer Networks, 33(1):337--346, 2000.
[7]
K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422--446, 2002.
[8]
C. Jenkins, C. L. Corritore, and S. Wiedenbeck. Patterns of information seeking on the web: A qualitative study of domain expertise and web expertise. IT and Society, 1(3):64--89, 2003.
[9]
P. Vakkari, M. Pennanen, and S. Serola. Changes of search terms and tactics while writing a research proposal: A longitudinal case study. Inf. Process. Manage., 39(3):445--463, 2003.
[10]
B. M. Wildemuth. The effects of domain knowledge on search tactic formulation. Journal of the American Society for Information Science and Technology, 55(3):246--258, 2003.
[11]
X. Zhang, H. G. B. Anghelescu, and X. Yuan. Domain knowledge, search behavior, and search effectiveness of engineering and science students: an exploratory study. Information Research, 10(2), 2005.

Cited By

View all
  • (2024)LLMs can be Fooled into Labelling a Document as Relevant: best café near me; this paper is perfectly relevantProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698431(32-41)Online publication date: 8-Dec-2024
  • (2023)Query sampler: generating query sets for analyzing search engines using keyword research toolsPeerJ Computer Science10.7717/peerj-cs.14219(e1421)Online publication date: 7-Jun-2023
  • (2023)On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze AssessmentsACM Transactions on Information Systems10.1145/360022742:1(1-31)Online publication date: 21-Aug-2023
  • Show More Cited By

Index Terms

  1. How evaluator domain expertise affects search result relevance judgments

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge management
    October 2008
    1562 pages
    ISBN:9781595939913
    DOI:10.1145/1458082
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 October 2008

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. domain expertise
    2. information retrieval
    3. search evaluation

    Qualifiers

    • Research-article

    Conference

    CIKM08
    CIKM08: Conference on Information and Knowledge Management
    October 26 - 30, 2008
    California, Napa Valley, USA

    Acceptance Rates

    Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

    Upcoming Conference

    CIKM '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)6
    • Downloads (Last 6 weeks)1
    Reflects downloads up to 23 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)LLMs can be Fooled into Labelling a Document as Relevant: best café near me; this paper is perfectly relevantProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698431(32-41)Online publication date: 8-Dec-2024
    • (2023)Query sampler: generating query sets for analyzing search engines using keyword research toolsPeerJ Computer Science10.7717/peerj-cs.14219(e1421)Online publication date: 7-Jun-2023
    • (2023)On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze AssessmentsACM Transactions on Information Systems10.1145/360022742:1(1-31)Online publication date: 21-Aug-2023
    • (2023)The Impact of Judgment Variability on the Consistency of Offline Effectiveness MeasuresACM Transactions on Information Systems10.1145/359651142:1(1-31)Online publication date: 18-Aug-2023
    • (2022)In Search of Ambiguity: A Three-Stage Workflow Design to Clarify Annotation Guidelines for Crowd WorkersFrontiers in Artificial Intelligence10.3389/frai.2022.8281875Online publication date: 18-May-2022
    • (2022)Understanding and Evaluating Search ExperienceSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S01166ED1V01Y202202ICR07714:1(1-105)Online publication date: 28-Mar-2022
    • (2021)The Challenge of Variable Effort Crowdsourcing and How Visible Gold Can HelpProceedings of the ACM on Human-Computer Interaction10.1145/34760735:CSCW2(1-26)Online publication date: 18-Oct-2021
    • (2021)CURE: Collection for Urdu Information Retrieval Evaluation and Ranking2021 International Conference on Digital Futures and Transformative Technologies (ICoDT2)10.1109/ICoDT252288.2021.9441510(1-6)Online publication date: 20-May-2021
    • (2019)The Practice of CrowdsourcingSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00904ED1V01Y201903ICR06611:1(1-149)Online publication date: 28-May-2019
    • (2019)Variations in Assessor Agreement in Due DiligenceProceedings of the 2019 Conference on Human Information Interaction and Retrieval10.1145/3295750.3298945(243-247)Online publication date: 8-Mar-2019
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media