research-article

Identifying Expert Intent Queries

Authors:

Manoj Agarwal,

Jaldeep VasavadaAuthors Info & Claims

WWW '20: Companion Proceedings of the Web Conference 2020

Pages 113 - 115

https://doi.org/10.1145/3366424.3382724

Published: 20 April 2020 Publication History

Get Access

Abstract

Search engines assess the quality of their results using nDCG@k for the queries in regular judgement sets. Since these sets are human annotated, the judgement sets are limited in size. Hence, these sets may not have proper representation from “Expert Topics”, e.g., <Machine Learning>, if an Expert Topic represents a tail segment. Therefore, query sets specific to expert topics are prepared to assess the search quality on such topics, which are judged by the domain experts. Such expert judgements are typically 5X-10X more expensive compared to regular judgements. Hence, “Expert Query Sets” must be prepared such that these queries are likely to be judged differently by an expert and a regular judge.

In this paper, we define a novel problem, to identify a sub set of users queries, from a topic specific query set, such that the judgement difference between expert judges and regular judges, called yield rate, is high. Our results, across different expert topics, show that the expert query sets identified through our methodology have ∼15% more yield rate.

References

[1]

A. Aula, R. M. Khan, Z. Guan. 2010. How does Search Behavior Change as Search Becomes More Difficult? In CHI 2010.

Digital Library

Google Scholar

[2]

Y. He, K. Chakrabarti, T. Cheng, T. Tylenda. 2016. Automatic Discovery of Attribute Synonyms Using Query Logs and Table Corpora. In WWW, 2016

Digital Library

Google Scholar

[3]

Jingjing Liu, 2010. Search Behaviors in Different Task Types. In JCDL 2010.

Google Scholar

[4]

K. A. Kinney, S. B. Huffman, J. Zhai. 2008. How Evaluator Domain Expertise Affects Search Result Relevance Judgments. In CIKM 2008.

Google Scholar

[5]

A. Border. 2002. A taxonomy of web search. In SIGIR forum, Fall 2002, Vol. 36, No.2.

Google Scholar

Index Terms

Identifying Expert Intent Queries
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
    2. Information retrieval query processing
  2. Information systems applications

Index terms have been assigned to the content through auto-classification.

Recommendations

Classifying web queries by topic and user intent
CHI EA '10: CHI '10 Extended Abstracts on Human Factors in Computing Systems

In this research, we investigate a methodology to classify automatically Web queries by topic and user intent. Taking a 20,000 plus Web query data set sectioned by topic, we manually classified each query using a three-level hierarchy of user intent. We ...
Determining the user intent of web search engine queries
WWW '07: Proceedings of the 16th international conference on World Wide Web

Determining the user intent of Web searches is a difficult problem due to the sparse data available concerning the searcher. In this paper, we examine a method to determine the user intent underlying Web search engine queries. We qualitatively analyze ...
A qualitative exploration of secondary assessor relevance judging behavior
IIiX '14: Proceedings of the 5th Information Interaction in Context Symposium

Secondary assessors frequently differ in their relevance judgments. Primary assessors are those that originate a search topic and whose judgments truly reflect the assessor's relevance criteria. Secondary assessors do not originate the search and must ...

Comments

Information & Contributors

Information

Published In

WWW '20: Companion Proceedings of the Web Conference 2020

April 2020

854 pages

ISBN:9781450370240

DOI:10.1145/3366424

Editors:
Amal El Fallah Seghrouchni
Sorbonne University, France
,
Gita Sukthankar
University of Central Florida, United States
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
85
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 02 Mar 2025

Other Metrics

View Author Metrics

Citations

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

HTML Format

View this article in HTML Format.

HTML Format

Abstract

References

Index Terms

Recommendations

Classifying web queries by topic and user intent

Determining the user intent of web search engine queries

A qualitative exploration of secondary assessor relevance judging behavior

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Login options

Full Access

View options

PDF

eReader

HTML Format

Share

Share this Publication link

Share on social media

Affiliations