research-article

Beyond binary relevance: preferences, diversity, and set-level judgments

Authors:

Paul N. Bennett,

Ben Carterette,

Olivier Chapelle,

Thorsten JoachimsAuthors Info & Claims

ACM SIGIR Forum, Volume 42, Issue 2

Pages 53 - 58

https://doi.org/10.1145/1480506.1480516

Published: 30 November 2008 Publication History

Get Access

Cited By

View all

Arabzadeh NVtyurina AYan XClarke C(2022)Shallow pooling for sparse labelsInformation Retrieval Journal10.1007/s10791-022-09411-025:4(365-385)Online publication date: 20-Jul-2022
https://doi.org/10.1007/s10791-022-09411-0
Kim JYilmaz EBaeza-Yates RLalmas MMoffat ARibeiro-Neto B(2015)IR EvaluationProceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/2766462.2767875(1129-1132)Online publication date: 9-Aug-2015
https://dl.acm.org/doi/10.1145/2766462.2767875
Urbano JSchedl MSerra X(2013)Evaluation in Music Information RetrievalJournal of Intelligent Information Systems10.1007/s10844-013-0249-441:3(345-369)Online publication date: 1-Dec-2013
https://dl.acm.org/doi/10.1007/s10844-013-0249-4
Show More Cited By

Index Terms

Beyond binary relevance: preferences, diversity, and set-level judgments
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
        Supervised learning by classification
    2. Machine learning approaches
      1. Classification and regression trees
2. Information systems

Recommendations

Relevance dimensions in preference-based IR evaluation
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

Evaluation of information retrieval (IR) systems has recently been exploring the use of preference judgments over two search result lists. Unlike the traditional method of collecting relevance labels per single result, this method allows to consider the ...
Relevance Judgments: Preferences, Scores and Ties
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Conventionally, relevance judgments were assessed using ordinal relevance scales such as binary and Sormunen categories [9]. Such judgments record how much overlap there is between the document and the topic. However they have been argued as unreliable ...
Beyond independent relevance: methods and evaluation metrics for subtopic retrieval
SIGIR '03: Proceedings of the 26th annual international ACM SIGIR conference on Research and development in informaion retrieval

We present a non-traditional retrieval problem we call subtopic retrieval. The subtopic retrieval problem is concerned with finding documents that cover many different subtopics of a query topic. In such a problem, the utility of a document in a ranking ...

Comments

Information & Contributors

Information

Published In

ACM SIGIR Forum Volume 42, Issue 2

December 2008

101 pages

ISSN:0163-5840

DOI:10.1145/1480506

Issue’s Table of Contents

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 30 November 2008

Published in SIGIR Volume 42, Issue 2

Check for updates

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
167
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Arabzadeh NVtyurina AYan XClarke C(2022)Shallow pooling for sparse labelsInformation Retrieval Journal10.1007/s10791-022-09411-025:4(365-385)Online publication date: 20-Jul-2022
https://doi.org/10.1007/s10791-022-09411-0
Kim JYilmaz EBaeza-Yates RLalmas MMoffat ARibeiro-Neto B(2015)IR EvaluationProceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/2766462.2767875(1129-1132)Online publication date: 9-Aug-2015
https://dl.acm.org/doi/10.1145/2766462.2767875
Urbano JSchedl MSerra X(2013)Evaluation in Music Information RetrievalJournal of Intelligent Information Systems10.1007/s10844-013-0249-441:3(345-369)Online publication date: 1-Dec-2013
https://dl.acm.org/doi/10.1007/s10844-013-0249-4
Tao KHauff CHouben G(2013)Building a Microblog Corpus for Search Result DiversificationInformation Retrieval Technology10.1007/978-3-642-45068-6_22(251-262)Online publication date: 2013
https://doi.org/10.1007/978-3-642-45068-6_22
Lin GPeng HMa QWei JQin J(2010)Improving diversity in Web search results re-ranking using absorbing random walks2010 International Conference on Machine Learning and Cybernetics10.1109/ICMLC.2010.5580733(2116-2421)Online publication date: Jul-2010
https://doi.org/10.1109/ICMLC.2010.5580733
Radlinski FBennett PCarterette BJoachims T(2009)Redundancy, diversity and interdependent document relevanceACM SIGIR Forum10.1145/1670564.167057243:2(46-52)Online publication date: 14-Dec-2009
https://dl.acm.org/doi/10.1145/1670564.1670572

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Cited By

Index Terms

Recommendations

Relevance dimensions in preference-based IR evaluation

Relevance Judgments: Preferences, Scores and Ties

Beyond independent relevance: methods and evaluation metrics for subtopic retrieval

Comments

Information

Published In

Publisher

Publication History

Check for updates

Qualifiers

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations