research-article

When does Relevance Mean Usefulness and User Satisfaction in Web Search?

Authors:

Hengliang LuoAuthors Info & Claims

SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

Pages 463 - 472

https://doi.org/10.1145/2911451.2911507

Published: 07 July 2016 Publication History

Abstract

Relevance is a fundamental concept in information retrieval (IR) studies. It is however often observed that relevance as annotated by secondary assessors may not necessarily mean usefulness and satisfaction perceived by users. In this study, we confirm the difference by a laboratory study in which we collect relevance annotations by external assessors, usefulness and user satisfaction information by users, for a set of search tasks. We also find that a measure based on usefulness rather than relevance annotated has a better correlation with user satisfaction. However, we show that external assessors are capable of annotating usefulness when provided with more search context information. In addition, we also show that it is possible to generate automatically usefulness labels when some training data is available. Our findings explain why traditional system-centric evaluation metrics are not well aligned with user satisfaction and suggest that a usefulness-based evaluation method can be defined to better reflect the quality of search systems perceived by the users.

References

[1]

A. Al-Maskari, M. Sanderson, and P. Clough. The relationship between ir effectiveness measures and user satisfaction. In Proc. SIGIR '07, pages 773--774, New York, NY, USA, 2007. ACM.

Digital Library

[2]

A. Al-Maskari, M. Sanderson, and P. Clough. Relevance judgments between trec and non-trec assessors. In Proc. SIGIR '08, pages 683--684, New York, NY, USA, 2008. ACM.

Digital Library

[3]

N. J. Belkin, M. Cole, and R. Bierig. Is relevance the right criterion for evaluating interactive information retrieval. In Proc. SIGIR '08 Workshop on Beyond Binary Relevance: Preferences, Diversity, and Set-Level Judgments., 2008.

[4]

G. Buscher, L. van Elst, and A. Dengel. Segment-level display time as implicit feedback: A comparison to eye tracking. In Proc. SIGIR '09, pages 67--74, New York, NY, USA, 2009. ACM.

Digital Library

[5]

J. Carbonell and J. Goldstein. The use of mmr, diversity-based reranking for reordering documents and producing summaries. In Proc. SIGIR '98, pages 335--336, New York, NY, USA, 1998. ACM.

Digital Library

[6]

B. Carterette, E. Kanoulas, M. Hall, and P. Clough. Overview of the trec 2014 session track. 2013.

[7]

O. Chapelle, D. Metlzer, Y. Zhang, and P. Grinspan. Expected reciprocal rank for graded relevance. In Proc. CIKM '09, pages 621--630, New York, NY, USA, 2009. ACM.

Digital Library

[8]

O. Chapelle and Y. Zhang. A dynamic bayesian network click model for web search ranking. In Proc. WWW '09, pages 1--10, 2009.

Digital Library

[9]

C. L. Clarke, M. Kolla, G. V. Cormack, O. Vechtomova, A. Ashkan, S. Büttcher, and I. MacKinnon. Novelty and diversity in information retrieval evaluation. In Proc. SIGIR '08, pages 659--666, 2008.

Digital Library

[10]

C. Cleverdon. The cranfield tests on index language devices. In Aslib proceedings, volume 19, pages 173--194. MCB UP Ltd, 1967.

[11]

J. Cohen. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin, 70(4):213, 1968.

[12]

J. Cohen and P. Cohen. Applied multiple regression/correlation analysis for the behavioral sciences, chapter 2, pages 53--54. Lawrence Erlbaum Associates, 1975.

[13]

M. Cole, J. Liu, N. Belkin, R. Bierig, J. Gwizdka, C. Liu, J. Zhang, and X. Zhang. Usefulness as the criterion for evaluation of interactive information retrieval. Proc. HCIR, pages 1--4, 2009.

[14]

W. S. Cooper. On selecting a measure of retrieval effectiveness. JASIS, 24(2):87--100, 1973.

[15]

Z. Dou, R. Song, and J.-R. Wen. A large-scale evaluation and analysis of personalized search strategies. In Proc. WWW '07, pages 581--590, New York, NY, USA, 2007. ACM.

Digital Library

[16]

S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM TOIS, 23(2):147--168, 2005.

Digital Library

[17]

J. H. Friedman. Greedy function approximation: a gradient boosting machine. Annals of statistics, pages 1189--1232, 2001.

[18]

A. Hassan, R. Jones, and K. Klinkner. Beyond dcg: User behavior as a predictor of a successful search. In Proc. WSDM '10, pages 221--230, 2010.

Digital Library

[19]

A. Hassan, R. W. White, S. T. Dumais, and Y.-M. Wang. Struggling or exploring?: Disambiguating long search sessions. In Proc. WSDM '14, pages 53--62, New York, NY, USA, 2014. ACM.

Digital Library

[20]

S. Huffman and M. Hochster. How well does result relevance predict session satisfaction? In Proc. SIGIR '07, pages 567--574, 2007.

Digital Library

[21]

K. Jarvelin and J. Kekalainen. Cumulated gain-based evaluation of ir techniques. ACM TOIS, 20(4):422--446, Oct. 2002.

Digital Library

[22]

K. J\"arvelin, S. L. Price, L. M. Delcambre, and M. L. Nielsen. Discounted cumulated gain based evaluation of multiple-query ir sessions. In Advances in Information Retrieval, pages 4--15. 2008.

Digital Library

[23]

J. Jiang, A. Hassan Awadallah, X. Shi, and R. W. White. Understanding and predicting graded search satisfaction. In Proc. WSDM '15, pages 57--66, New York, NY, USA, 2015. ACM.

Digital Library

[24]

S. Jung, J. L. Herlocker, and J. Webster. Click data as implicit relevance feedback in web search. Information Processing & Management, 43(3):791--807, 2007.

Digital Library

[25]

E. Kanoulas, B. Carterette, P. Clough, and M. Sanderson. Evaluating multi-query sessions. In Proc. SIGIR '11, pages 1053--1062, 2011.

Digital Library

[26]

J. Kekalainen and K. Jarvelin. Using graded relevance assessments in ir evaluation. JASIST, 53(13):1120--1129, 2002.

Digital Library

[27]

D. Kelly. Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1--2):1--224, 2009.

Digital Library

[28]

D. Kelly, X. Fu, and C. Shah. Effects of rank and precision of search results on users' evaluations of system performance. University of North Carolina, 2007.

[29]

Y. Kim, A. Hassan, R. W. White, and I. Zitouni. Modeling dwell time to predict click-level satisfaction. In Proc. WSDM '14, pages 193--202, New York, NY, USA, 2014. ACM.

Digital Library

[30]

J. R. Landis and G. G. Koch. The measurement of observer agreement for categorical data. biometrics, pages 159--174, 1977.

[31]

C. Liu, J. Liu, N. Belkin, M. Cole, and J. Gwizdka. Using dwell time as an implicit measure of usefulness in different task types. Proc. ASIST, 48(1):1--4, 2011.

[32]

Y. Liu, Y. Chen, and et. al. Different users, different opinions: Predicting search satisfaction with mouse movement information. In Proc. SIGIR '15, pages 493--502, 2015.

Digital Library

[33]

A. Moffat, P. Thomas, and F. Scholer. Users versus models: What observation tells us about effectiveness metrics. In Proc. CIKM '13, pages 659--668, New York, NY, USA, 2013. ACM.

Digital Library

[34]

D. Odijk, R. W. White, A. Hassan Awadallah, and S. T. Dumais. Struggling and success in web search. In Proc. CIKM '15, pages 1551--1560, New York, NY, USA, 2015. ACM.

Digital Library

[35]

T. Sakai and R. Song. Evaluating diversified search results using per-intent graded relevance. In Proc. SIGIR '11, pages 1043--1052.

Digital Library

[36]

M. Sanderson, M. L. Paramita, P. Clough, and E. Kanoulas. Do user preferences and evaluation measures line up? In Proc. SIGIR '10, pages 555--562, New York, NY, USA, 2010. ACM.

Digital Library

[37]

T. Saracevic. Relevance reconsidered. In the Second Conference on Conceptions of Library and Information Science, volume 1, pages 201--218, 1996.

[38]

M. Shokouhi, R. W. White, P. Bennett, and F. Radlinski. Fighting search engine amnesia: Reranking repeated results. In Proc. SIGIR '13, pages 273--282, New York, NY, USA, 2013. ACM.

Digital Library

[39]

J. Teevan, S. T. Dumais, and E. Horvitz. Personalizing search via automated analysis of interests and activities. In Proc. SIGIR '05, pages 449--456, New York, NY, USA, 2005. ACM.

Digital Library

[40]

A. Turpin and F. Scholer. User performance versus precision measures for simple search tasks. In Proc. SIGIR '06, pages 11--18, 2006.

Digital Library

[41]

P. Vakkari and E. Sormunen. The influence of relevance levels on the effectiveness of interactive information retrieval. JASIST, 55(11):963--969, 2004.

Digital Library

[42]

S. Verberne, M. Heijden, M. Hinne, M. Sappelli, S. Koldijk, E. Hoenkamp, and W. Kraaij. Reliability and validity of query intent assessments. JASIST, 64(11):2224--2237, 2013.

[43]

E. M. Voorhees. The philosophy of information retrieval evaluation. In the Second Workshop of the Cross-Language Evaluation Forum on Evaluation of Cross-Language Information Retrieval Systems, CLEF '01, pages 355--370, 2002.

Digital Library

[44]

E. M. Voorhees and D. Harman. Overview of trec 2001. In Trec, 2001.

[45]

R. W. White and S. M. Drucker. Investigating behavioral variability in web search. In Proc. WWW '07, pages 21--30, 2007.

Digital Library

[46]

R. W. White and D. Kelly. A study on the effects of personalization and task information on implicit feedback performance. In Proc. CIKM '06, pages 297--306, New York, NY, USA, 2006. ACM.

Digital Library

[47]

E. Yilmaz, M. Verma, N. Craswell, F. Radlinski, and P. Bailey. Relevance and effort: An analysis of document utility. In Proc. CIKM '14, pages 91--100, New York, NY, USA, 2014. ACM.

Digital Library

Cited By

Chen NLiu JFang HLuo YSakai TWu X(2024)Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System VulnerabilityACM Transactions on Information Systems10.1145/370888443:2(1-58)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3708884
Zhu RCheng CSong YCai Z(2024)Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information SeekingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997508:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699750
Chen NLiu JDong XLiu QSakai TWu XSakai TIshita EOhshima HHasibi FMao JJose J(2024)AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance AssessmentProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698420(54-63)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698420
Show More Cited By

Index Terms

When does Relevance Mean Usefulness and User Satisfaction in Web Search?
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
    2. Users and interactive retrieval

Recommendations

User's perceptions of perceived usefulness, satisfaction, and intentions of mobile application

Mobile applications (apps) have become ubiquitous in our daily lives. This research is focused on users' perceptions of mobile apps' usefulness and the influence they have on user satisfaction and intentions toward these mobile apps. The data was ...
Effects of Perceived Ease of Use and Perceived Usefulness of Technology Acceptance Model on Intention to Continue Using Generative AI: Focusing on the Mediating Effect of Satisfaction and Moderating Effect of Innovation Resistance
Advances in Conceptual Modeling
Abstract
This study analyzed whether perceived ease of use and perceived usefulness influence the intention to continue using generative AI based on the Technology Acceptance Model (TAM). Furthermore, this study analyzed whether user satisfaction has a ...
The influences of system usability and user satisfaction on continued Internet banking services usage intention: empirical evidence from Taiwan

This study proposes an extended technology acceptance model to investigate the effects of system usability and satisfaction on users' intention to continue using Internet banking services. Based on a survey data from 304 respondents, structural equation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

July 2016

1296 pages

ISBN:9781450340694

DOI:10.1145/2911451

General Chairs:
Raffaele Perego
ISTI-CNR, Italy
,
Fabrizio Sebastiani
Qatar Computing Research Institute, HBKU, Qatar
,
Program Chairs:
Javed Aslam
Northeastern University, US
,
Ian Ruthven
University of Strathclyde, UK
,
Justin Zobel
University of Melbourne, Australia

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '16

Sponsor:

SIGIR

SIGIR '16: The 39th International ACM SIGIR conference on research and development in Information Retrieval

July 17 - 21, 2016

Pisa, Italy

Acceptance Rates

SIGIR '16 Paper Acceptance Rate 62 of 341 submissions, 18%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

70
Total Citations
View Citations
806
Total Downloads

Downloads (Last 12 months)67
Downloads (Last 6 weeks)8

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Chen NLiu JFang HLuo YSakai TWu X(2024)Decoy Effect in Search Interaction: Understanding User Behavior and Measuring System VulnerabilityACM Transactions on Information Systems10.1145/370888443:2(1-58)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3708884
Zhu RCheng CSong YCai Z(2024)Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information SeekingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997508:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699750
Chen NLiu JDong XLiu QSakai TWu XSakai TIshita EOhshima HHasibi FMao JJose J(2024)AI Can Be Cognitively Biased: An Exploratory Study on Threshold Priming in LLM-Based Batch Relevance AssessmentProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region10.1145/3673791.3698420(54-63)Online publication date: 8-Dec-2024
https://dl.acm.org/doi/10.1145/3673791.3698420
Wang BLiu J(2024)Understanding users' dynamic perceptions of search gain and cost in sessions: An expectation confirmation modelJournal of the Association for Information Science and Technology10.1002/asi.24935Online publication date: 17-Jun-2024
https://doi.org/10.1002/asi.24935
Van der Sluis FFaure JHomnual S(2024)An empirical exploration of the subjectivity problem of information qualitiesJournal of the Association for Information Science and Technology10.1002/asi.24884Online publication date: 25-Mar-2024
https://doi.org/10.1002/asi.24884
Moffat AMackenzie J(2024)How much freedom does an effectiveness metric really have?Journal of the Association for Information Science and Technology10.1002/asi.2487475:6(686-703)Online publication date: 15-Feb-2024
https://dl.acm.org/doi/10.1002/asi.24874
Urman AMakhortykh MUlloa RSydorova MKulshrestha J(2023)Constants and Variables: How Does the Visual Representation of the Holocaust by AI Change Over TimeEastern European Holocaust Studies10.1515/eehs-2023-00551:2(365-371)Online publication date: 27-Nov-2023
https://doi.org/10.1515/eehs-2023-0055
Bauer CCarterette BFerro NFuhr NBeel JBreuer TClarke CCrescenzi ADemartini GDi Nunzio GDietz LFaggioli GFerwerda BFröbe MHagen MHanbury AHauff CJannach DKando NKanoulas EKnijnenburg BKruschwitz ULi MMaistro MMichiels LPapenmeier APotthast MRosso PSaid ASchaer PSeifert CSpina DStein BTintarev NUrbano JWachsmuth HWillemsen MZobel J(2023)Report on the Dagstuhl Seminar on Frontiers of Information Access Experimentation for Research and EducationACM SIGIR Forum10.1145/3636341.363635157:1(1-28)Online publication date: 4-Dec-2023
https://dl.acm.org/doi/10.1145/3636341.3636351
Massiah JYilmaz EJiao YKazai GFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)On the Reliability of User Feedback for Evaluating the Quality of Conversational AgentsProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615286(4185-4189)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615286
Ai QWang XBendersky MChen HDuh WHuang HKato MMothe JPoblete B(2023)Metric-agnostic Ranking OptimizationProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591935(2669-2680)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591935
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten