research-article

Segment-level display time as implicit feedback: a comparison to eye tracking

Authors:

Ludger van Elst,

Andreas DengelAuthors Info & Claims

SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval

Pages 67 - 74

https://doi.org/10.1145/1571941.1571955

Published: 19 July 2009 Publication History

Abstract

We examine two basic sources for implicit relevance feedback on the segment level for search personalization: eye tracking and display time. A controlled study has been conducted where 32 participants had to view documents in front of an eye tracker, query a search engine, and give explicit relevance ratings for the results. We examined the performance of the basic implicit feedback methods with respect to improved ranking and compared their performance to a pseudo relevance feedback baseline on the segment level and the original ranking of a Web search engine.

Our results show that feedback based on display time on the segment level is much coarser than feedback from eye tracking. But surprisingly, for re-ranking and query expansion it did work as well as eye-tracking-based feedback. All behavior-based methods performed significantly better than our non-behavior-based baseline and especially improved poor initial rankings of the Web search engine.

The study shows that segment-level display time yields comparable results as eye-tracking-based feedback. Thus, it should be considered in future personalization systems as an inexpensive but precise method for implicit feedback.

References

[1]

E. Agichtein, E. Brill, and S. Dumais. Improving web search ranking by incorporating user behavior information. In SIGIR '06, pp. 19--26, 2006.

Digital Library

[2]

J.W. Ahn, P. Brusilovsky, D. He, J. Grady, and Q. Li. Personalized web exploration with task models. In WWW '08, pp. 1--10, 2008.

Digital Library

[3]

I. Arapakis, J.M. Jose, and P.D. Gray. Affective feedback: an investigation into the role of emotions in the information seeking process. In SIGIR '08, pp. 395--402, 2008.

Digital Library

[4]

N.J. Belkin. Some(what) grand challenges for information retrieval. SIGIR Forum, 42(1):47--54, 2008.

Digital Library

[5]

G. Buscher, A. Dengel, and L. van Elst. Eye movements as implicit relevance feedback. In CHI '08: CHI '08 extended abstracts on Human factors in computing systems, pp. 2991--2996, 2008.

Digital Library

[6]

G. Buscher, A. Dengel, and L. van Elst. Query expansion using gaze-based feedback on the sub-document level. In SIGIR '08, pp. 387--394, 2008.

Digital Library

[7]

M. Claypool, P. Le, M. Wased, and D. Brown. Implicit interest indicators. In IUI '01: Proc. of the 6th intl. conf. on Intelligent user interfaces, pp. 33--40, 2001.

Digital Library

[8]

S. Fox, K. Karnawat, M. Mydland, S. Dumais, and T. White. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems, 23(2):147--168, 2005.

Digital Library

[9]

G. Golovchinsky, M.N. Price, and B.N. Schilit. From reading to retrieval: freeform ink annotations as queries. In SIGIR '99, pp. 19--25, 1999.

Digital Library

[10]

K. Gyllstrom and C. Soules. Seeing is retrieving: building information context from what the user sees. In IUI '08: Proc. of the 13th intl. conf. on Intelligent user interfaces, pp. 189--198, 2008.

Digital Library

[11]

K. Jarvelin and J. Kekäläinen. Ir evaluation methods for retrieving highly relevant documents. In SIGIR '00, pp. 41--48, 2000.

Digital Library

[12]

T. Joachims, L. Granka, B. Pan, H. Hembrooke, F. Radlinski, and G. Gay. Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems (TOIS), 25(2), April 2007.

Digital Library

[13]

D. Kelly and N.J. Belkin. Reading time, scrolling and interaction: exploring implicit sources of user preferences for relevance feedback. In SIGIR '01, pp. 408--409, 2001.

Digital Library

[14]

D. Kelly and N.J. Belkin. Display time as implicit feedback: understanding task effects. In SIGIR '04, pp. 377--384, 2004.

Digital Library

[15]

D. Kelly and J. Teevan. Implicit feedback for inferring user preference: a bibliography. SIGIR Forum, 37(2):18--28, 2003.

Digital Library

[16]

M. Melucci and R.W. White. Discovering hidden contextual factors for implicit feedback. In Proc. of the CIR'07 Workshop on Context-Based Information Retrieval in conjunction with CONTEXT-07, 2007.

[17]

M. Morita and Y. Shinoda. Information filtering based on user behavior analysis and best match text retrieval. In SIGIR '94, pp. 272--281, 1994. Springer.

Digital Library

[18]

D.W. Oard and J. Kim. Modeling information content using observable behavior. In Proc. of the 64 Annual Meeting of the American Society for Information Science and Technology, pp. 38--45, 2001.

[19]

R. Rafter and B. Smyth. Passive profiling from server logs in an online recruitment environment. In ITWP 2001: Proc. of the IJCAI Workshop on Intelligent Techniques for Web Personalization, 2001.

[20]

J. Rocchio. The SMART Retrieval System: Experiments in Automatic Document Processing, chapter Relevance feedback in information retrieval, pp. 313--323. Prentice Hall, 1971.

[21]

J. Salojarvi, K. Puolamäki, and S. Kaski. Implicit relevance feedback from eye movements. In ICANN'05, pp. 513--518, 2005.

Digital Library

[22]

R.W. White and D. Kelly. A study on the effects of personalization and task information on implicit feedback performance. In CIKM '06: Proc. of the 15th ACM intl. conf. on Information and knowledge management, pp. 297--306, 2006.

Digital Library

Cited By

Zhu RCheng CSong YCai Z(2024)Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information SeekingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997508:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699750
Alby T(2024)Bridging the Analytics Gap: Optimizing Content Performance using Actionable Knowledge DiscoveryProceedings of the 35th ACM Conference on Hypertext and Social Media10.1145/3648188.3675121(185-192)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1145/3648188.3675121
Nam DMacvean AMyers BVasilescu B(2024)Understanding Documentation Use Through Log Analysis: A Case Study of Four Cloud ServicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642721(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642721
Show More Cited By

Index Terms

Segment-level display time as implicit feedback: a comparison to eye tracking
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment

Recommendations

Query expansion using gaze-based feedback on the subdocument level
SIGIR '08: Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval

We examine the effect of incorporating gaze-based attention feedback from the user on personalizing the search process. Employing eye tracking data, we keep track of document parts the user read in some way. We use this information on the subdocument ...
Click data as implicit relevance feedback in web search

Search sessions consist of a person presenting a query to a search engine, followed by that person examining the search results, selecting some of those search results for further review, possibly following some series of hyperlinks, and perhaps ...
Image ranking with implicit feedback from eye movements
ETRA '10: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications

In order to help users navigate an image search system, one could provide explicit information on a small set of images as to which of them are relevant or not to their task. These rankings are learned in order to present a user with a new set of images ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval

July 2009

896 pages

ISBN:9781605584836

DOI:10.1145/1571941

General Chairs:
James Allan
University of Massachusetts Amherst, USA
,
Javed Aslam
Northeastern University, USA
,
Program Chairs:
Mark Sanderson
University of Sheffield, UK
,
ChengXiang Zhai
University of Illinois at Urbana-Champaign, USA
,
Justin Zobel
University of Melbourne, Australia

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '09

Sponsor:

SIGIR '09: The 32nd International ACM SIGIR conference on research and development in Information Retrieval

July 19 - 23, 2009

MA, Boston, USA

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

55
Total Citations
View Citations
782
Total Downloads

Downloads (Last 12 months)17
Downloads (Last 6 weeks)4

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhu RCheng CSong YCai Z(2024)Modeling Attentive Interaction Behavior for Web Content Identification in Exploratory Information SeekingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997508:4(1-28)Online publication date: 21-Nov-2024
https://dl.acm.org/doi/10.1145/3699750
Alby T(2024)Bridging the Analytics Gap: Optimizing Content Performance using Actionable Knowledge DiscoveryProceedings of the 35th ACM Conference on Hypertext and Social Media10.1145/3648188.3675121(185-192)Online publication date: 10-Sep-2024
https://dl.acm.org/doi/10.1145/3648188.3675121
Nam DMacvean AMyers BVasilescu B(2024)Understanding Documentation Use Through Log Analysis: A Case Study of Four Cloud ServicesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642721(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642721
Barz MBhatti OSonntag D(2022)Implicit Estimation of Paragraph Relevance From Eye MovementsFrontiers in Computer Science10.3389/fcomp.2021.8085073Online publication date: 7-Jan-2022
https://doi.org/10.3389/fcomp.2021.808507
Liu MKittur AMyers B(2022)Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision MakingProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501968(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501968
Strzelecki A(2020)Eye-Tracking Studies of Web Search Engines: A Systematic Literature ReviewInformation10.3390/info1106030011:6(300)Online publication date: 3-Jun-2020
https://doi.org/10.3390/info11060300
Liu JLiu CBelkin N(2020)Personalization in text information retrievalJournal of the Association for Information Science and Technology10.1002/asi.2423471:3(349-369)Online publication date: 28-Jan-2020
https://dl.acm.org/doi/10.1002/asi.24234
Ma SZha ZWu F(2019)Knowing User Better: Jointly Predicting Click-Through and Playtime for Micro-Video2019 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME.2019.00088(472-477)Online publication date: Jul-2019
https://doi.org/10.1109/ICME.2019.00088
Chen LWang FPu PDíaz Agudo BGoel A(2017)Investigating users’ eye movement behavior in critiquing-based recommender systemsAI Communications10.3233/AIC-17073730:3-4(207-222)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.3233/AIC-170737
Mehrotra RAnderson ADiaz FSharma AWallach HYilmaz EBarrett RCummings RAgichtein EGabrilovich E(2017)Auditing Search Engines for Differential Satisfaction Across DemographicsProceedings of the 26th International Conference on World Wide Web Companion10.1145/3041021.3054197(626-633)Online publication date: 3-Apr-2017
https://dl.acm.org/doi/10.1145/3041021.3054197
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten