research-article

News stories relevance effects on eye-movements

Author:

Jacek GwizdkaAuthors Info & Claims

ETRA '14: Proceedings of the Symposium on Eye Tracking Research and Applications

Pages 283 - 286

https://doi.org/10.1145/2578153.2578198

Published: 26 March 2014 Publication History

Abstract

Relevance is a fundamental concept in information retrieval. We consider relevance from the user's perspective and ask if the degree of relevance can be inferred from eye-tracking data and if it is related to the cognitive effort involved in relevance judgments. To this end we conducted a study, in which participants were asked to find information in screen-long text documents containing news stories. Each participant responded to fourteen trials consisting of an information question followed by three documents each at a different level of relevance (irrelevant, partially relevant, and relevant). The results indicate that relevant documents tended to be continuously read, while irrelevant documents tended to be scanned. In most cases, cognitive effort inferred from eye-tracking data was highest for partially relevant documents and lowest for irrelevant documents.

References

[1]

Ajanki, A., Hardoon, D., Kaski, S., Puolamäki, K., & Shawe-Taylor, J. (2009). Can eyes reveal interest? Implicit queries from gaze patterns. User Modeling and User-Adapted Interaction, 19(4), 307--339

Digital Library

[2]

Balatsoukas, P., Ruthven, I. (2012). An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine. Journal of the American Society for Information Science & Technology, 63(9), 1728--1746.

Digital Library

[3]

Borlund, P. (2003). The concept of relevance in IR. Journal of American Society for Information Science & Technology, 54(10), 913--925.

Digital Library

[4]

Buscher, G., Dengel, A., Biedert, R., & Elst, L. V. (2012). Attentive documents: Eye tracking as implicit feedback for information retrieval and beyond. ACM Transactions on Interactive and Intelligent Systems, 1(2), 9:1--9:30.

Digital Library

[5]

Cole, M., Gwizdka, J., Liu C., Bierig, R., Belkin, N., Zhang, X. (2011). Task amd User Effects on Reading Patterns in Information Search. Interacting with Computers. 23(4), 346--362.

[6]

Graff, D. (2002). The AQUAINT Corpus of English News Text, Linguistic Data Consortium, Philadelphia.

[7]

Kelly, D. and Belkin, N. J. (2004). Display Time As Implicit Feedback: Understanding Task Effects. SIGIR'04. 377--384.

Digital Library

[8]

Loboda, T. D., Brusilovsky, P., & Brunstein, J. (2011). Inferring word relevance from eye-movements of readers. In Proceedings of IUI'11. (pp. 175--184). New York, NY: ACM.

Digital Library

[9]

Saracevic, T. (1975). Relevance: A review of and a framework for the thinking on the notion in information science. Society for Information Science, 26, 321--343.

[10]

Rayner, K., Pollatsek, A., Ashby, J., & Jr, C. C. (2011). Psychology of Reading. Psychology Press

[11]

Rayner, K., Chace, K. H., Slattery, T. J., & Ashby, J. (2006). Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading, 10, 241--255.

[12]

Reichle, E. D., Pollatsek, A., & Rayner, K. (2006). E-Z Reader: A cognitive-control, serial-attention model of eye-movement behavior during reading. Cognitive Systems Research, 7, 4--22.

Digital Library

[13]

Villa, R., & Halvey, M. (2013). Is relevance hard work?: evaluating the effort of making relevant assessments. In Proceedings of SIGIR'13. (pp. 765--768). New York, NY: ACM.

Digital Library

[14]

Voorhees, E. M., Dang, H. T. (2005) Overview of the TREC 2005 Question Answering Track. NIST.

[15]

White, R., & Kelly, D. (2006). A study on the effects of personalization and task information on implicit feedback performance. Proceedings of CIKM'06, (pp. 297--306).

Digital Library

[16]

Xie, B., & Salvendy, G. (2000). Prediction of Metal Workload in Single and Multiple Task Environments. International Journal of Cognitive Ergonomics. 4(3), 213--242.

Cited By

Mohamed Selim ABhatti OBarz MSonntag D(2024)Perceived Text Relevance Estimation Using Scanpaths and GNNsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685736(418-427)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685736
Hansen CHansen CSimonsen JLarsen BAlstrup SLioma CHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Factuality Checking in News Headlines with Eye TrackingProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401221(2013-2016)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401221
Bhattacharya NRakshit SGwizdka J(2020)Towards Real-time Webpage Relevance Prediction UsingConvex Hull Based Eye-tracking FeaturesACM Symposium on Eye Tracking Research and Applications10.1145/3379157.3391302(1-10)Online publication date: 2-Jun-2020
https://dl.acm.org/doi/10.1145/3379157.3391302
Show More Cited By

Index Terms

News stories relevance effects on eye-movements
1. Information systems
  1. Information retrieval

Recommendations

Characterizing relevance with eye-tracking measures
IIiX '14: Proceedings of the 5th Information Interaction in Context Symposium

Relevance, a fundamental concept in information search and retrieval, is 80-years old [4]. The recent decades have been ripe with work that brought a much better understanding of this rich concept. Yet, we still don't know which cognitive and affective ...
Is relevance hard work?: evaluating the effort of making relevant assessments
SIGIR '13: Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval

The judging of relevance has been a subject of study in information retrieval for a long time, especially in the creation of relevance judgments for test collections. While the criteria by which assessors? judge relevance has been intensively studied, ...
Inferring word relevance from eye-movements of readers
IUI '11: Proceedings of the 16th international conference on Intelligent user interfaces

Reading is one of the most important skills in today's society. The ubiquity of this activity has naturally affected many information systems; the only goal of some is the presentation of textual information. One concrete task often performed on a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ETRA '14: Proceedings of the Symposium on Eye Tracking Research and Applications

March 2014

394 pages

ISBN:9781450327510

DOI:10.1145/2578153

Conference Chairs:
Pernilla Qvarfordt
FX Palo Alto Laboratory
,
Dan Witzner Hansen
IT University, Denmark

Copyright © 2014 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 March 2014

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

ETRA '14

Sponsor:

ETRA '14: Eye Tracking Research and Applications

March 26 - 28, 2014

Florida, Safety Harbor

Acceptance Rates

Overall Acceptance Rate 69 of 137 submissions, 50%

Upcoming Conference

ETRA '25

Sponsor:
sigchi
sigchi

2025 Symposium on Eye Tracking Research and Applications

May 26 - 29, 2025

Tokyo , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
246
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mohamed Selim ABhatti OBarz MSonntag D(2024)Perceived Text Relevance Estimation Using Scanpaths and GNNsProceedings of the 26th International Conference on Multimodal Interaction10.1145/3678957.3685736(418-427)Online publication date: 4-Nov-2024
https://dl.acm.org/doi/10.1145/3678957.3685736
Hansen CHansen CSimonsen JLarsen BAlstrup SLioma CHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)Factuality Checking in News Headlines with Eye TrackingProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401221(2013-2016)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401221
Bhattacharya NRakshit SGwizdka J(2020)Towards Real-time Webpage Relevance Prediction UsingConvex Hull Based Eye-tracking FeaturesACM Symposium on Eye Tracking Research and Applications10.1145/3379157.3391302(1-10)Online publication date: 2-Jun-2020
https://dl.acm.org/doi/10.1145/3379157.3391302
Bhattacharya NRakshit SGwizdka JKogut PO'Brien HFreund LArapakis IHoeber OLopatovska I(2020)Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural NetworksProceedings of the 2020 Conference on Human Information Interaction and Retrieval10.1145/3343413.3377960(223-233)Online publication date: 14-Mar-2020
https://dl.acm.org/doi/10.1145/3343413.3377960
Jasiewicz JKisilowska MJupowicz-Ginalska A(2019)Biometric Tools in Information Science. The Example of an Information Literacy Study – A Holiday Planning ExperimentInformation Literacy in Everyday Life10.1007/978-3-030-13472-3_3(23-32)Online publication date: 20-Feb-2019
https://doi.org/10.1007/978-3-030-13472-3_3
Jayarathna SShanmugathasan SChen JGonçalves MAllen JFox EKan MPetras V(2018)Evaluating Saccade-Bounded Eye Movement Features for the User ModelingProceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries10.1145/3197026.3197072(21-24)Online publication date: 23-May-2018
https://dl.acm.org/doi/10.1145/3197026.3197072
Gwizdka JMostafa J(2016)NeuroIR 2015ACM SIGIR Forum10.1145/2888422.288843549:2(83-88)Online publication date: 29-Jan-2016
https://dl.acm.org/doi/10.1145/2888422.2888435
Gwizdka JJose JMostafa JWilson MBaeza-Yates RLalmas MMoffat ARibeiro-Neto B(2015)NeuroIR 2015Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/2766462.2767856(1151-1153)Online publication date: 9-Aug-2015
https://dl.acm.org/doi/10.1145/2766462.2767856
Rieh SGwizdka JFreund LCollins‐Thompson K(2015)Searching as learning: Novel measures for information interaction researchProceedings of the American Society for Information Science and Technology10.1002/meet.2014.1450510102151:1(1-4)Online publication date: 24-Apr-2015
https://doi.org/10.1002/meet.2014.14505101021
Zhang YZhang JLease MGwizdka JGeva STrotman ABruza PClarke CJärvelin K(2014)Multidimensional relevance modeling via psychometrics and crowdsourcingProceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval10.1145/2600428.2609577(435-444)Online publication date: 3-Jul-2014
https://dl.acm.org/doi/10.1145/2600428.2609577

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten