research-article

Investigating Examination Behavior in Mobile Search

Authors:

Mark Sanderson,

Shaoping MaAuthors Info & Claims

WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

Pages 771 - 779

https://doi.org/10.1145/3336191.3371797

Published: 22 January 2020 Publication History

Abstract

Examination is one of the most important user interactions in Web search. A number of works studied examination behavior in Web search and helped researchers better understand how users allocate their attention on search engine result pages (SERPs). Compared to desktop search, mobile search has a number of differences such as fewer results on the screen. These differences bring in mobile-specific factors affecting users' examination behavior. However, there still lacks research on users' attention allocation mechanism via viewports in mobile search. Therefore, we design a lab-based study to collect user's rich interaction behavior in mobile search. Based on the collected data, we first analyze how users examine SERPs and allocate their attention to heterogeneous results. Then we investigate the effect of mobile-specific factors and other common factors on users allocating attention. Finally, we apply the findings of user attention allocation from the user study into click model construction efforts, which significantly improves the state-of-the-art click model. Our work brings insights into a better understanding of users' interaction patterns in mobile search and may benefit other mobile search-related research.

References

[1]

Judit Bar-Ilan, Kevin Keenoy, Mark Levene, and Eti Yaari. 2009. Presentation bias is significant in determining user preference for search results-A user study. Journal of the American Society for Information Science and Technology 60, 1 (2009), 135--149.

Digital Library

[2]

Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A Neural Click Model for Web Search. In WWW '16. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 531--541.

Digital Library

[3]

Georg Buscher, Susan T. Dumais, and Edward Cutrell. 2010. The Good, the Bad, and the Random: An Eye-tracking Study of Ad Quality in Web Search. In SIGIR '10. ACM, New York, NY, USA, 42--49.

Digital Library

[4]

Olivier Chapelle and Ya Zhang. 2009. A Dynamic Bayesian Network Click Model for Web Search Ranking. In WWW '09. ACM, New York, NY, USA, 1--10.

Digital Library

[5]

Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015. Click models for web search. Synthesis Lectures on Information Concepts, Retrieval, and Services 7, 3 (2015), 1--115.

[6]

Michael J. Cole, Chathra Hendahewa, Nicholas J. Belkin, and Chirag Shah. 2014. Discrimination Between Tasks with User Activity Patterns During Information Search. In SIGIR '14. ACM, New York, NY, USA, 567--576.

[7]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An Experimental Comparison of Click Position-bias Models. In WSDM '08. ACM, New York, NY, USA, 87--94.

[8]

Fernando Diaz, Ryen White, Georg Buscher, and Dan Liebling. 2013. Robust models of mouse movement on dynamic web search results pages. In CIKM '13. ACM, New York, NY, USA, 1451--1460.

Digital Library

[9]

Susan T. Dumais, Georg Buscher, and Edward Cutrell. 2010. Individual Differences in Gaze Patterns for Web Search. In IIiX '10. ACM, New York, NY, USA, 185--194.

[10]

Georges E. Dupret and Benjamin Piwowarski. 2008. A User Browsing Model to Predict Search Engine Click Data from Past Observations. In SIGIR '08. ACM, New York, NY, USA, 331--338.

[11]

Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76, 5 (1971), 378.

[12]

Laura A. Granka, Thorsten Joachims, and Geri Gay. 2004. Eye-tracking Analysis of User Behavior in WWW Search. In SIGIR '04. ACM, New York, NY, USA, 478--479.

[13]

Zhiwei Guan and Edward Cutrell. 2007. An Eye Tracking Study of the Effect of Target Rank on Web Search. In CHI '07. ACM, New York, NY, USA, 417--420.

[14]

Qi Guo and Eugene Agichtein. 2012. Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior. In WWW '12. ACM, 569--578.

Digital Library

[15]

Qi Guo, Haojian Jin, Dmitry Lagun, Shuai Yuan, and Eugene Agichtein. 2013. Mining touch interaction data on mobile devices to predict web search result relevance. In SIGIR '13. ACM, 153--162.

Digital Library

[16]

Samuel Ieong, Nina Mishra, Eldar Sadikov, and Li Zhang. 2012. Domain Bias in Web Search. In WSDM '12. ACM, New York, NY, USA, 413--422.

[17]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately Interpreting Clickthrough Data As Implicit Feedback. In SIGIR '05. ACM, New York, NY, USA, 154--161.

[18]

Jaewon Kim, Paul Thomas, Ramesh Sankaranarayana, Tom Gedeon, and HwanJin Yoon. 2015. Eye-tracking Analysis of User Behavior and Performance in Web Search on Large and Small Screens. J. Assoc. Inf. Sci. Technol. 66, 3 (March 2015), 526--544.

Digital Library

[19]

Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, and Vidhya Navalpakkam. 2014. Towards Better Measurement of Attention and Satisfaction in Mobile Search. In SIGIR '14. ACM, New York, NY, USA, 113--122.

[20]

Dmitry Lagun, Donal McMahon, and Vidhya Navalpakkam. 2016. Understanding Mobile Searcher Attention with Rich Ad Formats. In CIKM '16. ACM, New York, NY, USA, 599--608.

[21]

J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics (1977), 159--174.

[22]

Xiangsheng Li, Yiqun Liu, Jiaxin Mao, Zexue He, Min Zhang, and Shaoping Ma. 2018. Understanding Reading Attention Distribution During Relevance Judgement. In CIKM '18. ACM, New York, NY, USA, 733--742.

[23]

Yiqun Liu, Chao Wang, Ke Zhou, Jianyun Nie, Min Zhang, and Shaoping Ma. 2014. From Skimming to Reading: A Two-stage Examination Model for Web Search. In CIKM '14. ACM, New York, NY, USA, 849--858.

Digital Library

[24]

Zeyang Liu, Yiqun Liu, Ke Zhou, Min Zhang, and Shaoping Ma. 2015. Influence of Vertical Result in Web Search Examination. In SIGIR '15. ACM, New York, NY, USA, 193--202.

[25]

Cheng Luo, Yiqun Liu, Tetsuya Sakai, Fan Zhang, Min Zhang, and Shaoping Ma. 2017. Evaluating Mobile Search with Height-Biased Gain. In SIGIR '17. ACM, New York, NY, USA, 435--444.

[26]

Barry R Manor and Evian Gordon. 2003. Defining the temporal threshold for ocular fixation in free-viewing visuocognitive tasks. Journal of neuroscience methods 128, 1--2 (2003), 85--93.

[27]

Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. Constructing Click Models for Mobile Search. In SIGIR '18. ACM, New York, NY, USA, 775--784.

[28]

Vidhya Navalpakkam, LaDawn Jentzsch, Rory Sayres, Sujith Ravi, Amr Ahmed, and Alex Smola. 2013. Measurement and Modeling of Eye-mouse Behavior in the Presence of Nonlinear Page Layouts. In WWW '13. ACM, New York, NY, USA, 953--964.

[29]

Rachana S Rele and Andrew T Duchowski. 2005. Using eye tracking to evaluate alternative search results interfaces. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, Vol. 49. SAGE Publications Sage CA: Los Angeles, CA, 1459--1463.

[30]

Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007. Predicting Clicks: Estimating the Click-through Rate for New Ads. In WWW '07. ACM, New York, NY, USA, 521--530.

Digital Library

[31]

Milad Shokouhi and Qi Guo. 2015. From queries to cards: Re-ranking proactive card recommendations based on reactive search history. In SIGIR '15. ACM, 695--704.

Digital Library

[32]

Yang Song, Hao Ma, Hongning Wang, and Kuansan Wang. 2013. Exploring and Exploiting User Search Behavior on Mobile and Tablet Devices to Improve Search Relevance. In WWW '13. ACM, New York, NY, USA, 1201--1212.

Digital Library

[33]

Ellen M. Voorhees and Donna K. Harman. 2005. TREC: Experiment and Evaluation in Information Retrieval (Digital Libraries and Electronic Publishing). The MIT Press.

[34]

Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating Vertical Results into Search Click Models. In SIGIR '13. ACM, New York, NY, USA, 503--512.

Digital Library

[35]

Xiaochuan Wang, Ning Su, Zexue He, Yiqun Liu, and Shaoping Ma. 2018. A Large-Scale Study of Mobile Search Examination Behavior. In SIGIR '18. ACM, New York, NY, USA, 1129--1132.

Digital Library

[36]

Ryen W White, Fernando Diaz, and Qi Guo. 2017. Search Result Prefetching on Desktop and Mobile. ACM Transactions on Information Systems (TOIS) 35, 3 (2017), 23.

Digital Library

[37]

Yisong Yue, Rajan Patel, and Hein Roehrig. 2010. Beyond Position Bias: Examining Result Attractiveness As a Source of Presentation Bias in Clickthrough Data. In WWW '10. ACM, New York, NY, USA, 1011--1018.

Digital Library

[38]

Yukun Zheng, Jiaxin Mao, Yiqun Liu, Cheng Luo, Min Zhang, and Shaoping Ma. 2019. Constructing Click Model for Mobile Search with Viewport Time. ACM Trans. Inf. Syst. 37, 4, Article 43 (Sept. 2019), 34 pages.

Digital Library

Cited By

Reed JMacfarlane AMakri S(2024)Mobile search made easier: An ability-based mobile search prototype for people with dyslexiaProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638292(45-55)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638292
Liu XGuo CYao BSarikaya R(2024)A Self-Learning Framework for Large-Scale Conversational AI SystemsIEEE Computational Intelligence Magazine10.1109/MCI.2024.336397119:2(34-48)Online publication date: 5-Apr-2024
https://dl.acm.org/doi/10.1109/MCI.2024.3363971
Liu JWang YWang JWang MChu X(2024)Probabilistic graph model and neural network perspective of click models for web searchKnowledge and Information Systems10.1007/s10115-024-02145-z66:10(5829-5873)Online publication date: 6-Jun-2024
https://doi.org/10.1007/s10115-024-02145-z
Show More Cited By

Index Terms

Investigating Examination Behavior in Mobile Search
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
2. Information systems
  1. Information retrieval
    1. Search engine architectures and scalability
      1. Retrieval on mobile devices
  2. World Wide Web
    1. Web searching and information discovery
      1. Web search engines

Recommendations

Investigating collaborative mobile search behaviors
MobileHCI '13: Proceedings of the 15th international conference on Human-computer interaction with mobile devices and services

People use mobile devices to search, locate and discover local information around them. Mobile local search is frequently a social activity. This paper presents the results of a survey and an exploratory user study of collaborative mobile local search. ...
Behavior Modeling for Point of Interest Search
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

With the increasing popularity of location-based services, the point-of-interest (POI) search has received considerable attention in recent years. Existing studies on POI search mostly focus on how to construct better retrieval models to retrieve the ...
Investigating Examination Behavior of Image Search Users
SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Image search engines show results differently from general Web search engines in three key ways: (1) most Web-based image search engines adopt the two-dimensional result placement instead of the linear result list; (2) image searches show snapshots ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

January 2020

950 pages

ISBN:9781450368223

DOI:10.1145/3336191

General Chairs:
James Caverlee
Texas A&M University
,
Xia "Ben" Hu
Texas A&M University
,
Program Chairs:
Mounia Lalmas
Spotify
,
Wei Wang
University of California, Los Angeles

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 January 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Key Research and Devel- opment Program of China
Natural Science Foundation of China

Conference

WSDM '20

Sponsor:

WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining

February 3 - 7, 2020

TX, Houston, USA

Acceptance Rates

Overall Acceptance Rate 498 of 2,863 submissions, 17%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
327
Total Downloads

Downloads (Last 12 months)24
Downloads (Last 6 weeks)2

Reflects downloads up to 27 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Reed JMacfarlane AMakri S(2024)Mobile search made easier: An ability-based mobile search prototype for people with dyslexiaProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638292(45-55)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638292
Liu XGuo CYao BSarikaya R(2024)A Self-Learning Framework for Large-Scale Conversational AI SystemsIEEE Computational Intelligence Magazine10.1109/MCI.2024.336397119:2(34-48)Online publication date: 5-Apr-2024
https://dl.acm.org/doi/10.1109/MCI.2024.3363971
Liu JWang YWang JWang MChu X(2024)Probabilistic graph model and neural network perspective of click models for web searchKnowledge and Information Systems10.1007/s10115-024-02145-z66:10(5829-5873)Online publication date: 6-Jun-2024
https://doi.org/10.1007/s10115-024-02145-z
Wu ZMao JXu KSong DHuang H(2023)A Passage-Level Reading Behavior Model for Mobile SearchProceedings of the ACM Web Conference 202310.1145/3543507.3583343(3236-3246)Online publication date: 30-Apr-2023
https://dl.acm.org/doi/10.1145/3543507.3583343
Chen HAi QWu ZWang ZLiu YZhang MMa SHu JTan NChai HChen HDuh WHuang HKato MMothe JPoblete B(2023)Behavior Modeling for Point of Interest SearchProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591955(1843-1847)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591955
Cai XXiao RZeng ZGong PNi Y(2023)ITran: A novel transformer-based approach for industrial anomaly detection and localizationEngineering Applications of Artificial Intelligence10.1016/j.engappai.2023.106677125(106677)Online publication date: Oct-2023
https://doi.org/10.1016/j.engappai.2023.106677
Shao YMao JLiu YZhang MMa S(2022)From linear to non-linear: investigating the effects of right-rail results on complex SERPsAdvances in Computational Intelligence10.1007/s43674-021-00028-22:1Online publication date: 10-Jan-2022
https://doi.org/10.1007/s43674-021-00028-2
Li HLu HHuang SMa WZhang MLiu YMa SDiaz FShah CSuel TCastells PJones RSakai T(2021)Privacy-Aware Remote Information Retrieval User Experiments Logging ToolProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3462793(2615-2619)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3462793
Zheng YMao JLiu YXie XZhang MMa S(2020)Investigating Fine-Grained Usefulness Perception Process in Mobile SearchInformation Retrieval10.1007/978-3-030-56725-5_2(17-28)Online publication date: 10-Aug-2020
https://doi.org/10.1007/978-3-030-56725-5_2

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten