Abstract
With the development and popularization of smartphones, search on mobile devices has become more and more popular in recent years. Existing research found that users’ search interaction patterns in the mobile environment are different from those in the desktop environment. As we know, there are a number of vertical results and richly informative snippets in the ranked lists of mobile search engines. Users can perceive useful information from both the snippet and the landing page of a result. Therefore, we consider that it is necessary to investigate how users interact with mobile search engine result pages and their fine-grained usefulness perception processes. In this paper, we collected fine-grained usefulness annotations for mobile search results in a user study dataset. With the user behavior information in the dataset, we investigate the patterns of users’ examination and click behavior and propose a user model for the fine-grained usefulness perception process in mobile search. Our research sheds light on improving user models in mobile search evaluation metrics and other mobile search-related applications.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
- 2.
- 3.
We will publicly release the collected usefulness judgments after the review.
References
Bailey, P., Moffat, A., Scholer, F., Thomas, P.: User variability and IR system evaluation. In: SIGIR 2015, pp. 625–634. ACM (2015)
Chapelle, O., Metlzer, D., Zhang, Y., Grinspan, P.: Expected reciprocal rank for graded relevance. In: CIKM 2009, pp. 621–630. ACM (2009)
Guo, Q., Jin, H., Lagun, D., Yuan, S., Agichtein, E.: Mining touch interaction data on mobile devices to predict web search result relevance. In: SIGIR 2013, pp. 153–162. ACM (2013)
Järvelin, K., Kekäläinen, J.: IR evaluation methods for retrieving highly relevant documents. In: SIGIR 2000, pp. 41–48. ACM (2000)
Kim, J., Thomas, P., Sankaranarayana, R., Gedeon, T., Yoon, H.J.: Eye-tracking analysis of user behavior and performance in web search on large and small screens. J. Assoc. Inf. Sci. Technol. 66(3), 526–544 (2015)
Kim, J., Thomas, P., Sankaranarayana, R., Gedon, T., Yoon, H.J.: Understanding eye movements on mobile devices for better presentation designs of search results. J. Am. Soc. Inf. Sci. Technol. (2015)
Lagun, D., Hsieh, C.H., Webster, D., Navalpakkam, V.: Towards better measurement of attention and satisfaction in mobile search. In: SIGIR 2014, pp. 113–122. ACM (2014)
Lagun, D., McMahon, D., Navalpakkam, V.: Understanding mobile searcher attention with rich ad formats. In: CIKM 2016, pp. 599–608. ACM (2016)
Luo, C., Liu, Y., Sakai, T., Zhang, F., Zhang, M., Ma, S.: Evaluating mobile search with height-biased gain. In: SIGIR 2017, pp. 435–444. ACM (2017)
Mao, J., Luo, C., Zhang, M., Ma, S.: Constructing click models for mobile search. In: SIGIR 2018, p. 775–784. ACM (2018)
Moffat, A., Thomas, P., Scholer, F.: Users versus models: What observation tells us about effectiveness metrics. In: CIKM 2013, pp. 659–668. ACM (2013)
Moffat, A., Zobel, J.: Rank-biased precision for measurement of retrieval effectiveness. ACM Trans. Inf. Syst. 27(1), 1–27 (2008)
Raptis, D., Tselios, N., Kjeldskov, J., Skov, M.B.: Does size matter?: Investigating the impact of mobile phone screen size on users’ perceived usability, effectiveness and efficiency. In: MobileHCI 2013, pp. 127–136. ACM (2013)
Song, Y., Ma, H., Wang, H., Wang, K.: Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance. In: WWW 2013, pp. 1201–1212. ACM (2013)
Wang, X., Su, N., He, Z., Liu, Y., Ma, S.: A large-scale study of mobile search examination behavior. In: SIGIR 2018, pp. 1129–1132. ACM (2018)
Wu, Z., Mao, J., Liu, Y., Zhang, M., Ma, S.: Investigating passage-level relevance and its role in document-level relevance judgment. In: SIGIR 2019, pp. 605–614. ACM (2019)
Zhang, F., Liu, Y., Li, X., Zhang, M., Xu, Y., Ma, S.: Evaluating web search with a bejeweled player model. In: SIGIR 2017, pp. 425–434. ACM (2017)
Zheng, Y., Mao, J., Liu, Y., Luo, C., Zhang, M., Ma, S.: Constructing click model for mobile search with viewport time. ACM Trans. Inf. Syst. 37(4), 1–34 (2019)
Zheng, Y., Mao, J., Liu, Y., Sanderson, M., Zhang, M., Ma, S.: Investigating examination behavior in mobile search. In: WSDM 2020. ACM (2020)
Acknowledgements
This work is supported by the National Key Research and Development Program of China (2018YFC0831700) and Natural Science Foundation of China (Grant No. 61622208, 61732008, 61532011).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Zheng, Y., Mao, J., Liu, Y., Xie, X., Zhang, M., Ma, S. (2020). Investigating Fine-Grained Usefulness Perception Process in Mobile Search. In: Dou, Z., Miao, Q., Lu, W., Mao, J., Jia, G. (eds) Information Retrieval. CCIR 2020. Lecture Notes in Computer Science(), vol 12285. Springer, Cham. https://doi.org/10.1007/978-3-030-56725-5_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-56725-5_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-56724-8
Online ISBN: 978-3-030-56725-5
eBook Packages: Computer ScienceComputer Science (R0)