Abstract
Relevance judgments are central to Information retrieval evaluation. With increasing number of hand held devices at users disposal today, and continuous improvement in web standards and browsers, it has become essential to evaluate whether such devices and dynamic page layouts affect users notion of relevance. Given dynamic web pages and content rendering, we know little about what kind of pages are relevant on devices other than desktop. With this work, we take the first step in characterizing relevance on mobiles and desktop. We collect crowd sourced judgments on mobile and desktop to systematically determine whether screen size of a device and page layouts impact judgments. Our study shows that there are certain difference between mobile and desktop judgments. We also observe different judging times, despite similar inter-rater agreement on both devices. Finally, we also propose and evaluate display and viewport specific features to predict relevance. Our results indicate that viewport based features can be used to reliably predict mobile relevance.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
- 3.
Viewport is the framed area on a display screen of mobile or desktop for viewing information.
- 4.
AMT (https://requester.mturk.com/) is a crowd sourcing marketplace to conduct experiments by recruiting multiple participants in exchange for compensation.
- 5.
rel = high-rel+rel, non-rel=some-rel+non-rel.
- 6.
References
Borlund, P.: The concept of relevance in IR. J. Am. Soc. Inf. Sci. Technol. 54(10), 913–925 (2003)
Buchanan, G., Farrant, S., Jones, M., Thimbleby, H., Marsden, G., Pazzani, M.: Improving mobile internet usability. In: Proceedings of WWW. ACM (2001)
Church, K., Oliver, N.: Understanding mobile web and mobile search usein today’s dynamic mobile landscape. In: Proceedings of MobileHCI. ACM (2011)
Church, K. Smyth, B.: Understanding the intent behind mobile information needs. In: Proceedings of IUI. ACM (2009)
Church, K., Smyth, B., Bradley, K., Cotter, P.: A large scale studyof european mobile search behaviour. In: Proceedings MobileHCI. ACM (2008)
Church, K., Smyth, B., Cotter, P., Bradley, K.: Mobile informationaccess: a study of emerging search behavior on the mobile internet. ACM Trans. Web 1(1), 4 (2007)
Guo, Q., Jin, H., Lagun, D., Yuan, S., Agichtein, E.: Miningtouch interaction data on mobile devices to predict web search result relevance. In: Proceedings of SIGIR. ACM (2013)
Kamvar, M., Baluja, S.: A large scale study of wireless searchbehavior: Google mobile search. In: Proceedings SIGCHI. ACM (2006)
Kamvar, M., Baluja, S.: Deciphering trends in mobile search. Computer 40(8), 58–62 (2007)
Kamvar, M., Kellar, M., Patel, R., Xu, Y.: Computers and iphonesand mobile phones, oh my!: a logs-based comparison of search users on differentdevices. In: Proceedings of WWW. ACM (2009)
Kazai, G., Kamps, J., Milic-Frayling, N.: An analysis of humanfactors and label accuracy in crowdsourcing relevance judgments. Inf. Retr. (2013)
Li, J., Huffman, S., Tokuda, A.: Good abandonment in mobile andpc internet search. In: Proceedings of SIGIR. ACM (2009)
Schamber, L., Eisenberg, M.: Relevance: The search for a definition (1988)
Shokouhi, M., Jones, R., Ozertem, U., Raghunathan, K., Diaz, F.: Mobile query reformulations. In: Proceedings SIGIR. ACM (2014)
Song, Y., Ma, H., Wang, H., Wang, K.: Exploring and exploiting user search behavior on mobile and tablet devices to improve search relevance. In: Proceedings of WWW (2013)
Tombros, A., Ruthven, I., Jose, J.M.: How users assess web pages for information seeking. J. Am. Soc. Inf. Sci. Technol. 56(4), 327–344 (2005)
Tossell, C., Kortum, P., Rahmati, A., Shepard, C., Zhong, L.: Characterizing web use on smartphones. In: Proceedings of the SIGCHI. ACM (2012)
Xu, Y.C., Chen, Z.: Relevance judgment: What do informationusers consider beyond topicality? JASIST 57(7), 961–973 (2006)
Yi, J., Maghoul, F., Pedersen, J.: Deciphering mobile search pat-terns: a study of yahoo! mobile search queries. In: Proceedings of the WWW. ACM (2008)
Zhang, Y., Zhang, J., Lease, M., Gwizdka, J.: Multidimensional relevance modeling via psychometrics and crowdsourcing. In: Proceedings of the SIGIR. ACM (2014)
Zhu, J., Zou, H., Rosset, S., Hastie, T.: Multi-class adaboost. Stat. Interface 2(3), 349–360 (2009)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Verma, M., Yilmaz, E. (2016). Characterizing Relevance on Mobile and Desktop. In: Ferro, N., et al. Advances in Information Retrieval. ECIR 2016. Lecture Notes in Computer Science(), vol 9626. Springer, Cham. https://doi.org/10.1007/978-3-319-30671-1_16
Download citation
DOI: https://doi.org/10.1007/978-3-319-30671-1_16
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-30670-4
Online ISBN: 978-3-319-30671-1
eBook Packages: Computer ScienceComputer Science (R0)