skip to main content
10.1145/3404835.3462793acmconferencesArticle/Chapter ViewAbstractPublication PagesirConference Proceedingsconference-collections
short-paper

Privacy-Aware Remote Information Retrieval User Experiments Logging Tool

Published: 11 July 2021 Publication History

Abstract

User behaviors and experiences are the fundamental parts of information retrieval systems, but are often difficult to collect, bringing challenges to both applications and research. Recently, researchers have been exploring more fine-grained user behavior than simple clicks, such as time patterns, mouse/scroll patterns, etc., with their own specific laboratory experimental platforms. However, the lack of public available toolkits for logging user behaviors and experiences leads to difficulties on field study of remote user experiments in real scenarios. In this work, we propose a Privacy-Aware Remote User Logging Tool for remotely collecting user behaviors and explicit experience feedback, with a special care for user privacy. With this tool, participants can conduct the user experiments remotely without time and location constraints, giving researchers the possibility to observe users' more natural behaviors and experiences.

Supplementary Material

MP4 File (SIGIR21-fp1912.mp4)
RUS-toolkit presentation video

References

[1]
Eugene Agichtein, Eric Brill, and Susan Dumais. 2006. Improving web search ranking by incorporating user behavior information. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 19--26.
[2]
Antti Ajanki, David R Hardoon, Samuel Kaski, Kai Puolam"aki, and John Shawe-Taylor. 2009. Can eyes reveal interest? Implicit queries from gaze patterns. User Modeling and User-Adapted Interaction, Vol. 19, 4 (2009), 307--339.
[3]
Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In Proceedings of the 18th international conference on World wide web. ACM, 1--10.
[4]
Georges E Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 331--338.
[5]
Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2017. Accurately interpreting clickthrough data as implicit feedback. In ACM SIGIR Forum, Vol. 51. Acm, 4--11.
[6]
Diane Kelly and Jaime Teevan. 2003. Implicit feedback for inferring user preference: a bibliography. In Acm Sigir Forum, Vol. 37. ACM, 18--28.
[7]
Dmitry Lagun, Chih-Hung Hsieh, Dale Webster, and Vidhya Navalpakkam. 2014. Towards better measurement of attention and satisfaction in mobile search. In Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval. ACM, 113--122.
[8]
Yixuan Li, Pingmei Xu, Dmitry Lagun, and Vidhya Navalpakkam. 2017. Towards Measuring and Inferring User Interest from Gaze. In Proceedings of the 26th International Conference on World Wide Web Companion. International World Wide Web Conferences Steering Committee, 525--533.
[9]
Mengyang Liu, Yiqun Liu, J. Mao, Cheng Luo, Min Zhang, and S. Ma. 2018. "Satisfaction with Failure" or "Unsatisfied Success": Investigating the Relationship between Search Success and User Satisfaction. Proceedings of the 2018 World Wide Web Conference (2018).
[10]
Zeyang Liu, Jiaxin Mao, Chao Wang, Qingyao Ai, Yiqun Liu, and Jian-Yun Nie. 2017. Enhancing click models with mouse movement information. Information Retrieval Journal, Vol. 20, 1 (2017), 53--80.
[11]
Jiaxin Mao, Yiqun Liu, N. Kando, Min Zhang, and S. Ma. 2018. How Does Domain Expertise Affect Users' Search Interaction and Outcome in Exploratory Search? ACM Transactions on Information Systems (TOIS), Vol. 36 (2018), 1 -- 30.
[12]
F. Zhang, J. Mao, Yiqun Liu, W. Ma, M. Zhang, and Shaoping Ma. 2020. Cascade or Recency: Constructing Better Evaluation Metrics for Session Search. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (2020).
[13]
Yukun Zheng, J. Mao, Yiqun Liu, M. Sanderson, Min Zhang, and S. Ma. 2020. Investigating Examination Behavior in Mobile Search. Proceedings of the 13th International Conference on Web Search and Data Mining (2020).

Cited By

View all
  • (2021)Competitive Caching with Machine Learned AdviceJournal of the ACM10.1145/344757968:4(1-25)Online publication date: 14-Jul-2021
  • (2021)A Comprehensive Analysis of Privacy Protection Techniques Developed for COVID-19 PandemicIEEE Access10.1109/ACCESS.2021.31306109(164159-164187)Online publication date: 2021

Index Terms

  1. Privacy-Aware Remote Information Retrieval User Experiments Logging Tool

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    SIGIR '21: Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval
    July 2021
    2998 pages
    ISBN:9781450380379
    DOI:10.1145/3404835
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 11 July 2021

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. experimental toolkit
    2. privacy protection
    3. user behavior logging

    Qualifiers

    • Short-paper

    Funding Sources

    • Natural Science Foundation of China
    • Tsinghua University Guoqiang Research Institute
    • China Postdoctoral Science Foundation
    • Shuimu Tsinghua ScholarProgram
    • National KeyResearch and Development Program of China

    Conference

    SIGIR '21
    Sponsor:

    Acceptance Rates

    Overall Acceptance Rate 792 of 3,983 submissions, 20%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)19
    • Downloads (Last 6 weeks)3
    Reflects downloads up to 27 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2021)Competitive Caching with Machine Learned AdviceJournal of the ACM10.1145/344757968:4(1-25)Online publication date: 14-Jul-2021
    • (2021)A Comprehensive Analysis of Privacy Protection Techniques Developed for COVID-19 PandemicIEEE Access10.1109/ACCESS.2021.31306109(164159-164187)Online publication date: 2021

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media