Lei Han
Gianluca Demartini
ABSTRACT
Understanding user interaction behaviors remains a challenging problem. Quantifying behavior dynamics over time as users complete tasks has only been done in specific domains. In this paper, we present a user behavior model built using behavior embeddings to compare behaviors and their change over time. To this end, we first define the formal model and train the model using both action (e.g., copy/paste) embeddings and user interaction feature (e.g., length of the copied text) embeddings. Having obtained vector representations of user behaviors, we then define three measurements to model behavior dynamics over time, namely: behavior position, displacement, and velocity. To evaluate the proposed methodology, we use three real world datasets: (i) tens of users completing complex data curation tasks in a lab setting, (ii) hundreds of crowd workers completing structured tasks in a crowdsourcing setting, and (iii) thousands of editors completing unstructured editing tasks on Wikidata. Through these datasets, we show that the proposed methodology can: (i) surface behavioral differences among users; (ii) recognize relative behavioral changes; and (iii) discover directional deviations of user behaviors. Our approach can be used (i) to capture behavioral semantics from data in a consistent way, (ii) to quantify behavioral diversity for a task and among different users, and (iii) to explore the temporal behavior evolution with respect to various task properties (e.g., structure and difficulty).
Supplemental Material
- Mikhail Ageev, Qi Guo, Dmitry Lagun, and Eugene Agichtein. 2011. Find it if you can: a game for modeling different types of web search success using interaction data. In Proceedings of SIGIR. 345--354.Google Scholar
Digital Library
- Eugene Agichtein, Eric Brill, Susan Dumais, and Robert Ragno. 2006. Learning user interaction models for predicting web search result preferences. In Proceedings of SIGIR. 3--10.Google ScholarDigital Library
- Anne Aula, Rehan M Khan, and Zhiwei Guan. 2010. How does search behavior change as search becomes more difficult?. In Proceedings of CHI. 35--44.Google ScholarDigital Library
- Charles Chen, Sungchul Kim, Hung Bui, Ryan Rossi, Eunyee Koh, Branislav Kveton, and Razvan Bunescu. 2018. Predictive analysis by leveraging temporal user behavior and user embeddings. In Proceedings of CIKM. 2175--2182.Google ScholarDigital Library
- Edward Choi, Mohammad Taha Bahadori, Elizabeth Searles, Catherine Coffey, Michael Thompson, James Bost, Javier Tejedor-Sojo, and Jimeng Sun. 2016. Multi-layer representation learning for medical concepts. In KDD. 1495--1504.Google Scholar
- Nemanja Djuric, Vladan Radosavljevic, Mihajlo Grbovic, and Narayan Bhamidipati. 2014. Hidden conditional random fields with deep user embeddings for ad targeting. In ICDM. IEEE, 779--784.Google Scholar
- Steve Fox, Kuldeep Karnawat, Mark Mydland, Susan Dumais, and Thomas White. 2005. Evaluating implicit measures to improve web search. ACM Transactions on Information Systems (TOIS), Vol. 23, 2 (2005), 147--168.Google ScholarDigital Library
- Tanya Goyal, Tyler McDonnell, Mucahid Kutlu, Tamer Elsayed, and Matthew Lease. 2018. Your Behavior Signals Your Reliability: Modeling Crowd Behavioral Traces to Ensure Quality Relevance Annotations. In Proceedings of HComp.Google ScholarCross Ref
- Mihajlo Grbovic, Nemanja Djuric, Vladan Radosavljevic, Fabrizio Silvestri, and Narayan Bhamidipati. 2015a. Context-and content-aware embeddings for query rewriting in sponsored search. In Proceedings of SIGIR. 383--392.Google ScholarDigital Library
- Mihajlo Grbovic, Vladan Radosavljevic, Nemanja Djuric, Narayan Bhamidipati, Jaikit Savla, Varun Bhagwan, and Doug Sharp. 2015b. E-commerce in your inbox: Product recommendations at scale. In Proceedings of KDD. 1809--1818.Google ScholarDigital Library
- William L Hamilton, Jure Leskovec, and Dan Jurafsky. 2016. Diachronic Word Embeddings Reveal Statistical Laws of Semantic Change. In Proceedings of the 54th Annual Meeting of ACL. 1489--1501.Google ScholarCross Ref
- Lei Han, Tianwa Chen, Gianluca Demartini, Marta Indulska, and Shazia Sadiq. 2020 a. On Understanding Data Worker Interaction Behaviors. In Proceedings of SIGIR. ACM, 269--278.Google ScholarDigital Library
- Lei Han, Eddy Maddalena, Alessandro Checco, Cristina Sarasua, Ujwal Gadiraju, Kevin Roitero, and Gianluca Demartini. 2020 b. Crowd worker strategies in relevance judgment tasks. In Proceedings of WSDM. ACM, 241--249.Google ScholarDigital Library
- Lei Han, Kevin Roitero, Ujwal Gadiraju, Cristina Sarasua, Alessandro Checco, Eddy Maddalena, and Gianluca Demartini. 2019. The impact of task abandonment in crowdsourcing. IEEE Transactions on Knowledge and Data Engineering (2019).Google ScholarCross Ref
- Gabriella Kazai and Imed Zitouni. 2016. Quality management in crowdsourcing using gold judges behavior. In Proceedings of WSDM. ACM, 267--276.Google ScholarDigital Library
- Youngho Kim, Ahmed Hassan, Ryen W White, and Imed Zitouni. 2014. Modeling dwell time to predict click-level satisfaction. In Proceedings of WSDM. 193--202.Google ScholarDigital Library
- Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems. 1097--1105.Google Scholar
- Vivek Kulkarni, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of WWW. 625--635.Google ScholarDigital Library
- Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International Conference on Machine Learning. 1188--1196.Google ScholarDigital Library
- Jingjing Liu, Chang Liu, Michael Cole, Nicholas J Belkin, and Xiangmin Zhang. 2012. Exploring and predicting search task difficulty. In Proceedings of CIKM. ACM, 1313--1322.Google ScholarDigital Library
- Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics (1947), 50--60.Google Scholar
- Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems. 3111--3119.Google Scholar
- Alistair Moffat, Paul Thomas, and Falk Scholer. 2013. Users versus models: What observation tells us about effectiveness metrics. In Proceedings of CIKM. 659--668.Google ScholarDigital Library
- Ricky KP Mok, Rocky KC Chang, and Weichao Li. 2017. Detecting low-quality workers in QoE crowdtesting: A worker behavior-based approach. IEEE Transactions on Multimedia, Vol. 19, 3 (2017), 530--543.Google ScholarDigital Library
- Eleanor O'Rourke, Kyla Haimovitz, Christy Ballweber, Carol Dweck, and Zoran Popović. 2014. Brain points: a growth mindset incentive structure boosts persistence in an educational game. In Proceedings of CHI. 3339--3348.Google ScholarDigital Library
- Kira Radinsky, Krysta Svore, Susan Dumais, Jaime Teevan, Alex Bocharov, and Eric Horvitz. 2012. Modeling and predicting behavioral dynamics on the web. In Proceedings of WWW. 599--608.Google ScholarDigital Library
- Al M Rashid, Kimberly Ling, Regina D Tassone, Paul Resnick, Robert Kraut, and John Riedl. 2006. Motivating participation by displaying the value of contribution. In Proceedings of CHI. 955--958.Google ScholarDigital Library
- Cristina Sarasua, Alessandro Checco, Gianluca Demartini, Djellel Difallah, Michael Feldman, and Lydia Pintscher. 2019. The evolution of power and standard Wikidata editors: comparing editing behavior over time to predict lifespan and volume of edits. CSCW, Vol. 28, 5 (2019), 843--882.Google Scholar
- Michael E Tipping and Christopher M Bishop. 1999. Probabilistic principal component analysis. Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 61, 3 (1999), 611--622.Google ScholarCross Ref
- Thanh Tran, Kyumin Lee, Yiming Liao, and Dongwon Lee. 2018. Regularizing matrix factorization with user and item embeddings for recommendation. In Proceedings of CIKM. 687--696.Google ScholarDigital Library
- Ryen W White, Paul N Bennett, and Susan T Dumais. 2010. Predicting short-term interests using activity-based search context. In Proceedings of CIKM. 1009--1018.Google ScholarDigital Library
- Ryen W White and Steven M Drucker. 2007. Investigating behavioral variability in web search. In Proceedings of WWW. 21--30.Google ScholarDigital Library
Index Terms
- Modelling User Behavior Dynamics with Embeddings
Recommendations
Predictive Analysis by Leveraging Temporal User Behavior and User Embeddings
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge ManagementThe rapid growth of mobile devices has resulted in the generation of a large number of user behavior logs that contain latent intentions and user interests. However, exploiting such data in real-world applications is still difficult for service ...
Modeling Online User Behavior
ICEBE '08: Proceedings of the 2008 IEEE International Conference on e-Business EngineeringA framework for online user behavior soft modeling is presented in this work. Behavior models of users in dynamic virtual environments has been described in the literature in terms of timed transition automata which can be compiled in a planning domain. ...
Accounting for patterns of collective behavior in crowd locomotor dynamics for realistic simulations
Transactions on Edutainment VIIDo people in a crowd behave like a set of isolated individuals or like a cohesive group? Studies of crowd modeling usually consider pedestrian behavior either from the point of view of an isolated individual or from that of large swarms. We introduce ...
Comments