Tiffany C.K. Kwok
Eugene Yujun Fu
Grace Ngai
ABSTRACT
User experience could be enhanced if the computer could understand human interaction intention. For instance, it could react to intercept and prevent interaction errors. This paper presents an approach to predicting users intention in interaction tasks based on past mouse movements. We adopt a long short-term memory (LSTM) model to predict the users» intention via their next mouse click interaction, upon being trained with past mouse interaction behaviors. To evaluate, we consider two scenarios in daily computer usage: a more structured crowdsourcing annotation task and a more free-form, open-ended web search task. Our results indicate that we could predict the next interaction event with reasonable accuracy. We also conducted a pilot study to investigate the possibility of applying our model for non-intentional mouse click detection. We believe that our findings would be beneficial towards the development of better intelligent agents.
- Brown, E.T., Ottley, A., Zhao, H., et al.Finding Waldo: Learning about Users from their Interactions. IEEE Transactions on Visualization and Computer Graphics 20, 12 (2014), 1663--1672.Google Scholar
Cross Ref
- Chang, C., Amick, B.C., Menendez, C.C., et al.Daily Computer Usage Correlated with Undergraduate Students? Musculoskeletal Symptoms. American Journal of Industrial Medicine 50, 6 (2007), 481--488.Google Scholar
- Diaz, F., Guo, Q., and White, R.W.Search Result Prefetching Using Cursor Movement. Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval - SIGIR '16, (2016), 609--618. Google ScholarDigital Library
- Elkahky, A.M., Song, Y., and He, X.A Multi-View Deep Learning Approach for Cross Domain User Modeling in Recommendation Systems. WWW '15 Proceedings of the 24th International Conference on World Wide Web, (2015), 278--288. Google ScholarDigital Library
- Fu, Y., Kwok, T.C.K., Wu, Y.E., Leong, H.V., and Ngai, G.Your Mouse Revals Your Next Activity?: Towards Predicting User Intention from Mouse Interaction. COMPSAC '17 IEEE Computer Software and Applications Conference.Google Scholar
- Georgescu, M. and Zhu, X.Aggregation of Crowdsourced Labels Based on Worker History. WIMS '14 Proceedings of the 4th International Conference on Web Intelligence, Mining and Semantics (WIMS14), (2014). Google ScholarDigital Library
- Gers, F.A., Schmidhuber, J.A., and Cummins, F.A.Learning to Forget: Continual Prediction with LSTM. Neural Computation 12, 10 (2000), 2451--2471. Google ScholarDigital Library
- Google.A google a day. http://www.agoogleaday.com/.Google Scholar
- Gould, S.J.J., Cox, A.L., Brumby, D.P., and Wickersham, A.Now Check Your Input: Brief Task Lockouts Encourage Checking, Longer Lockouts Encourage Task Switching. CHI '16 Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, (2016), 3311--3323. Google ScholarDigital Library
- Guo, Q. and Agichtein, E.Ready to Buy or Just Browsing?: Detecting Web Searcher Goals from Interaction Data. SIGIR '10 Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, (2010), 130--137. Google ScholarDigital Library
- Hochreiter, S. and Schmidhuber, J.Long Short-Term Memory. Neural Computation 9, 8 (1997), 1735--1780. Google ScholarDigital Library
- Huang, M.X., Li, J., Ngai, G., and Leong, H.V.StressClick. Proceedings of the 2016 ACM on Multimedia Conference - MM '16, (2016), 1395--1404. Google ScholarDigital Library
- Lalle, S., Toker, D., Conati, C., and Carenini, G.Prediction of Users? Learning Curves for Adaptation while Using an Information Visualization. IUI '15 Proceedings of the 20th International Conference on Intelligent User Interfaces, ACM (2015), 357--368. Google ScholarDigital Library
- Maniu, S., O'Hare, N., Aiello, L.M., Chiarandini, L., and Jaimes, A.Search Behaviour on Photo Sharing Platforms. Proceedings of IEEE International Conference on Multimedia and Expo (ICME), (2013), 1--6.Google ScholarCross Ref
- Mavadati, S.M., Mahoor, M.H., Bartlett, K., Trinh, P., and Cohn, J.F.DISFA: A Spontaneous Facial Action Intensity Database. IEEE Transactions On Affective Computing 4, 2 (2013). Google ScholarDigital Library
- Pasqual, P.T. and Wobbrock, J.O.Mouse Pointing Endpoint Prediction Using Kinematic Template Matching. CHI '14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (2014), 743--752. Google ScholarDigital Library
- Pilankar, P.S. and Padiya, P.Multi-Phase Mouse Dynamics Authentication System using Behavioural Biometrics. International Conference on Signal Processing, Communication, Power and Embedded System (SCOPES), 2016, (2016), 1947--1950.Google ScholarCross Ref
- Ratwani, R.M., McCurry, J.M., and Trafton, J.G.Predicting Postcompletion Errors Using Eye Movements. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (2008), 539--542. Google ScholarDigital Library
- Schwarz, J., Marais, C.C., Leyvand, T., Hudson, S.E., and Mankoff, J.Combining Body Pose, Gaze, and Gesture to Determine Intention to Interact in Vision-Based Interfaces. CHI '14 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, (2014), 3443--3452. Google ScholarDigital Library
- Ziebart, B., Dey, A., and Bagnell, J.A.Probabilistic Pointing Target Prediction via Inverse Optimal Control. IUI '12 Proceedings of the 2012 ACM international conference on Intelligent User Interfaces, (2012), 1--10. Google ScholarDigital Library
Index Terms
- Every Little Movement Has a Meaning of Its Own: Using Past Mouse Movements to Predict the Next Interaction
Recommendations
Predicting Mouse Click Position Using Long Short-Term Memory Model Trained by Joint Loss Function
CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing SystemsKnowing where users might click in advance can potentially improve the efficiency of user interaction in desktop user interfaces. In this paper, we propose a machine learning approach to predict mouse click location. Our model, which is LSTM (long short-...
Drag-and-drop versus point-and-click mouse interaction styles for children
This research investigates children's use of two common mouse interaction styles, drag-and-drop and point-and-click, to determine whether the choice of interaction style impacts children's performance in interactive learning environments. The ...
Educational software for children: analysis of interaction techniques for direct manipulation
IHM '09: Proceedings of the 21st International Conference on Association Francophone d'Interaction Homme-MachineThis paper is concerned with the investigationof children's use of direct manipulation mouse interaction styles when using educational software. The results of an experiment comparing two interaction styles (point-and-click and point-and-select) are ...
Comments