Julian Steil
Andreas Bulling
ABSTRACT
Human visual behaviour has significant potential for activity recognition and computational behaviour analysis, but previous works focused on supervised methods and recognition of predefined activity classes based on short-term eye movement recordings. We propose a fully unsupervised method to discover users' everyday activities from their long-term visual behaviour. Our method combines a bag-of-words representation of visual behaviour that encodes saccades, fixations, and blinks with a latent Dirichlet allocation (LDA) topic model. We further propose different methods to encode saccades for their use in the topic model. We evaluate our method on a novel long-term gaze dataset that contains full-day recordings of natural visual behaviour of 10 participants (more than 80 hours in total). We also provide annotations for eight sample activity classes (outdoor, social interaction, focused work, travel, reading, computer work, watching media, eating) and periods with no specific activity. We show the ability of our method to discover these activities with performance competitive with that of previously published supervised methods.
- Barger, T. S., Brown, D. E., and Alwan, M. Health-status monitoring through analysis of behavioral patterns. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 35, 1 (2005), 22--27. Google Scholar
Digital Library
- Begole, J. B., Tang, J. C., and Hill, R. Rhythm modeling, visualizations and applications. In Proc. UIST (2003), 11--20. Google ScholarDigital Library
- Blei, D. M., Ng, A. Y., and Jordan, M. I. Latent dirichlet allocation. The Journal of Machine Learning Research 3 (2003), 993--1022. Google ScholarDigital Library
- Bulling, A., and Roggen, D. Recognition of Visual Memory Recall Processes Using Eye Movement Analysis. In Proc. UbiComp (2011), 455--464. Google ScholarDigital Library
- Bulling, A., Ward, J. A., and Gellersen, H. Multimodal Recognition of Reading Activity in Transit Using Body-Worn Sensors. ACM Transactions on Applied Perception 9, 1 (2012), 2:1--2:21. Google ScholarDigital Library
- Bulling, A., Ward, J. A., Gellersen, H., and Tröster, G. Robust recognition of reading activity in transit using wearable electrooculography. In Proc. Pervasive 2008 (2008), 19--37. Google ScholarDigital Library
- Bulling, A., Ward, J. A., Gellersen, H., and Tröster, G. Eye Movement Analysis for Activity Recognition Using Electrooculography. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 4 (2011), 741--753. Google ScholarDigital Library
- Bulling, A., Weichel, C., and Gellersen, H. EyeContext: Recognition of High-level Contextual Cues from Human Visual Behaviour. In Proc. CHI 2013 (2013), 305--308. Google ScholarDigital Library
- Chen, S., Epps, J., and Chen, F. Automatic and continuous user task analysis via eye activity. In Proc. IUI (2013), 57--66. Google ScholarDigital Library
- Chuk, T., Chan, A. B., and Hsiao, J. H. Understanding eye movements in face recognition using hidden markov models. Journal of Vision 14, 11 (2014).Google ScholarCross Ref
- Csurka, G., Dance, C., Fan, L., Willamowski, J., and Bray, C. Visual categorization with bags of keypoints. In Proc. ECCVW, vol. 1 (2004), 1--2.Google Scholar
- Dempere-Marco, L., Hu, X.-P., MacDonald, S. L., Ellis, S. M., Hansell, D. M., and Yang, G.-Z. The use of visual search for knowledge gathering in image decision support. IEEE Transactions on Medical Imaging 21, 7 (2002), 741--754.Google ScholarCross Ref
- Elhelw, M., Nicolaou, M., Chung, A., Yang, G.-Z., and Atkins, M. S. A gaze-based study for investigating the perception of visual realism in simulated scenes. ACM Transactions on Applied Perception 5, 1 (2008), 3. Google ScholarDigital Library
- Farrahi, K., and Gatica-Perez, D. What did you do today?: Discovering daily routines from large-scale mobile data. In Proc. MM (2008), 849--852. Google ScholarDigital Library
- Gu, T., Chen, S., Tao, X., and Lu, J. An unsupervised approach to activity recognition and segmentation based on object-use fingerprints. Data & Knowledge Engineering 69, 6 (2010), 533--544. Google ScholarDigital Library
- Hofmann, T. Unsupervised learning by probabilistic latent semantic analysis. Machine learning 42, 1--2 (2001), 177--196. Google ScholarDigital Library
- Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., and Van de Weijer, J. Eye tracking: A comprehensive guide to methods and measures. Oxford University Press, 2011.Google Scholar
- Hoppe, S., Morey, S., Loetscher, T., and Bulling, A. Recognition of curiosity using eye movement analysis. In Adj. Proc. UbiComp (2015). Google ScholarDigital Library
- Huynh, T., Fritz, M., and Schiele, B. Discovery of activity patterns using topic models. In Proc. UbiComp (2008), 10--19. Google ScholarDigital Library
- Ishiguro, Y., Mujibiya, A., Miyaki, T., and Rekimoto, J. Aided eyes: eye activity sensing for daily life. In Proc. AH (2010), 25. Google ScholarDigital Library
- Ishimaru, S., Weppner, J., Kunze, K., Kise, K., Dengel, A., Lukowicz, P., and Bulling, A. In the Blink of an Eye - Combining Head Motion and Eye Blink Frequency for Activity Recognition with Google Glass. In Proc. AH (2014). Google ScholarDigital Library
- Just, M. A., and Carpenter, P. A. Eye fixations and cognitive processes. Cognitive psychology 8, 4 (1976), 441--480.Google Scholar
- Kassner, M., Patera, W., and Bulling, A. Pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. In Adj. Proc. UbiComp (2014), 1151--1160. Google ScholarDigital Library
- Kunze, K., Bulling, A., Utsumi, Y., Yuki, S., and Kise, K. I know what you are reading -- recognition of document types using mobile eye tracking. In Proc. ISWC (2013), 113--116. Google ScholarDigital Library
- Kunze, K., Kawaichi, H., Yoshimura, K., and Kise, K. Towards inferring language expertise using eye tracking. In Ext. Abstr. CHI (2013), 217--222. Google ScholarDigital Library
- Kunze, K., Kawaichi, H., Yoshimura, K., and Kise, K. The wordometer--estimating the number of words read using document image retrieval and mobile eye tracking. In Proc. ICDAR (2013), 25--29. Google ScholarDigital Library
- Marshall, S. P. The index of cognitive activity: Measuring cognitive workload. In Proc. Human factors and power plants (2002), 7--5.Google ScholarCross Ref
- Niebles, J. C., Wang, H., and Fei-Fei, L. Unsupervised learning of human action categories using spatial-temporal words. International Journal of Computer Vision 79, 3 (2008), 299--318. Google ScholarDigital Library
- Palinko, O., Kun, A. L., Shyrokov, A., and Heeman, P. Estimating cognitive load using remote eye tracking in a driving simulator. In Proc. ETRA (2010), 141--144. Google ScholarDigital Library
- Salvucci, D. D., and Anderson, J. R. Automated eye-movement protocol analysis. Human-Computer Interaction 16, 1 (2001), 39--86. Google ScholarDigital Library
- Salvucci, D. D., and Goldberg, J. H. Identifying fixations and saccades in eye-tracking protocols. In Proc. ETRA (2000), 71--78. Google ScholarDigital Library
- Schleicher, R., Galley, N., Briest, S., and Galley, L. Blinks and saccades as indicators of fatigue in sleepiness warnings: looking tired? Ergonomics 51, 7 (July 2008), 982--1010.Google ScholarCross Ref
- Seiter, J., Amft, O., Rossi, M., and Tröster, G. Discovery of activity composites using topic models: An analysis of unsupervised methods. Pervasive and Mobile Computing 15 (2014), 215--227.Google ScholarDigital Library
- Shiga, Y., Toyama, T., Utsumi, Y., Kise, K., and Dengel, A. Daily Activity Recognition Combining Gaze Motion and Visual Features. In Adj. Proc. UbiComp (2014), 1103--1111. Google ScholarDigital Library
- Steichen, B., Carenini, G., and Conati, C. User-adaptive information visualization: using eye gaze data to infer visualization tasks and user cognitive abilities. In Proc. IUI (2013), 317--328. Google ScholarDigital Library
- Steyvers, M., and Griffiths, T. Probabilistic topic models. Handbook of latent semantic analysis 427, 7 (2007), 424--440.Google Scholar
- Tessendorf, B., Bulling, A., Roggen, D., Stiefmeier, T., Feilner, M., Derleth, P., and Tröster, G. Recognition of Hearing Needs From Body and Eye Movements to Improve Hearing Instruments. In Proc. Pervasive (2011), 314--331. Google ScholarDigital Library
- Vidal, M., Turner, J., Bulling, A., and Gellersen, H. Wearable Eye Tracking for Mental Health Monitoring. Computer Communications 35, 11 (2012), 1306--1311. Google ScholarDigital Library
Index Terms
- Discovery of everyday human activities from long-term visual behaviour using topic models
Recommendations
Weakly Supervised Joint Sentiment-Topic Detection from Text
Sentiment analysis or opinion mining aims to use automated tools to detect subjective information such as opinions, attitudes, and feelings expressed in text. This paper proposes a novel probabilistic modeling framework called joint sentiment-topic (JST)...
Topic sentiment mixture: modeling facets and opinions in weblogs
WWW '07: Proceedings of the 16th international conference on World Wide WebIn this paper, we define the problem of topic-sentiment analysis on Weblogs and propose a novel probabilistic model to capture the mixture of topics and sentiments simultaneously. The proposed Topic-Sentiment Mixture (TSM) model can reveal the latent ...
Latent topic model-based group activity discovery
Special Issue on ICVGIP 2010Surveillance videos of public places often consist of group activities composed from multiple co-occurring individual activities. However, latent topic models, such as Latent Dirichlet Allocation (LDA), which have been successfully used to discover ...
Comments