Abstract
Assuming that eye tracking will be a common input device in the near future in notebooks and mobile devices like iPads, it is possible to implicitly gain information about images and image regions from these users’ gaze movements. In this paper, we investigate the principle idea of finding specific objects shown in images by looking at the users’ gaze path information only. We have analyzed 547 gaze paths from 20 subjects viewing different image-tag-pairs with the task to decide if the tag presented is actually found in the image or not. By analyzing the gaze paths, we are able to correctly identify 67% of the image regions and significantly outperform two baselines. In addition, we have investigated if different regions of the same image can be differentiated by the gaze information. Here, we are able to correctly identify two different regions in the same image with an accuracy of 38%.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Castagnos, S., Jones, N., Pu, P.: Eye-tracking product recommenders’ usage. In: Proceedings of the Fourth ACM Conference on Recommender Systems, pp. 29–36. ACM (2010)
Hajimirza, S.N., Izquierdo, E.: Gaze movement inference for implicit image annotation. In: Image Analysis for Multimedia Interactive Services. IEEE (2010)
Jaimes, A.: Using human observer eye movements in automatic image classifiers. In: SPIE (2001)
Klami, A.: Inferring task-relevant image regions from gaze data. In: Workshop on Machine Learning for Signal Processing. IEEE (2010)
Klami, A., Saunders, C., De Campos, T.E., Kaski, S.: Can relevance of images be inferred from eye movements? In: Multimedia Information Retrieval, ACM (2008)
Kozma, L., Klami, A., Kaski, S.: GaZIR: gaze-based zooming interface for image retrieval. In: Multimodal Interfaces. ACM (2009)
Ramanathan, S., Katti, H., Huang, R., Chua, T.-S., Kankanhalli, M.: Automated localization of affective objects and actions in images via caption text-cum-eye gaze analysis. In: Multimedia (2009)
Rowe, N.C.: Finding and labeling the subject of a captioned depictive natural photograph. IEEE Transactions on Knowledge and Data Engineering, 202–207 (2002)
Russell, B.C., Torralba, A., Murphy, K.P., Freeman, W.T.: LabelMe: a database and web-based tool for image annotation. Journal of Computer Vision 77(1), 157–173 (2008)
Santella, A., Agrawala, M., DeCarlo, D., Salesin, D., Cohen, M.: Gaze-based interaction for semi-automatic photo cropping. In: CHI, p. 780. ACM (2006)
Shimojo, S., Simion, C., Shimojo, E., Scheier, C.: Gaze bias both reflects and influences preference. Nature Neuroscience 6(12), 1317–1322 (2003)
von Ahn, L., Liu, R., Blum, M.: Peekaboom: a game for locating objects in images. In: CHI. ACM (2006)
Walber, T., Scherp, A., Staab, S.: Towards improving the understanding of image semantics by gaze-based tag-to-region assignments. Technical Report 08/2011, Institut WeST, Universität Koblenz-Landau (2011), http://www.uni-koblenz.de/~fb4reports/2011/2011_08_Arbeitsberichte.pdf
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2012 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Walber, T., Scherp, A., Staab, S. (2012). Identifying Objects in Images from Analyzing the Users’ Gaze Movements for Provided Tags. In: Schoeffmann, K., Merialdo, B., Hauptmann, A.G., Ngo, CW., Andreopoulos, Y., Breiteneder, C. (eds) Advances in Multimedia Modeling. MMM 2012. Lecture Notes in Computer Science, vol 7131. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-27355-1_15
Download citation
DOI: https://doi.org/10.1007/978-3-642-27355-1_15
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-642-27354-4
Online ISBN: 978-3-642-27355-1
eBook Packages: Computer ScienceComputer Science (R0)