Abstract
Recently, image sentiment analysis has become more and more attractive to many researchers due to an increasing number of applications developed to understand images e.g. image retrieval systems and social networks. Many studies aim to improve the performance of the classifier by many approaches. This work aims to predict the emotional response of a person who is exposed to images. The prediction model makes use of eye movement data captured while users are looking at images to enhance the prediction performance. An image can stimulate different emotions in different users depending on where and how their eyes move on the image. Two image datasets were used, i.e. abstract images and images with context information, by using leave-one-user-out and leave-one-image-out cross-validation techniques. It was found that eye movement data is useful and able to improve the prediction performance only in leave-one-image-out cross-validation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Auer, P., Hussain, Z., Kaski, S., Klami, A., Kujala, J., Laaksonen, J., Leung, A.P., Pasupa, K., Shawe-Taylor, J.: Pinview: implicit feedback in content-based image retrieval. In: Proceeding of the Workshop on Applications of Pattern Analysis, WAPA 2010, Windsor, UK, pp. 51–57 (2010)
Avital, T.: Art versus Nonart: Art Out of Mind. Cambridge University Press Cambridge, UK (2003)
Chandon, P., Hutchinson, J., Bradlow, E., Young, S.H.: Measuring the value of point-of-purchase marketing with commercial eye-tracking data. INSEAD Working Paper Collection, vol. 22, pp. 1 (2007)
Duchowski, A.T.: Eye Tracking Methodology: Theory and Practice, no. 328. Springer, Cham (2017). doi:10.1007/978-3-319-57883-5
Feist, G.J., Brady, T.R.: Openness to experience, non-conformity, and the preference for abstract art. Empirical Stud. Arts 22(1), 77–89 (2004)
Hardoon, D.R., Pasupa, K., Shawe-Taylor, J.: Image ranking with implicit feedback from eye movements. In: Proceeding of the 6th Biennial Symposium on Eye Tracking Research and Applications, ETRA 2010, Austin, USA, pp. 291–298 (2010)
Hussain, Z., Leung, A.P., Pasupa, K., Hardoon, D.R., Auer, P., Shawe-Taylor, J.: Exploration-exploitation of eye movement enriched multiple feature spaces for content-based image retrieval. In: Balcázar, J.L., Bonchi, F., Gionis, A., Sebag, M. (eds.) ECML PKDD 2010. LNCS, vol. 6321, pp. 554–569. Springer, Heidelberg (2010). doi:10.1007/978-3-642-15880-3_41
Müller, H., Michoux, N., Bandon, D., Geissbuhler, A.: A review of content-based image retrieval systems in medical applications-clinical benefits and future directions. Int. J. Med. Inform. 73(1), 1–23 (2004)
Pasupa, K., Chatkamjuncharoen, P., Wuttilertdeshar, C., Sugimoto, M.: Using image features and eye tracking device to predict human emotions towards abstract images. In: Bräunl, T., McCane, B., Rivera, M., Yu, X. (eds.) PSIVT 2015. LNCS, vol. 9431, pp. 419–430. Springer, Cham (2016). doi:10.1007/978-3-319-29451-3_34
Pasupa, K., Szedmak, S.: Utilising Kronecker decomposition and tensor-based multi-view learning to predict where people are looking in images. Neurocomputing 248, 80–93 (2017)
Rui, Y., Huang, T.S., Chang, S.F.: Image retrieval: Current techniques, promising directions, and open issues. J. Vis. Commun. Image Representation 10(1), 39–62 (1999)
Schapiro, M.: Nature of Abstract Art. American Marxist Association (1937)
Smeulders, A.W., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1349–1380 (2000)
Strandvall, T.: Eye Tracking in Human-Computer Interaction and Usability Research. In: Gross, T., Gulliksen, J., Kotzé, P., Oestreicher, L., Palanque, P., Prates, R.O., Winckler, M. (eds.) INTERACT 2009. LNCS, vol. 5727, pp. 936–937. Springer, Heidelberg (2009). doi:10.1007/978-3-642-03658-3_119
Wang, W., He, Q.: A survey on emotional semantic image retrieval. In: Proceeding of 15th IEEE International Conference on Image Processing, ICIP 2008, CA, USA, pp. 117–120 (2008)
Wei-ning, W., Ying-lin, Y., Sheng-ming, J.: Image retrieval by emotional semantics: a study of emotional space and feature extraction. In: Proceeding of the IEEE International Conference on Systems, Man, and Cybernetics, SMC 2006, Taipei, Taiwan, pp. 3534–3539 (2006)
You, Q., Luo, J., Jin, H., Yang, J.: Robust image sentiment analysis using progressively trained and domain transferred deep networks. In: Proceeding of the 29th AAAI Conference on Artificial Intelligence, AAAI 2015, Austin, Texas, USA, pp. 381–388 (2015)
Acknowledgments
This work was supported by the Faculty of Information Technology, King Mongkut’s Institute of Technology Ladkrabang under grant agreement number 2560-06-002.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this paper
Cite this paper
Pasupa, K., Sunhem, W., Loo, C.K., Kuroki, Y. (2017). Can Eye Movement Improve Prediction Performance on Human Emotions Toward Images Classification?. In: Liu, D., Xie, S., Li, Y., Zhao, D., El-Alfy, ES. (eds) Neural Information Processing. ICONIP 2017. Lecture Notes in Computer Science(), vol 10637. Springer, Cham. https://doi.org/10.1007/978-3-319-70093-9_88
Download citation
DOI: https://doi.org/10.1007/978-3-319-70093-9_88
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-70092-2
Online ISBN: 978-3-319-70093-9
eBook Packages: Computer ScienceComputer Science (R0)