Abstract
Users’ emotional states influence decision making and are essential for the knowledge and explanation of users’ behavior with computer applications. However, collecting emotional states during the interaction time with users is a onerous task because it requires very careful handling of the empirical observation, leading researchers to carry out assessments of emotional responses only at the end of the interaction. This paper reports our research in assessing users’ behavior at interaction time and also describes the results of a case study which analyzed users’ emotional responses while interacting with a game. We argue that capturing emotions during interaction time can help us in making changes on users’ behavior (e.g., changing from stressed to a less stressed state) or even suggesting an user to have a break. This can be all possible if both (1) emotions are captured during interaction and (2) changes are suggested at runtime (e.g., through persuasion). The results of this study suggest that there are significant differences between emotional responses captured during the interaction and those declared at the end.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
The study was approved by the Human Research Ethics Committee of the University of Sao Paulo Medical School, Sao Paulo, Brazil (CAAE-45081415.0.0000.0065).
References
Bradley MM, Lang PJ (1994) Measuring emotion: the self-assessment manikin and the semantic differential. J Behav Ther Exp Psychiatry 25(1):49–59
Brave S, Nass C (2003) Emotion in human-computer interaction. In: Jacko JA, Sears A (eds) The human-computer interaction handbook. L. Erlbaum Associates Inc., Hillsdale, pp 81–96
Chanel G, Kierkels JJ, Soleymani M, Pun T (2009) Short-term emotion assessment in a recall paradigm. Int J Hum Comput Stud 67(8):607–627
Davidson RJ, Scherer KR, Goldsmith H (2003) Handbook of affective sciences. Oxford University Press, Oxford
Desmet P (2004) Funology. In: Blythe MA, Overbeeke K, Monk AF, Wright PC (eds) Measuring emotion: development and application of an instrument to measure emotional responses to products. Kluwer Academic Publishers, Norwell, pp 111–123
Duncker K, Lees LS (1945) On problem-solving. Psychol Monogr 58(5):i
Ericsson K, Simon H (1993) Protocol analysis; verbal reports as data (revised edition). bradfordbooks
Filho GPR, Ueyama J, Villas LA, Pinto AR, Gonalves VP, Pessin G, Pazzi RW, Braun T (2014) Nodepm: a remote monitoring alert system for energy consumption using probabilistic techniques. Sensors 14(1):848–867
Fontaine JR, Poortinga YH, Setiadi B, Markam SS (2002) Cognitive structure of emotion terms in indonesia and the netherlands. Cogn Emot 16(1):61–86
Frantzidis CA, Bratsas C, Klados MA, Konstantinidis E, Lithari CD, Vivas AB, Papadelis CL, Kaldoudi E, Pappas C, Bamidis PD (2010) On the classification of emotional biosignals evoked while viewing affective pictures: an integrated data-mining-based approach for healthcare applications. Inform Technol Biomed IEEE Trans 14(2):309–318
Gonçalves VP, de Almeida Neris VP, Seraphini S, Dias TCM, Pessin G, Johnson T, Ueyama J (2015) Providing adaptive smartphone interfaces targeted at elderly people: an approach that takes into account diversity among the elderly. Univ Access Inf Soc 1–21. doi:10.1007/s10209-015-0429-9
Hou X, Liu Y, Sourina O, Tan YRE, Wang L, Mueller-Wittig W (2015) EEG based Stress Monitoring. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC), 2015, pp 3110–3115. doi:10.1109/SMC.2015.540
Kahou SE, Bouthillier X, Lamblin P, Gulcehre C, Michalski V, Konda K, Jean S, Froumenty P, Dauphin Y, Boulanger-Lewandowski N, Ferrari RC, Mirza M, Warde-Farley D, Courville A, Vincent P, Memisevic R, Pal C, Bengio Y (2015) EmoNets: Multimodal deep learning approaches for emotion recognition in video. J Multimodal User Interfaces 1–13. doi:10.1007/s12193-015-0195-2
Kukolja D, Popović S, Horvat M, Kovač B, Ćosić K (2014) Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int J Hum Comput Stud 72(10):717–727
Lan Z, Sourina O, Wang LP, Liu Y (2014) Stability of features in real-time EEG-based emotion recognition algorithm. In: International Conference on Cyberworlds (CW), 2014, pp 137–144. doi:10.1109/CW.2014.27
Lathia N, Rachuri K, Mascolo C, Roussos G (2013) Open source smartphone libraries for computational social science. In: Proceedings of the 2013 ACM conference on Pervasive and ubiquitous computing adjunct publication. ACM, pp 911–920
Li T, An C, Campbell AT, Zhou X (2015) Hilight: Hiding bits in pixel translucency changes. ACM SIGMOBILE Mobile Comput Commun Rev 18(3):62–70
LiKamWa R, Liu Y, Lane ND, Zhong L (2013) Moodscope: building a mood sensor from smartphone usage patterns. In: Proceeding of the 11th annual international conference on Mobile systems, applications, and services. ACM. pp 389–402
Mahlke S, Minge M (2008) Consideration of multiple components of emotions in human-technology interaction. In: Peter C, Beale R (eds) Affect and Emotion in Human-Computer Interaction, Lecture Notes in Computer Science, vol 4868. Springer, Berline, pp 51–62
Mano LY, Giancristofaro GT, Faiçal B, Libralon G, Pessin G, Gomes PH, Ueyama J (2015) Exploiting the Use of Ensemble Classifiers to Enhance the Precision of User’s Emotion Classification. In: Proceedings of the 16th International Conference on Engineering Applications of Neural Networks (INNS)
Mitchell TM (1997) Machine learning. McGraw Hill, Boston, MA
Monard MC, Baranauskas JA (2003) Conceitos sobre aprendizado de máquina. Sistemas Inteligentes-Fundamentos e Aplicações 1:1
Nahin ANH, Alam JM, Mahmud H, Hasan K (2014) Identifying emotion by keystroke dynamics and text pattern analysis. Behav Inform Technol 33(9):987–996
Niedenthal PM, Krauth-Gruber S, Ric F (2006) Psychology of emotion: interpersonal, experiential, and cognitive approaches. Psychology Press, New York
Pedersen C, Togelius J, Yannakakis GN (2010) Modeling player experience for content creation. IEEE Trans Comput Intell AI Games 2(1):54–67
Peter C, Urban B (2012) Emotion in human-computer interaction. In: Expanding the frontiers of visual analytics and visualization. Springer, pp 239–262
Ramakrishnan S, El Emary IM (2013) Speech emotion recognition approaches in human computer interaction. Telecommun Syst 52(3):1467–1478
Reijneveld K, de Looze M, Krause F, Desmet P (2003) Measuring the emotions elicited by office chairs. In: Proceedings of the 2003 International Conference on Designing Pleasurable Products and Interfaces, ser. DPPI ’03. ACM, New York. p 6–10
Rosales GCM, Borges de Araujo R, Otsuka JL, da Rocha RV (2011) Using logical sensors network to the accurate monitoring of the learning process in distance education courses. In: Advanced Learning Technologies (ICALT), 2011 11th IEEE International Conference on. IEEE, pp 573–575
Russell J (1980) A circumplex model of affect. J Personal Soc Psychol 39(6):1161–1178
Russell JA, Weiss A, Mendelsohn GA (1989) Affect grid: a single-item scale of pleasure and arousal. J Personal Soc Psychol 57(3):493
Santos V, Coca S, Libralon G, Romero R (2013) Imitation of facial expressions for learning emotions in social robotics. In: Proceedings of the 2013 Simposio Brasileiro de Automacao Inteligente
Saragih JM, Lucey S, Cohn JF (2011) Deformable model fitting by regularized landmark mean-shift. Int J Comput Vis 91(2):200–215
Scherer K (2005) What are emotions? And how can they be measured? Soc Sci Inf 44:695–729
Van Someren MW, Barnard YF, Sandberg JA et al (1994) The think aloud method: a practical guide to modelling cognitive processes, vol 2. Academic Press, London
Wang R, Chen F, Chen Z, Li T, Harari G, Tignor S, Zhou X, Ben-Zeev D, Campbell AT (2014) Studentlife: assessing mental health, academic performance and behavioral trends of college students using smartphones. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM. pp 3–14
Wrzus C, Mehl MR (2015) Lab and/or field? Measuring personality processes and their social consequences. Eur J Personal 29(2):250–271
Xavier RAC, de Almeida Neris VP (2014) Measuring the emotional experience of users through a hybrid semantic approach. In: Proceedings of the 13th Brazilian Symposium on Human Factors in Computing Systems. Sociedade Brasileira de Computação, pp 226–235
Xavier RAC, Garcia FE, de Almeida Neris VP (2012) Decisoes de design de interfaces ruins e o impacto delas na interacao: Um estudo preliminar considerando o estado emocional de idosos. In: Proceedings of the 11th Brazilian Symposium on Human Factors in Computing Systems, IHC ’12. Brazilian Computer Society, Porto Alegre, Brazil. pp 127–136
Zhou F, Qu X, Helander MG, Jiao JR (2011) Affect prediction from physiological measures via visual stimuli. Int J Hum Comput Stud 69(12):801–819
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that there is no conflict of interests.
Additional information
Communicated by V. Loia.
Rights and permissions
About this article
Cite this article
Gonçalves, V.P., Giancristofaro, G.T., Filho, G.P.R. et al. Assessing users’ emotion at interaction time: a multimodal approach with multiple sensors. Soft Comput 21, 5309–5323 (2017). https://doi.org/10.1007/s00500-016-2115-0
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00500-016-2115-0