research-article

Neural Network Based Facial Expression Analysis of GameEvents: A Cautionary Tale

Authors:

Shaghayegh Roohi,

J. Matias Kivikangas,

Perttu HämäläinenAuthors Info & Claims

CHI PLAY '18: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play

Pages 429 - 437

https://doi.org/10.1145/3242671.3242701

Published: 23 October 2018 Publication History

Abstract

We present an exploratory study of analyzing and visualizing player facial expressions from video with deep neural networks. We contribute a novel data processing and visualization technique we call Affect Gradients, which provides descriptive statistics of the expressive responses to game events, such as player death or collecting a power-up. As an additional contribution, we show that although there has been tremendous recent progress in deep neural networks and computer vision, interpreting the results as direct read-outs of experiential states is not advised. According to our data, getting killed appears to make players happy, and much more so than killing enemies, although one might expect the exact opposite. A visual inspection of the data reveals that our classifier works as intended, and our results illustrate the limitations of making inferences based on facial images and discrete emotion labels. For example, players may laugh off the death, in which case the closest label for the facial expression is "happy", but the true emotional state is complex and ambiguous. On the other hand, players may frown in concentration while killing enemies or escaping a tight spot, which can easily be interpreted as an "angry" expression.

References

[1]

Laith Al-Shawaf, Daniel Conroy-Beam, Kelly Asao, and David M Buss. 2016. Human emotions: An evolutionary psychological perspective. Emotion Review 8, 2 (2016), 173--186.

[2]

Paris Mavromoustakos Blom, Sander Bakkes, Chek Tien Tan, Shimon Whiteson, Diederik M Roijers, Roberto Valenti, Theo Gevers, and others. 2014. Towards Personalised Gaming via Facial Expression Recognition. In AIIDE.

Digital Library

[3]

Boyan Bontchev. 2016. Adaptation in affective video games: A literature review. Cybernetics and Information Technologies 16, 3 (2016), 3--34.

Digital Library

[4]

Boyan Bontchev and Dessislava Vassileva. 2016. Assessing engagement in an emotionally-adaptive applied game. In Proceedings of the Fourth International Conference on Technological Ecosystems for Enhancing Multiculturality. ACM, 747--754.

Digital Library

[5]

Julia Ayumi Bopp, Elisa D Mekler, and Klaus Opwis. 2016. Negative emotion, positive experience?: emotionally moving moments in digital games. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2996--3006.

Digital Library

[6]

Margaret M Bradley and Peter J Lang. 1994. Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25, 1 (1994), 49--59.

[7]

J. *Cabrieto, J.M. *Kivikangas, K. Fayn, P. Kuppens, and E. Ceulemans. 2018. Finding emotional coherence using automatic change point detection methods on physiological data. In Manuscript in preparation. https://jmatiaskivikangas.net/publications/

[8]

Franccois Chollet and others. 2015. Keras. https://keras.io. (2015).

[9]

Thomas Christy and Ludmila I Kuncheva. 2014. Technological advancements in affective gaming: A historical survey. GSTF Journal on Computing (JoC) 3, 4 (2014), 32.

[10]

Cicero dos Santos and Maira Gatti. 2014. Deep convolutional neural networks for sentiment analysis of short texts. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. 69--78.

[11]

Paul Ekman. 1971. Universals and cultural differences in facial expressions of emotion. In Nebraska symposium on motivation. University of Nebraska Press.

[12]

José-Miguel Fernández-Dols. 2013. Advances in the study of facial expression: an introduction to the special section. Emotion Review 5, 1 (2013), 3--7.

[13]

Kiel Gilleade, Alan Dix, and Jen Allanson. 2005. Affective videogames and modes of affective gaming: assist me, challenge me, emote me. DiGRA 2005: Changing Views--Worlds in Play. (2005).

[14]

Eva Hudlicka. 2008. Affective computing for game design. In Proceedings of the 4th Intl. North American Conference on Intelligent Games and Simulation. 5--12.

[15]

Susanne Kaiser and Thomas Wehrle. 2001. Facial expressions as indicators of appraisal processes. Appraisal processes in emotion: Theory, methods, research (2001), 285--300.

[16]

S. Kaiser, T. Wehrle, and S. Schmidt. 1998. Emotional episodes, facial expressions, and reported feelings in human-computer interactions. In Proceedings of the Xth Conference of the International Society for Research on Emotions.

[17]

Kostas Karpouzis, Georgios N Yannakakis, Noor Shaker, and Stylianos Asteriadis. 2015. The platformer experience dataset. In Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on. IEEE, 712--718.

Digital Library

[18]

Jun H Kim, Daniel V Gunn, Eric Schuh, Bruce Phillips, Randy J Pagulayan, and Dennis Wixon. 2008. Tracking real-time user experience (TRUE): a comprehensive instrumentation solution for complex systems. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 443--452.

Digital Library

[19]

Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755--1758.

Digital Library

[20]

J Matias Kivikangas and others. 2015. Emotion and social context in a digital game experience. Institute of Behavioural Sciences, Studies in Psychology (2015).

[21]

J Matias Kivikangas, Guillaume Chanel, Ben Cowley, Inger Ekman, Mikko Salminen, Simo Järvelä, and Niklas Ravaja. 2011. A review of the use of psychophysiological methods in game research. journal of gaming & virtual worlds 3, 3 (2011), 181--199.

[22]

J Matias Kivikangas and Niklas Ravaja. 2013. Emotional responses to victory and defeat as a function of opponent. IEEE Transactions on Affective Computing 4, 2 (2013), 173--182.

Digital Library

[23]

Ming Liang and Xiaolin Hu. 2015. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3367--3375.

[24]

Hugo Lövheim. 2012. A new three-dimensional model for emotions and monoamine neurotransmitters. Medical hypotheses 78, 2 (2012), 341--348.

[25]

Ben Medler, Michael John, and Jeff Lane. 2011. Data cracker: developing a visual game analytic tool for analyzing online gameplay. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2365--2374.

Digital Library

[26]

Pejman Mirza-Babaei, Lennart E Nacke, John Gregory, Nick Collins, and Geraldine Fitzpatrick. 2013. How does it play better?: exploring user testing and biometric storyboards in games user research. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 1499--1508.

Digital Library

[27]

Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International Conference on Machine Learning (ICML), Vol. 2017.

Digital Library

[28]

Niklas Ravaja, Marko Turpeinen, Timo Saari, Sampsa Puttonen, and Liisa Keltikangas-Järvinen. 2008. The psychophysiology of James Bond: Phasic emotional responses to violent video game events. Emotion 8, 1 (2008), 114.

[29]

Shaghayegh Roohi, Jari Takatalo, Christian Guckelsberger, and Perttu Hämäläinen. 2018. Review of Intrinsic Motivation in Simulation-based Game Testing. In Accepted for CHI 2018.

Digital Library

[30]

James A Russell. 2003. Core affect and the psychological construction of emotion. Psychological review 110, 1 (2003), 145.

[31]

James A Russell. 2014. Four perspectives on the psychology of emotion: An introduction. Emotion Review 6, 4 (2014), 291.

[32]

Klaus R Scherer. 1999. Appraisal theory. Handbook of cognition and emotion (1999), 637--663.

[33]

Jürgen Schmidhuber. 1991. A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats. 222--227.

Digital Library

[34]

Paul J Silvia. 2005. What is interesting? Exploring the appraisal structure of interest. Emotion 5, 1 (2005), 89.

[35]

Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).

[36]

Jonathan Sykes and Simon Brown. 2003. Affective gaming: measuring emotion through the gamepad. In CHI'03 extended abstracts on Human factors in computing systems. ACM, 732--733.

Digital Library

[37]

Chek Tien Tan, Sander Bakkes, and Yusuf Pisan. 2014. Correlation between Facial Expressions and the Game Experience Questionnaire. In Entertainment Computing-ICEC 2014: 13th International Conference, ICEC 2014, Sydney, Australia, October 1--3, 2014, Proceedings, Vol. 8770. Springer, 229.

[38]

Wouter Van den Hoogen, Karolien Poels, Wijnand IJsselsteijn, and Yvonne de Kort. 2012. Between challenge and defeat: Repeated player-death and game enjoyment. Media Psychology 15, 4 (2012), 443--459.

[39]

Bret Victor. 2011. Up and Down the Ladder of Abstraction, A Systematic Approach to Interactive Visualization. http://worrydream.com/LadderOfAbstraction/. (2011).

[40]

Georgios N Yannakakis, Pieter Spronck, Daniele Loiacono, and Elisabeth André. 2013. Player modeling. In Dagstuhl Follow-Ups, Vol. 6. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.

Cited By

Yannakakis GMelhart D(2023)Affective Game Computing: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3315689111:10(1423-1444)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3315689
Bosman IBuruk OJørgensen KHamari J(2023)The effect of audio on the experience in virtual reality: a scoping reviewBehaviour & Information Technology10.1080/0144929X.2022.215837143:1(165-199)Online publication date: 2-Jan-2023
https://doi.org/10.1080/0144929X.2022.2158371
Siqueira EFleury MLamar MDrachen ACastanho CJacobi R(2022)An automated approach to estimate player experience in game events from psychophysiological dataMultimedia Tools and Applications10.1007/s11042-022-13845-582:13(19189-19220)Online publication date: 15-Oct-2022
https://dl.acm.org/doi/10.1007/s11042-022-13845-5
Show More Cited By

Index Terms

Neural Network Based Facial Expression Analysis of GameEvents: A Cautionary Tale
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
    2. Interaction techniques
      1. Gestural input
  2. Visualization
    1. Visualization design and evaluation methods

Recommendations

Emotional facial expression interface: effects of displayed facial designs
Ergo'IA '10: Proceedings of the Ergonomie et Informatique Avancee Conference

In supporting the visual-mediated emotional recognition, little research has centred on analysing the effect of presenting different combinatorial facial designs. Moreover, in the theoretical field of emotions, research on the recognition of emotional ...
Introducing EVG: an emotion evoking game
IVA'06: Proceedings of the 6th international conference on Intelligent Virtual Agents

A dungeon role playing game intended to induce emotions such as boredom, surprise, joy, anger and disappointment is introduced. From the preliminary study, facial expressions indicating boredom and anger were observed. Individual differences were found ...
A Hybrid Approach for Facial Expression Recognition
IMCOM '18: Proceedings of the 12th International Conference on Ubiquitous Information Management and Communication

Facial expression recognition is an emerging research area for improving human and computer interaction. This research plays a significant role in the field of social communication, commercial enterprise, law enforcement, and other computer ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI PLAY '18: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play

October 2018

563 pages

ISBN:9781450356244

DOI:10.1145/3242671

General Chairs:
Florian 'Floyd' Mueller
Exertion Games Lab, RMIT University, Melbourne, Australia
,
Daniel Johnson
Queensland University of Technology, Brisbane, Australia
,
Ben Schouten
Eindhoven University of Technology and Amsterdam University of Applied Sciences, the Netherlands
,
Program Chairs:
Phoebe O. Toups Dugas
Play & Interactive Experiences for Learning Lab, New Mexico State University, Las Cruces, New Mexico, USA
,
Peta Wyeth
Queensland University of Technology, Brisbane, Australia

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Academy of Finland

Conference

CHI PLAY '18

Sponsor:

SIGCHI

CHI PLAY '18: The annual symposium on Computer-Human Interaction in Play

October 28 - 31, 2018

VIC, Melbourne, Australia

Acceptance Rates

CHI PLAY '18 Paper Acceptance Rate 43 of 123 submissions, 35%;

Overall Acceptance Rate 421 of 1,386 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

7
Total Citations
View Citations
277
Total Downloads

Downloads (Last 12 months)40
Downloads (Last 6 weeks)1

Reflects downloads up to 28 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yannakakis GMelhart D(2023)Affective Game Computing: A SurveyProceedings of the IEEE10.1109/JPROC.2023.3315689111:10(1423-1444)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3315689
Bosman IBuruk OJørgensen KHamari J(2023)The effect of audio on the experience in virtual reality: a scoping reviewBehaviour & Information Technology10.1080/0144929X.2022.215837143:1(165-199)Online publication date: 2-Jan-2023
https://doi.org/10.1080/0144929X.2022.2158371
Siqueira EFleury MLamar MDrachen ACastanho CJacobi R(2022)An automated approach to estimate player experience in game events from psychophysiological dataMultimedia Tools and Applications10.1007/s11042-022-13845-582:13(19189-19220)Online publication date: 15-Oct-2022
https://dl.acm.org/doi/10.1007/s11042-022-13845-5
Ulrich LNonis FVezzetti EMoos SCaruso GShi YMarcolin F(2021)Can ADAS Distract Driver’s Attention? An RGB-D Camera and Deep Learning-Based AnalysisApplied Sciences10.3390/app11241158711:24(11587)Online publication date: 7-Dec-2021
https://doi.org/10.3390/app112411587
Stahlke SNova AMirza-Babaei PMirza-Babaei PMcArthur VVanden Abeele VBirk M(2020)Artificial Players in the Design Process: Developing an Automated Testing Tool for Game Level and World DesignProceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3410404.3414249(267-280)Online publication date: 2-Nov-2020
https://dl.acm.org/doi/10.1145/3410404.3414249
Melhart DYannakakis GLiapis A(2020)I Feel I Feel You: A Theory of Mind Experiment in GamesKI - Künstliche Intelligenz10.1007/s13218-020-00641-234:1(45-55)Online publication date: 4-Feb-2020
https://doi.org/10.1007/s13218-020-00641-2
Roohi SMekler ETavast MBlomqvist THämäläinen PArnedo JNacke LVanden Abeele VToups Dugas P(2019)Recognizing Emotional Expression in Game StreamsProceedings of the Annual Symposium on Computer-Human Interaction in Play10.1145/3311350.3347197(301-311)Online publication date: 17-Oct-2019
https://dl.acm.org/doi/10.1145/3311350.3347197

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten