Skip to main content
SpringerLink
Log in

Non-intrusive Measurement of Player Engagement and Emotions - Real-Time Deep Neural Network Analysis of Facial Expressions During Game Play

  • Conference paper
  • First Online:
HCI in Games (HCII 2020)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 12211))

Included in the following conference series:

Abstract

Prior research suggests and reveals that there is a correlation between human emotional responses and the subjective qualities of digital interactive experiences. Using facial analysis done by deep neural networks presents a true non-intrusive way of measuring emotional responses and engagement assessed as the desire to continue playing. This paper proposes a tool to measure emotional responses across eight different emotions and in real time of any game. The emotional recognition system achieves an accuracy of 98% and the continuation desire system achieves 93.3% accuracy in a pilot test with a two player game and 78.5% accuracy in a single player game. This forms a strong tool that shows a correlation between emotions and the continuation desire of a player, which can be used to evaluate engagement in games and digital interactive experiences, e.g. in critical stages of development of said content.

Supported by Samsung Media Innovation Lab for Education (SMILE Lab).

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    https://www.playstation.com/en-us/games/littlebigplanet-3-ps4/.

  2. 2.

    John Cooney, 2009 (jmtb02) http://www.jmtb02.com/this-is-the-only-level.

References

  1. Nonverbal Communication: Science and Applications AU - Matsumoto, David AU - Hwang, Hyi Sung. SAGE Publications, Inc., Thousand Oaks (2013). https://doi.org/10.4135/9781452244037. http://sk.sagepub.com/books/nonverbal-communication

  2. Modl.ai (2019). http://modl.ai/

  3. Sensum (2019). https://sensum.co/

  4. Albanie, S., Nagrani, A., Vedaldi, A., Zisserman, A.: Emotion recognition in speech using cross-modal transfer in the wild. arXiv preprint arXiv:1808.05561 (2018)

  5. Amir Zaib Abbasi, D.H.T., Hlavacs, H.: Engagement in games: developing an instrument to measure consumer videogame engagement and its validation. Int. J. Comput. Games Technol. 2017, 1–10 (2017)

    Article  Google Scholar 

  6. Arriaga, O., Valdenegro-Toro, M., Ploger, P.: Real-time convolutional neural networks for emotion and gender classification. CoRR abs/1710.07557 (2017)

    Google Scholar 

  7. Bitouk, D., Verma, R., Nenkova, A.: Class-level spectral features for emotion recognition. Speech Commun. 52(7), 613–625 (2010)

    Article  Google Scholar 

  8. Brett, A., Smith, M., Price, E., Huitt, W.: Overview of the affective domain. Educ. Psychol. Interact, 1–21 (2003)

    Google Scholar 

  9. Brown, S., Vaughan, C.: Play: How it Shapes the Brain, Opens the Imagination, and Invigorates the Soul. Avery (2009). https://books.google.dk/books?id=ESQDsgqfgusC

  10. Buchanan, R., Csikszentmihalyi, M.: Flow: the psychology of optimal experience. Des. Issues 8(1), 80-1 (1991)

    Google Scholar 

  11. Cacioppo, J.T., Tassinary, L.G., Berntson, G.: Handbook of Psychophysiology. Cambridge University Press, Cambridge (2007)

    Google Scholar 

  12. Chen, J.: Flow in games (and everything else). Commun. ACM 50, 31–34 (2007)

    Article  Google Scholar 

  13. Christy, T., Kuncheva, L.I.: Technological advancements in affective gaming: a historical survey. GSTF Int. J. Comput. (JoC) 3(4), 32–41 (2013)

    Google Scholar 

  14. Darwin, C.: The Expression of the Emotions in Man and Animals. D. Appleton and Co., New York (1872)

    Book  Google Scholar 

  15. Debeauvais, T.: Challenge and Retention in Games. UC Irvine. ProQuest ID: Debeauvais\(\_\)uci\(\_\)0030D\(\_\)13948. Merritt ID: ark:/13030/m53n6r1p (2016). https://escholarship.org/uc/item/6k3357qx

  16. Drachen, A., Nacke, L.E., Yannakakis, G., Pedersen, A.L.: Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In: Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games, pp. 49–54. ACM (2010)

    Google Scholar 

  17. Ekman, P.: Body position, facial expression, and verbal behavior during interviews. 68, 295 (1964). https://doi.org/10.1037/h0040225

  18. Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)

    Article  Google Scholar 

  19. Ekman, P.: Emotions revealed. 12 (2004). https://doi.org/10.1136/sbmj.0405184

  20. Frith, C.: Role of facial expressions in social interactions. Philos. Trans. R. Soc. Lond. B Biol. Sci. 364(1535), 3453–3458 (2009)

    Article  Google Scholar 

  21. Goodfellow, I.J., et al.: Challenges in representation learning: a report on three machine learning contests. In: Lee, M., Hirose, A., Hou, Z.-G., Kil, R.M. (eds.) ICONIP 2013. LNCS, vol. 8228, pp. 117–124. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-42051-1_16

    Chapter  Google Scholar 

  22. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. CoRR abs/1512.03385 (2015). http://arxiv.org/abs/1512.03385

  23. Hilgard, E.R.: The trilogy of mind: cognition, affection, and conation. J. Hist. Behav. Sci. 16(2), 107–117 (1980)

    Article  Google Scholar 

  24. Huang, G., Liu, Z., Weinberger, K.Q.: Densely connected convolutional networks. CoRR abs/1608.06993 (2016). http://arxiv.org/abs/1608.06993

  25. Hudlicka, E., Broekens, J.: Foundations for modelling emotions in game characters: modelling emotion effects on cognition, pp. 1–6 (2009). https://doi.org/10.1109/ACII.2009.5349473

  26. Huitt, W., Cain, S.: An overview of the conative domain. Educ. Psychol. Interact., 1–20 (2005)

    Google Scholar 

  27. Ijsselsteijn, W., et al.: Measuring the experience of digital game enjoyment. In: Proceedings of Measuring Behavior (2008)

    Google Scholar 

  28. Jang, E.H., Park, B.J., Kim, S.H., Eum, Y., Sohn, J.H.: Identification of the optimal emotion recognition algorithm using physiological signals, pp. 1–6, November 2011

    Google Scholar 

  29. Jennett, C., et al.: Measuring and defining the experience of immersion in games. Int. J. Hum.-Comput. Stud. 66(9), 641–661 (2008)

    Article  Google Scholar 

  30. Lee, C., Yoo, S., Park, Y., Kim, N., Jeong, K., Lee, B.: Using neural network to recognize human emotions from heart rate variability and skin resistance. In: Engineering in Medicine and Biology 27th Annual Conference (2005)

    Google Scholar 

  31. Lombard, M., Ditton, T.: At the heart of it all: the concept of presence. J. Comput. Mediat. Commun. 3(2), 0 (1997)

    Article  Google Scholar 

  32. Matsumoto, D., Willingham, B.: Spontaneous facial expressions of emotion of congenitally and noncongenitally blind individuals. J. Pers. Soc. Psychol. 96, 1–10 (2009). https://doi.org/10.1037/a0014037

    Article  Google Scholar 

  33. Matsumotol, D., Hwang, H.S.: Reading facial expressions of emotion. Psychol. Sci. 115, 541–558 (2011)

    Google Scholar 

  34. Moore, S., Bowden, R.: Local binary patterns for multi-view facial expression recognition. Comput. Vis. Image Underst. 115(4), 541–558 (2011)

    Article  Google Scholar 

  35. O’Brien, H.L., Toms, E.G.: What is user engagement? A conceptual framework for defining user engagement with technology. J. Am. Soc. Inform. Sci. Technol. 59(6), 938–955 (2008). https://doi.org/10.1002/asi.20801

    Article  Google Scholar 

  36. Ouellet, S.: Real-time emotion recognition for gaming using deep convolutional network features. CoRR abs/1408.3750 (2014)

    Google Scholar 

  37. Picard, R.W.: Emotion research by the people, for the people. Emot. Rev. 2(3), 250–254 (2010)

    Article  Google Scholar 

  38. Pollreisz, D., TaheriNejad, N.: A simple algorithm for emotion recognition, using physiological signals of a smart watch. In: 2017 39th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2353–2356 (2017)

    Google Scholar 

  39. Posner, J., Russell, J.A., Peterson, B.S.: The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17(3), 715–734 (2005)

    Article  Google Scholar 

  40. Rashid, M., Abu-Bakar, S.A.R., Mokji, M.: Human emotion recognition from videos using spatio-temporal and audio features. Vis. Comput. 29(12), 1269–1275 (2012). https://doi.org/10.1007/s00371-012-0768-y

    Article  Google Scholar 

  41. Russell, J.A.: A circumplex model of affect. Pers. Soc. Psychol. 39, 1161 (1980)

    Article  Google Scholar 

  42. Russell, J.A., Barrett, L.F.: Core affect, prototypical emotional episodes, and other things called emotion: dissecting the elephant. J. Pers. Soc. Psychol. 76, 805 (1999)

    Article  Google Scholar 

  43. Saarni, C.: The Development of Emotional Competence. Guilford Series on Social and Emotional Development. Guilford Publications, New York (1999)

    Google Scholar 

  44. Sagi, B.R., Silvestrini, R.: Application of combinatorial tests in video game testing. 29, 745–759 (2017). https://doi.org/10.1080/08982112.2017.1300919

  45. Schoenau-Fog, H.: Hooked! – evaluating engagement as continuation desire in interactive narratives. In: Si, M., Thue, D., André, E., Lester, J.C., Tanenbaum, J., Zammitto, V. (eds.) ICIDS 2011. LNCS, vol. 7069, pp. 219–230. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-25289-1_24

    Chapter  Google Scholar 

  46. Schoenau-Fog, H.: The player engagement process - an exploration of continuation desire in digital games. In: DiGRA - Proceedings of the 2011 DiGRA International Conference: Think Design Play. DiGRA/Utrecht School of the Arts, January 2011

    Google Scholar 

  47. Schoenau-Fog, H.: At the Core of Player Experience: Continuation Desire in Digital Games, pp. 388–410. Wiley (2014). https://doi.org/10.1002/9781118796443.ch14

  48. Schoenau-Fog, H.: Designing and evaluating conative game-based learning scenarios. In: Busch, C. (ed.) Proceedings of The 8th European Conference on Games Based Learning – ECGBL 2014, pp. 512–518. Academic Conferences and Publishing International, October 2014

    Google Scholar 

  49. Schoenau-Fog, H., Birke, A., Reng, L.: Evaluation of continuation desire as an iterative game development method. In: Proceeding of the 16th International Academic MindTrek Conference, MindTrek 2012, pp. 241–243. ACM, New York (2012). https://doi.org/10.1145/2393132.2393185

  50. Schoenau-Fog, H., Louchart, S., Lim, T., Soto-Sanfiel, M.T.: Narrative engagement in games-a continuation desire perspective. In: FDG, pp. 384–387 (2013)

    Google Scholar 

  51. Selvig, D., Stovring, N., Sjoblom, A., Korsholm, J.: Continuation desire and its physiological underpinnings - unpublished (2018)

    Google Scholar 

  52. Seng, K.P., Ang, L., Ooi, C.S.: A combined rule-based amp; machine learning audio-visual emotion recognition approach. IEEE Trans. Affect. Comput. 9(1), 3–13 (2018)

    Article  Google Scholar 

  53. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  54. Szegedy, C., Ioffe, S., Vanhoucke, V.: Inception-v4, inception-resnet and the impact of residual connections on learning. CoRR abs/1602.07261 (2016)

    Google Scholar 

  55. Tawari, A., Trivedi, M.M.: Face expression recognition by cross modal data association. IEEE Trans. Multimedia 15(7), 1543–1552 (2013)

    Article  Google Scholar 

  56. Tawari, A., Trivedi, M.M.: Audio visual cues in driver affect characterization: issues and challenges in developing robust approaches, pp. 2997–3002 (2011). https://doi.org/10.1109/IJCNN.2011.6033615

  57. Tokuno, S., et al.: Usage of emotion recognition in military health care, pp. 1–5 (2011). https://doi.org/10.1109/DSR.2011.6026823

  58. Vyvey, T., Castellar, E.N., Van Looy, J.: Loaded with fun? The impact of enjoyment and cognitive load on brand retention in digital games. 18, 72–82 (2018). https://doi.org/10.1080/15252019.2018.1446370

  59. Wang, C., Pun, T., Chanel, G.: A comparative survey of methods for remote heartrate detection from frontal face videos. Front. Bioeng. Biotechnol. 6, 33 (2018)

    Article  Google Scholar 

  60. Xerfi: Global video game turnover between 2011 and 2019 (2019). https://www.statista.com/statistics/862278/global-video-game-revenues-worldwide/. Accessed 19 Mar 2019

  61. Yu, S.N., Chen, S.F.: Emotion state identification based on heart rate variability and genetic algorithm, pp. 538–541, August 2015. https://doi.org/10.1109/EMBC.2015.7318418

  62. Zhang, T., Zheng, W., Cui, Z., Zong, Y., Li, Y.: Spatial-temporal recurrent neural network for emotion recognition. IEEE Trans. Cybern. 49(3), 839–847 (2018)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Selvig Consultancy, Copenhagen, Denmark

    Dines Rae Selvig

  2. Aalborg University, Copenhagen, Denmark

    Henrik Schoenau-Fog

Authors
  1. Dines Rae Selvig
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henrik Schoenau-Fog
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Dines Rae Selvig or Henrik Schoenau-Fog .

Editor information

Editors and Affiliations

  1. DePaul University, Chicago, IL, USA

    Xiaowen Fang

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Rae Selvig, D., Schoenau-Fog, H. (2020). Non-intrusive Measurement of Player Engagement and Emotions - Real-Time Deep Neural Network Analysis of Facial Expressions During Game Play. In: Fang, X. (eds) HCI in Games. HCII 2020. Lecture Notes in Computer Science(), vol 12211. Springer, Cham. https://doi.org/10.1007/978-3-030-50164-8_24

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-50164-8_24

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-50163-1

  • Online ISBN: 978-3-030-50164-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation