Skip to main content
SpringerLink
Log in

Accuracy Evaluation of Remote Photoplethysmography Estimations of Heart Rate in Gaming Sessions with Natural Behavior

  • Conference paper
  • First Online:
Advances in Computer Entertainment Technology (ACE 2017)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 10714))

Included in the following conference series:

Abstract

Remote photoplethysmography (rPPG) can be used to remotely estimate heart rate (HR) of users to infer their emotional state. However natural body movement and facial actions of users significantly impact such techniques, so their reliability within contexts involving natural behavior must be checked. We present an experiment focused on the accuracy evaluation of an established rPPG technique in a gaming context. The technique was applied to estimate the HR of subjects behaving naturally in gaming sessions whose games were carefully designed to be casual-themed, similar to off-the-shelf games and have a difficulty level that linearly progresses from a boring to a stressful state. Estimations presented mean error of 2.99 bpm and Pearson correlation \(r=0.43\), \(p < 0.001\), however with significant variations among subjects. Our experiment is the first to measure the accuracy of an rPPG technique using boredom/stress-inducing casual games with subjects behaving naturally.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 179.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 229.00
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Allen, J.: Photoplethysmography and its application in clinical physiological measurement. Physiol. Meas. 28(3), R1–R39 (2007). https://doi.org/10.1088/0967-3334/28/3/r01

    Article  Google Scholar 

  2. Appelhans, B.M., Luecken, L.J.: Heart rate variability as an index of regulated emotional responding. Rev. Gen. Psychol. 10(3), 229 (2006)

    Article  Google Scholar 

  3. Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3430–3437. Institute of Electrical and Electronics Engineers (IEEE), June 2013. https://doi.org/10.1109/cvpr.2013.440

  4. Bevilacqua, F., Backlund, P., Engstrom, H.: Variations of facial actions while playing games with inducing boredom and stress. In: 2016 8th International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES), pp. 1–8. Institute of Electrical and Electronics Engineers (IEEE), IEEE, September 2016. https://doi.org/10.1109/vs-games.2016.7590374

  5. Bousefsaf, F., Maaoui, C., Pruski, A.: Continuous wavelet filtering on webcam photoplethysmographic signals to remotely assess the instantaneous heart rate. Biomed. Signal Process. Control 8(6), 568–574 (2013). https://doi.org/10.1016/j.bspc.2013.05.010

    Article  Google Scholar 

  6. Bousefsaf, F., Maaoui, C., Pruski, A.: Remote assessment of the heart rate variability to detect mental stress. In: Proceedings of the ICTs for Improving Patients Rehabilitation Research Techniques, pp. 348–351. Institute for Computer Sciences, Social Informatics and Telecommunications Engineering (ICST), IEEE (2013). https://doi.org/10.4108/icst.pervasivehealth.2013.252181

  7. Boyle, E.A., Connolly, T.M., Hainey, T., Boyle, J.M.: Engagement in digital entertainment games: A systematic review. Comput. Hum. Behav. 28(3), 771–780 (2012)

    Article  Google Scholar 

  8. Brogni, A., Vinayagamoorthy, V., Steed, A., Slater, M.: Variations in physiological responses of participants during different stages of an immersive virtual environment experiment. In: Proceedings of the ACM Symposium on Virtual Reality Software and Technology - VRST 2006, pp. 376–382. ACM, Association for Computing Machinery (ACM) (2006). https://doi.org/10.1145/1180495.1180572

  9. Choi, J., Gutierrez-Osuna, R.: Using heart rate monitors to detect mental stress. In: 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks, pp. 219–223. IEEE, Institute of Electrical and Electronics Engineers (IEEE), June 2009. https://doi.org/10.1109/bsn.2009.13

  10. Datcu, D., Cidota, M., Lukosch, S., Rothkrantz, L.: Noncontact automatic heart rate analysis in visible spectrum by specific face regions. In: Proceedings of the 14th International Conference on Computer Systems and Technologies, CompSysTech 2013, pp. 120–127 (2013). ACM, New York. ISBN 978-1-4503-2021-4. https://doi.org/10.1145/2516775.2516805, http://doi.acm.org/10.1145/2516775.2516805

  11. de Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013). https://doi.org/10.1109/tbme.2013.2266196

    Article  Google Scholar 

  12. Edwards, G.J., Taylor, C.J., Cootes, T.F.: Interpreting face images using active appearance models. In: Proceedings of the 3rd International Conference on Face & Gesture Recognition, FG 1998, p. 300, Washington, DC, USA (1998). IEEE Computer Society. ISBN 0-8186-8344-9. http://dl.acm.org/citation.cfm?id=520809.796067

  13. Fenton-O’Creevy, M., Lins, J.T., Vohra, S., Richards, D.W., Davies, G., Schaaff, K.: Emotion regulation and trader expertise: Heart rate variability on the trading floor. J. Neurosci. Psychol. Econ. 5(4), 227 (2012)

    Article  Google Scholar 

  14. Garde, A., Laursen, B., Jørgensen, A., Jensen, B.: Effects of mental and physical demands on heart rate variability during computer work. Eur. J. Appl. Physiol. 87(4–5), 456–461 (2002)

    Article  Google Scholar 

  15. Giannakakis, G., Pediaditis, M., Manousos, D., Kazantzaki, E., Chiarugi, F., Simos, P.G., Marias, K., Tsiknakis, M.: Stress and anxiety detection using facial cues from videos. Biomed. Signal Process. Control 31, 89–101 (2017). https://doi.org/10.1016/j.bspc.2016.06.020

    Article  Google Scholar 

  16. Grundlehner, B., Brown, L., Penders, J., Gyselinckx, B.: The design and analysis of a real-time, continuous arousal monitor. In: 2009 Sixth International Workshop on Wearable and Implantable Body Sensor Networks, pp. 156–161. Institute of Electrical and Electronics Engineers, IEEE, June 2009. https://doi.org/10.1109/bsn.2009.21

  17. Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Trans. Intell. Transp. Syst. 6(2), 156–166 (2005)

    Article  Google Scholar 

  18. Hsu, Y.C., Lin, Y.-L., Hsu, W.: Learning-based heart rate detection from remote photoplethysmography features. In: 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4433–4437. Institute of Electrical and Electronics Engineers (IEEE), IEEE, May 2014. https://doi.org/10.1109/icassp.2014.6854440

  19. Irani, R., Nasrollahi, K., Moeslund, T.B.: Improved pulse detection from head motions using DCT. Institute for Systems and Technologies of Information, Control and Communication (2014)

    Google Scholar 

  20. Jerritta, S., Murugappan, M., Nagarajan, R., Wan, K.: Physiological signals based human emotion recognition: a review. In: 2011 IEEE 7th International Colloquium on Signal Processing and its Applications, pp. 410–415. IEEE, Institute of Electrical and Electronics Engineers (IEEE), March 2011. https://doi.org/10.1109/cspa.2011.5759912

  21. Kivikangas, J.M., Chanel, G., Cowley, B., Ekman, I., Salminen, M., Järvelä, S., Ravaja, N.: A review of the use of psychophysiological methods in game research. J. Gaming Virtual Worlds 3(3), 181–199 (2011). https://doi.org/10.1386/jgvw.3.3.181_1

  22. Kukolja, D., Popović, S., Horvat, M., Kovač, B., Ćosić, K.: Comparative analysis of emotion estimation methods based on physiological measurements for real-time applications. Int. J. Hum. Comput. Stud. 72(10–11), 717–727 (2014). https://doi.org/10.1016/j.ijhcs.2014.05.006

    Article  Google Scholar 

  23. Landowska, A.: Emotion monitoring verification of physiological characteristics measurement procedures. Metrol. Meas. Syst. 21(4), 719–732 (2014). ISSN 2300–1941. https://doi.org/10.2478/mms-2014-0049

  24. Li, X., Chen, J., Zhao, G., Pietikainen, M.: Remote heart rate measurement from face videos under realistic situations. In: 2014 IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271. Institute of Electrical & Electronics Engineers (IEEE), June 2014. https://doi.org/10.1109/cvpr.2014.543

  25. McDuff, D., Gontarek, S., Picard, R.: Remote measurement of cognitive stress via heart rate variability. In: 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 2957–2960. Institute of Electrical and Electronics Engineers (IEEE), August 2014. https://doi.org/10.1109/embc.2014.6944243

  26. McDuff, D., Gontarek, S., Picard, R.W.: Improvements in remote cardiopulmonary measurement using a five band digital camera. IEEE Trans. Biomed. Eng. 61(10), 2593–2601 (2014). https://doi.org/10.1109/tbme.2014.2323695

    Article  Google Scholar 

  27. McDuff, D.J., Estepp, J.R., Piasecki, A.M., Blackford, E.B.: A survey of remote optical photoplethysmographic imaging methods. In: 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 6398–6404. IEEE, Institute of Electrical and Electronics Engineers (IEEE), August 2015. https://doi.org/10.1109/embc.2015.7319857

  28. McDuff, D.J., Hernandez, J., Gontarek, S., Picard, R.W.: COGCAM: Contact-free measurement of cognitive stress during computer tasks with a digital camera. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems - CHI 2016. Association for Computing Machinery (ACM) (2016). https://doi.org/10.1145/2858036.2858247

  29. Monkaresi, H., Calvo, R.A., Yan, H.: A machine learning approach to improve contactless heart rate monitoring using a webcam. IEEE J. Biomed. Health Inform. 18(4), 1153–1160 (2014). https://doi.org/10.1109/jbhi.2013.2291900

    Article  Google Scholar 

  30. Poh, M.-Z., McDuff, D.J., Picard, R.W.: Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. Opt. Express 18(10), 10762 (2010). https://doi.org/10.1364/oe.18.010762

    Article  Google Scholar 

  31. Poh, M.-Z., McDuff, D.J., Picard, R.W.: Advancements in noncontact, multiparameter physiological measurements using a webcam. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011)

    Article  Google Scholar 

  32. Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human-robot interaction. Pattern Anal. Appl. 9(1), 58–69 (2006)

    Article  Google Scholar 

  33. Ravaja, N., Saari, T., Laarni, J., Kallinen, K., Salminen, M., Holopainen, J., Järvinen, A.: The psychophysiology of video gaming: Phasic emotional responses to game events. In: International DiGRA Conference (2005)

    Google Scholar 

  34. Roald, N.G.: Estimation of vital signs from ambient-light non-contact photoplethysmography (2013)

    Google Scholar 

  35. Rodriguez, A., Rey, B., Vara, M.D., Wrzesien, M., Alcaniz, M., Banos, R.M., Perez-Lopez, D.: A VR-based serious game for studying emotional regulation in adolescents. IEEE Comput. Grap. Appl. 35(1), 65–73 (2015). https://doi.org/10.1109/mcg.2015.8

    Article  Google Scholar 

  36. Rouast, P.V., Adam, M.T.P., Chiong, R., Cornforth, D., Lux, E.: Remote heart rate measurement using low-cost RGB face video: A technical literature review. Front. Comput. Sci. 1–15 (2016). https://doi.org/10.1007/s11704-016-6243-6

  37. Schubert, C., Lambertz, M., Nelesen, R.A., Bardwell, W., Choi, J.-B., Dimsdale, J.E.: Effects of stress on heart rate complexitya comparison between short-term and chronic stress. Biol. Psychol. 80(3), 325–332 (2009)

    Article  Google Scholar 

  38. Sharma, R., Khera, S., Mohan, A., Gupta, N., Ray, R.B.: Assessment of computer game as a psychological stressor. Indian J. Physiol. Pharmacol. 50(4), 367 (2006)

    Google Scholar 

  39. Takano, C., Ohta, Y.: Heart rate measurement based on a time-lapse image. Med. Eng. Phys. 29(8), 853–857 (2007). https://doi.org/10.1016/j.medengphy.2006.09.006

    Article  Google Scholar 

  40. Tijs, T.J.W., Brokken, D., IJsselsteijn, W.A.: Dynamic game balancing by recognizing affect. In: Markopoulos, P., de Ruyter, B., IJsselsteijn, W., Rowland, D. (eds.) Fun and Games. LNCS, vol. 5294, pp. 88–93. Springer, Heidelberg (2008). https://doi.org/10.1007/978-3-540-88322-7_9

    Chapter  Google Scholar 

  41. Tran, D.N., Lee, H., Kim, C.: A robust real time system for remote heart rate measurement via camera. In: 2015 IEEE International Conference on Multimedia and Expo (ICME), pp. 1–6. IEEE, Institute of Electrical and Electronics Engineers (IEEE), June 2015. https://doi.org/10.1109/icme.2015.7177484

  42. Vandeput, S., Taelman, J., Spaepen, A., Van Huffel, S.: Heart rate variability as a tool to distinguish periods of physical and mental stress in a laboratory environment. In: Proceedings of the 6th International Workshop on Biosignal Interpretation (BSI), New Haven, CT, pp. 187–190 (2009)

    Google Scholar 

  43. Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16(26), 21434 (2008). https://doi.org/10.1364/oe.16.021434

    Article  Google Scholar 

  44. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  45. Wang, W., den Brinker, A., Stuijk, S., de Haan, G.: Algorithmic principles of remote-PPG. IEEE Trans. Biomed. Eng. 1 (2016). https://doi.org/10.1109/tbme.2016.2609282

  46. Wang, W., Stuijk, S., de Haan, G.: A novel algorithm for remote photoplethysmography: Spatial subspace rotation. IEEE Trans. Biomed. Eng. 63(9), 1974–1984 (2016). https://doi.org/10.1109/TBME.2015.2508602. ISSN 0018–9294

    Article  Google Scholar 

  47. Xiao, X., Wang, J.: Towards attentive, bi-directional MOOC learning on mobile devices. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction - ICMI 2015, pp. 163–170. ACM, Association for Computing Machinery (ACM) (2015). https://doi.org/10.1145/2818346.2820754

  48. Yamaguchi, M., Wakasugi, J., Sakakima, J.: Evaluation of driver stress using biomarker in motor-vehicle driving simulator. In: 2006 International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 1834–1837. IEEE, Institute of Electrical and Electronics Engineers (IEEE), August 2006. https://doi.org/10.1109/iembs.2006.260001

  49. Yamakoshi, T., Yamakoshi, K., Tanaka, S., Nogawa, M., Shibata, M., Sawada, Y., Rolfe, P., Hirose, Y.: A preliminary study on driver’s stress index using a new method based on differential skin temperature measurement. In: 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pp. 722–725. IEEE, Institute of Electrical and Electronics Engineers (IEEE), August 2007. https://doi.org/10.1109/iembs.2007.4352392

  50. Yun, C., Shastri, D., Pavlidis, I., Deng. Z.: O’ game, can you feel my frustration? In: Proceedings of the 27th International Conference on Human Factors in Computing Systems - CHI 2009, pp. 2195–2204. ACM, Association for Computing Machinery (ACM) (2009). https://doi.org/10.1145/1518701.1519036

  51. Zhao, F., Li, M., Qian, Y., Tsien, J.Z.: Remote measurements of heart and respiration rates for telemedicine. PLoS ONE 8(10), e71384 (2013). https://doi.org/10.1371/journal.pone.0071384

    Article  Google Scholar 

Download references

Acknowledgment

The authors would like to thank the participants and all involved personnel for their valuable contributions. This work has been performed with support from: CNPq, Conselho Nacional de Desenvolvimento Científico e Tecnológico - Brasil; University of Skövde; EU Interreg ÖKS project Game Hub Scandinavia; UFFS, Federal University of Fronteira Sul.

Author information

Authors and Affiliations

  1. University of Skövde, Skövde, Sweden

    Fernando Bevilacqua, Henrik Engström & Per Backlund

  2. Federal University of Fronteira Sul, Chapecó, Brazil

    Fernando Bevilacqua

Authors
  1. Fernando Bevilacqua
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Henrik Engström
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Per Backlund
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fernando Bevilacqua .

Editor information

Editors and Affiliations

  1. City, University of London, London, United Kingdom

    Adrian David Cheok

  2. University of Tokyo, Tokyo, Japan

    Masahiko Inami

  3. NOVA University of Lisbon, Lisbon, Portugal

    Teresa Romão

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG, part of Springer Nature

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Bevilacqua, F., Engström, H., Backlund, P. (2018). Accuracy Evaluation of Remote Photoplethysmography Estimations of Heart Rate in Gaming Sessions with Natural Behavior. In: Cheok, A., Inami, M., Romão, T. (eds) Advances in Computer Entertainment Technology. ACE 2017. Lecture Notes in Computer Science(), vol 10714. Springer, Cham. https://doi.org/10.1007/978-3-319-76270-8_35

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-76270-8_35

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-76269-2

  • Online ISBN: 978-3-319-76270-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation