Abstract
Recognizing emotion from video is an active research theme with many applications such as human-computer interaction and affective computing. The classification of emotions from facial expression is a common approach but it is sometimes difficult to differentiate genuine emotions from faked emotions. In this paper, we use a remote video based cardiac activity sensing technique to obtain physiological data to identify emotional states. We show that from the remotely sensed cardiac pulse patterns alone, emotional states can be differentiated. Specifically, we conducted an experimental study on recognizing the emotions of people watching video clips. We recorded 26 subjects that all watched the same comedy and horror video clips and then we estimated their cardiac pulse signals from the video footage. From the cardiac pulse signal alone, we were able to classify whether the subjects were watching the comedy or horror video clip. We also compare against classifying for the same task using facial action units and discuss how the two modalities compare.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Cohen, I., Sebe, N., Garg, A., Chen, L.S., Huang, T.S.: Facial expression recognition from video sequences. In: Proceedings of the IEEE International Conference on Multimedia and Expo, vol. 2, pp. 121–124 (2002)
Zhang, S., Zhao, X., Lei, B.: Facial Expression recognition based on local binary patters and local fisher discriminant analysis. In: PMC (2011)
Kim, K.H., Bang, S.W., Kim, S.R.: Emotion recognition system using short-term monitoring of physiological signals. Med. Biol. Eng. Comput. 42, 419–427 (2004)
Zong, C., Chetouani, M.: Hilbert-Huang transform based physiological signals analysis for emotion recognition. In: Proceedings of the IEEE International Symposium on Signal Processing and Information Technology, pp. 334–339 (2009)
Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: analysis of affective physiological state. Proc. IEEE Trans. Pattern Anal. Mach. Intell. 23(10), 1176–1189 (2001)
Chanel, G., Kierkels, J.J.M., Soleymani, M., Pun, T.: Short-term emotion assessment in a recall paradigm. Int. J. Hum. Comput. Stud. 67, 607–627 (2009)
Li, X., Chen, J., Zhao, G., Pietkainen, M.: Remote heart rate measurement from face videos under realistic situations. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4264–4271 (2014)
Monkaresi, H., Sazzad, M., Calvo, R.A.: Using remote heart rate measurement for affect detection. In: The Twenty-Seventh International Flairs Conference of the Florida Artificial Intelligence Research Society Conference, pp. 119–123 (2014)
Wu, H.Y., Rubinstein, M., Shih, E., Guttag, J., Durand, F., Freeman, W.T.: Eulerian video magnification for revealing subtle changes in the world. ACM Trans. Graph. 31(4), 1–8 (2012)
Kwon, S., Kim, H., Park, K.S.: Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone. In: Proceedings of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 2174–2177, August 2012
Verkruysse, W., Svaasand, L.O., Nelson, J.S.: Remote plethysmographic imaging using ambient light. Opt. Express 16, 21434–21445 (2008)
Lam, A., Kuno, Y.: Robust heart rate measurement from video using select random patches. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 3640–3648 (2015)
Poh, M.Z., McDuff, D., Picard, R.: Advancements in noncontact, multiparameter physiological measurements using a webcam. Proc. IEEE Trans. Biomed. Eng. 58(1), 7–11 (2011)
Tulyakov, S., Alameda-Pineda, X., Ricci, E., Yin, L., Cohn, J.F., Sebe, N.: Self-adaptive matrix completion for heart rate estimation from face videos under realistic conditions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016 (2016)
Kaewkannate, K., Kim, S.: A comparison of wearable fitness devices (2016). https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-016-3059-0
Evenson, K.R., Goto, M., Furberg, R.D.: Systematic review of the validity and reliability of consumer-wearable activity trackers. Int. J. Behav. Nutr. Phys. Act. 12, 159 (2015)
Balakrishnan, G., Durand, F., Guttag, J.: Detecting pulse from head motions in video. In: Proceedings of the IEEE Computer Vision and Pattern Recognition, CVPR 2013 (2013)
Chakraborty, P.R., Zhang, L., Tjondronegoro, D., Chandra, V.: Using viewer’s facial expression and heart rate for sports video highlights detection, pp. 371–378. ACM (2015)
Amos, B., Ludwiczuk, B., Satyanarayanan, M.: OpenFace: a general-purpose face recognition library with mobile applications. CMU-CS-16-118, CMU School of Computer Science, Technical report (2016)
Ekman, P., Rosenberg, E.L.: What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS). Oxford University Press, New York (1997)
Lam, A., Otsu, K., Das, K., Kuno, Y.: Towards taking pulses over youtube to determine interest in video content. In: Proceedings of the IEEE International Conference on Computer Vision (IW-FCV). IEEE (2018)
de Haan, G., Jeanne, V.: Robust pulse rate from chrominance-based rPPG. IEEE Trans. Biomed. Eng. 60(10), 2878–2886 (2013)
Das, K., Ali, S., Otsu, K., Fukuda, H., Lam, A., Kobayashi, Y., Kuno, Y.: Detecting inner emotions from video based heart rate sensing. In: 13th International Conference on Intelligent Computing, ICIC 2017, pp. 48–57 (2017)
Acknowledgments
This work was supported by JSPS KAKENHI Grant Numbers JP17K12709, JP17K18850 and the Tateisi and Technology Foundation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG, part of Springer Nature
About this paper
Cite this paper
Das, K., Lam, A., Fukuda, H., Kobayashi, Y., Kuno, Y. (2018). Classification of Emotions from Video Based Cardiac Pulse Estimation. In: Huang, DS., Gromiha, M., Han, K., Hussain, A. (eds) Intelligent Computing Methodologies. ICIC 2018. Lecture Notes in Computer Science(), vol 10956. Springer, Cham. https://doi.org/10.1007/978-3-319-95957-3_33
Download citation
DOI: https://doi.org/10.1007/978-3-319-95957-3_33
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-95956-6
Online ISBN: 978-3-319-95957-3
eBook Packages: Computer ScienceComputer Science (R0)