Gwen C. Littlewort
ABSTRACT
We present initial results from the application of an automated facial expression recognition system to spontaneous facial expressions of pain. In this study, 26 participants were videotaped under three experimental conditions: baseline, posed pain, and real pain. In the real pain condition, subjects experienced cold pressor pain by submerging their arm in ice water. Our goal was to automatically determine which experimental condition was shown in a 60 second clip from a previously unseen subject. We chose a machine learning approach, previously used successfully to categorize basic emotional facial expressions in posed datasets as well as to detect individual facial actions of the Facial Action Coding System (FACS) (Littlewort et al, 2006; Bartlett et al., 2006). For this study, we trained 20 Action Unit (AU) classifiers on over 5000 images selected from a combination of posed and spontaneous facial expressions. The output of the system was a real valued number indicating the distance to the separating hyperplane for each classifier. Applying this system to the pain video data produced a 20 channel output stream, consisting of one real value for each learned AU, for each frame of the video. This data was passed to a second layer of classifiers to predict the difference between baseline and pained faces, and the difference between expressions of real pain and fake pain. Naíve human subjects tested on the same videos were at chance for differentiating faked from real pain, obtaining only 52% accuracy. The automated system was successfully able to differentiate faked from real pain. In an analysis of 26 subjects, the system obtained 72% correct for subject independent discrimination of real versus fake pain on a 2-alternative forced choice. Moreover, the most discriminative facial action in the automated system output was AU 4 (brow lower), which all was consistent with findings using human expert FACS codes.
- Bartlett M. S., Littlewort G. C., Frank M. G., Lainscsek C., Fasel I., and Movellan J. R., "Automatic recognition of facial actions in spontaneous expressions.," Journal of Multimedia., 1(6) p. 22--35.Google Scholar
- Cohn, J. F. & Schmidt, K. L. (2004). The timing of facial motion in posed and spontaneous smiles. J. Wavelets, Multi-resolution & Information Processing, Vol. 2, No. 2, pp. 121--132.Google Scholar
- Craig K. D, Hyde S., Patrick C. J. (1991). Genuine, supressed, and faked facial behaviour during exacerbation of chronic low back pain. Pain 46:161--172.Google ScholarCross Ref
- Craig K. D, Patrick C. J. (1985). Facial expression during induced pain. J Pers Soc Psychol. 48(4):1080--91.Google ScholarCross Ref
- Donato, G., Bartlett, M. S., Hager, J. C., Ekman, P. & Sejnowski, T. J. (1999). Classifying facial actions. IEEE Trans. Pattern Analysis and Machine Intelligence, Vol. 21, No. 10, pp. 974--989. Google ScholarDigital Library
- Ekman P. and Friesen, W. Facial Action Coding System: A Technique for the Measurement of Facial Movement, Consulting Psychologists Press, Palo Alto, CA, 1978.Google Scholar
- Ekman, P. (2001). Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage. W.W. Norton, New York, USA.Google Scholar
- Ekman, P. & Rosenberg, E. L., (Eds.), (2005). What the face reveals: Basic and applied studies of spontaneous expression using the FACS, Oxford University Press, Oxford, UK.Google Scholar
- Fasel I., Fortenberry B., Movellan J. R. "A generative framework for real-time object detection and classification.," Computer Vision and Image Understanding 98, 2005. Google ScholarDigital Library
- Fishbain D. A, Cutler R., Rosomoff H. L, Rosomoff R. S. (1999). Chronic pain disability exaggeration/malingering and submaximal effort research. Clin J Pain. 15(4):244--74.Google ScholarCross Ref
- Fishbain D. A, Cutler R., Rosomoff H. L, Rosomoff R. S. (2006). Accuracy of deception judgments. Pers Soc Psychol Rev. 10(3):214--34.Google ScholarCross Ref
- Frank M. G., Ekman P., Friesen W. V. (1993). Behavioral markers and recognizability of the smile of enjoyment. J Pers Soc Psychol. 64(1):83--93.Google ScholarCross Ref
- Grossman, S., Shielder, V., Swedeen, K., Mucenski, J. (1991). Correlation of patient and caregiver ratings of cancer pain. Journal of Pain and Symptom Management 6(2), p. 53--57.Google ScholarCross Ref
- Hadjistavropoulos H. D., Craig K. D., Hadjistavropoulos T., Poole GD. (1996). Subjective judgments of deception in pain expression: accuracy and errors. Pain. 65(2-3):251--8.Google ScholarCross Ref
- Hill M. L., Craig K. D. (2002) Detecting deception in pain expressions: the structure of genuine and deceptive facial displays. Pain. 98(1-2):135--44.Google ScholarCross Ref
- Kanade, T., Cohn, J. F. and Tian, Y., "Comprehensive database for facial expression analysis," in Proceedings of the fourth IEEE International conference on automatic face and gesture recognition (FG'00), Grenoble, France, 2000, pp. 46--53. Google ScholarDigital Library
- Larochette A. C., Chambers C. T., Craig K. D. (2006). Genuine, suppressed and faked facial expressions of pain in children. Pain. 2006 Dec 15;126(1-3):64--71.Google Scholar
- Littlewort, G., Bartlett, M. S., Fasel, I., Susskind, J. & Movellan, J. (2006). Dynamics of facial expression extracted automatically from video. J. Image & Vision Computing, Vol. 24, No. 6, pp. 615--625. Google ScholarDigital Library
- Morecraft R. J, Louie J. L., Herrick J. L., Stilwell-Morecraft KS. (2001). Cortical innervation of the facial nucleus in the non-human primate: a new interpretation of the effects of stroke and related subtotal brain trauma on the muscles of facial expression. Brain 124(Pt 1):176--208.Google ScholarCross Ref
- Pantic, M., Pentland, A., Nijholt, A. & Huang, T. (2006). Human Computing and machine understanding of human behaviour: A Survey, Proc. ACM Int'l Conf. Multimodal Interfaces, pp. 239--248. Google ScholarDigital Library
- Pantic, M. F. Valstar, R. Rademaker and L. Maat, "Web-based Database for Facial Expression Analysis", Proc. IEEE Int'l Conf. Multmedia and Expo (ICME'05), Amsterdam, The Netherlands, July 2005.Google ScholarCross Ref
- Prkachin K. M. (1992). The consistency of facial expressions of pain: a comparison across modalities. Pain. 51(3):297--306.Google ScholarCross Ref
- Prkachin K. M., Schultz I., Berkowitz J., Hughes E., Hunt D. Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav Res Ther. 40(5):595--607.Google Scholar
- Rinn W. E. The neuropsyhology of facial expression: a review of the neurological and psychological mechanisms for producing facial expression. Psychol Bull 95:52--77.Google Scholar
- Schmand B., Lindeboom J., Schagen S., Heijt R., Koene T., Hamburger H. L. Cognitive complaints in patients after whiplash injury: the impact of malingering. J Neurol Neurosurg Psychiatry. 64(3):339--43.Google Scholar
- Schmidt K. L., Cohn J. F., Tian Y. (2003). Signal characteristics of spontaneous facial expressions: automatic movement in solitary and social smiles. Biol Psychol. 65(1):49--66.Google ScholarCross Ref
- Schneiderman, H. and Kanade, T. (1998). Probabilistic Modeling of Local Appearance and Spatial Relationships for Object Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 45--51. Google ScholarDigital Library
- Viola, P. & Jones, M. (2004). Robust real-time face detection. J. Computer Vision, Vol. 57, No. 2, pp. 137--154. Google ScholarDigital Library
- Vural, E., Ercil, A., Littlewort, G.C., Bartlett, M.S., and Movellan, J.R. (2007).allMachine learning systems for detecting driver drowsiness. Proceedings of the Biennial Conference on Digital Signal Processing for in-Vehicle and Mobile Systems.Google Scholar
Index Terms
- Faces of pain: automated measurement of spontaneousallfacial expressions of genuine and posed pain
Recommendations
Recognizing Action Units for Facial Expression Analysis
Most automatic expression analysis systems attempt to recognize a small set of prototypic expressions, such as happiness, anger, surprise, and fear. Such prototypic expressions, however, occur rather infrequently. Human emotions and intentions are more ...
Facial expression recognition with Convolutional Neural Networks
Facial expression recognition has been an active research area in the past 10 years, with growing application areas including avatar animation, neuromarketing and sociable robots. The recognition of facial expressions is not an easy problem for machine ...
An Approach to Automatic Recognition of Spontaneous Facial Actions
FGR '02: Proceedings of the Fifth IEEE International Conference on Automatic Face and Gesture RecognitionWe present ongoing work on a project for automatic recognition of spontaneous facial actions. Spontaneous facial expressions differ substantially from posed expressions, similar to how spontaneous speech differs from directed speech. Previous methods ...
Comments