Skip to main content

Advertisement

SpringerLink
Log in

A Survey on Image Acquisition Protocols for Non-posed Facial Expression Recognition Systems

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Several research methodologies and human face image databases have been developed based on deliberately produced facial expressions of prototypical emotions. However, real-time and spontaneous facial expression recognition cannot be adequately handled by those existing methods and datasets. To address this problem, research efforts have been made to create spontaneous facial expression image datasets as well as to develop algorithms that can process naturally induced affective behavior. This paper introduces these advances and focuses on a small and specific area of spontaneous facial expression recognition. In this paper, we are concentrating on non-posed image acquisition protocols, which strongly influence the subjects for evoking expressions as natural as possible. We categorize the acquisition protocols into four different parts: image acquisition while playing video games, watching emotional videos, during interviews and from other sources. The taxonomy of facial expression acquisition protocols tells about the typical conditions responsible for producing specific facial expressions in that condition. We also address some important design issues related to spontaneous facial expression recognition systems and list the facial expression databases, which are strictly not acted and non-posed. We also put light on the applications of spontaneously evoked facial expression acquisition and recognition because they have potential medical significance. Moreover, we provide a comprehensive analysis and summary of spontaneous facial expression recognition methods by revealing their pros and cons for future researchers.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Aghevli MA, Blanchard JJ, Horan WP (2003) The expression and experience of emotion in schizophrenia: a study of social interactions. Psychiatry Res 119(3):261–270

    Article  Google Scholar 

  2. Aina S, Zhou M, Chambers JA, Phan RC-W (2014) A new spontaneous expression database and a study of classification-based expression analysis methods, Proc. 22nd European Conf. Signal Processing, pp. 2505–2509

  3. Alves NT (2013) Recognition of static and dynamic facial expressions: a study review. Estud Psicol 18(1):125–130

    Article  Google Scholar 

  4. Antonin G, Popovici V, Thiran JP (2003) Independent component analysis and support vector machine for face feature extraction, in Proc. 4th International Conference on Audio- and Video-Based Biometric Person Authentication, pp. 111–118

  5. Ashraf AB, Lucey S, Cohn JF, Chen T, Ambadar Z, Prkachin KM, Solomon PE (2009) The painful face-pain expression recognition using active appearance models. Img and vis Comput 27(12):1788–1796

    Article  Google Scholar 

  6. Bänziger T, Mortillaro M, Scherer KR (2012) Introducing the Geneva Multimodal Expression corpus for experimental research on emotion perception. Emotion 12(5):1161–1179

    Article  Google Scholar 

  7. Baron-Cohen S, Wheelwright S, Jolliffe T (1997) Is there a “language of the eyes”? Evidence from normal adults, and adults with autism or Asperger Syndrome. Vis Cogn 4(3):311–331

    Article  Google Scholar 

  8. Bartlett MS, Littlewort G, Frank M, Lainscsek C, Fasel I, Movellan J (2005) Recognizing Facial Expression: Machine Learning and Application to Spontaneous Behavior, IEEE Conf. Computer Vision and Pattern Recognition, pp. 568–573

  9. Berenbaum H (1992) Posed facial expressions of emotion in schizophrenia and depression. Psychol Med 22(4):929–937

    Article  Google Scholar 

  10. Berenbaum H, Oltmanns TF (1992) Emotional experience and expression in schizophrenia and depression. J Abnorm Psychol 101(1):37–44

    Article  Google Scholar 

  11. Bettadapura V (2012) Facial Expression Recognition and Analysis: The State of the Art, CoRR, abs/1203.6722

  12. Biel J-I, Teijeiro-Mosquera L, Gatica-Perez D (2012) “FaceTube: predicting personality from facial expressions of emotion in online conversational video,” in Proc. ACM ICMI

  13. Blom PM, Bakkes S, Tan CT, Whiteson S, Roijers D, Valenti R, Gevers T (2014) Towards Personalised Gaming via Facial Expression Recognition, Proc. Tenth Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, pp. 30–36

  14. Cerezo E, Hupont I (2006) Emotional facial expression classification for multimodal user interfaces, Proc. 4th Int. Conf. AMDO, Port d’Andratx, Mallorca, Spain, pp. 405–413

  15. Cho S-Y, Teoh T-T, Nguwi Y-Y (2009) Development of an Intelligent Facial Expression Recognizer for Mobile Applications, New Advances in Intelligent Decision Technologies, Studies in Computational Intelligence, pp. 21–29

  16. Cohen I, Garg A, Huang TS (2000) Emotion Recognition from Facial Expressions using Multilevel HMM, presented at the Neural Inf. Process. Syst. (NIPS) Workshop Affective Comput., Colorado

  17. Cohen L, Sebe N, Garg A, Chen L, Huang T (2003) Facial expression recognition from video sequences: Temporal and static modeling. Comp Vis and Img Understanding 91(1–2):160–1878

    Article  Google Scholar 

  18. Cohn JF, Tronick EZ (1983) Three-month-old infants’ reaction to simulated maternal depression. Child Dev 54:185–193

    Article  Google Scholar 

  19. Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, Zhou F, De la Torre F (2009) Detecting depression from facial actions and vocal prosody, Proc. 3rd Int. Conf. on Affective Computing and Intelligent Interaction, Amsterdam, Netherlands

  20. Colbry D, Stockman G, Jain AK (2005) Detection of anchor points for 3d face verification. Proc. IEEE Workshop on Computer Vision and Pattern Recognition, San Diego, CA, USA

    Google Scholar 

  21. Cowing R, Douglas-Cowie E, Tsapatsoulis N, Votsis G, Kollias S, Fellenz W, Taylor JG (2001) Emotion recognition in human-computer interaction, IEEE Signal Processing Magazine, pp. 33–80

  22. De la Torre F and Cohn JF (2011) Facial Expression Analysis, in V. K. T. B. Moeslund, A. Hilton, and L. Sigal,(Eds.) Guide to Visual Analysis of Humans: Looking at People, New York: Springer-Verlag

  23. Dhall A, Goecke R, Lucey S, Gedeon T (2012) Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia 19(3):34–41

    Article  Google Scholar 

  24. Diaz RL, Wong U, Hodgins DC, Chiu CG, Goghari VM (2015) Violent Video Game Players and Non-Players Differ on Facial Emotion Recognition. Aggress Behav 9999:1–13

    Google Scholar 

  25. Douglas-Cowie E, Campbell N, Cowie R, Roach P (2003) Emotional speech: Towards a new generation of databases. Speech Comm 40:33–60

    Article  MATH  Google Scholar 

  26. G. J. Edwards, T. F. Cootes, and C.J. Taylor, ªFace Recognition Using Active Appearance Models,” Proc European Conf Computer Vision, vol. 2, pp. 581–695, 1998.

  27. Ekman P (2003) Darwin, deception, and facial expression. Ann N Y Acad Sci 1000(1):205–221

    Article  Google Scholar 

  28. Ekman P, Friesen W (1982) Felt, false, and miserable smiles. J Nonverbal Behav 6(4):238–252

    Article  Google Scholar 

  29. Ekman P, Rosenberg E (1997) What the Face Reveals: Basic and Applied Studies of Spontaneous Expression using the Facial Action Coding System (FACS). Oxford University Press, New York

    Google Scholar 

  30. Ekman P, Roper G, Hager JC (1980) Deliberate facial movement. Child Dev 51:886–891

    Article  Google Scholar 

  31. Ekman P, Hager J, Friesen WV (1981) The symmetry of emotional and deliberate facial actions. Psychophysiology 18(2):101–106

    Article  Google Scholar 

  32. Ekman P, Friesen WV, Ellsworth P (1982) What are the rela-tive contributions of facial behavior and contextual information to the judgment of emotion? In P. Ekman (Ed.), Emotion in the human face, Cambridge: Cambridge University Press, pp. 111–127

  33. Eysenc MW (2004) Psychology: An International Perspective. Psychology Press, UK

    Google Scholar 

  34. Fasel B, Luettin J (2003) Automatic facial expression analysis: a survey. Pattern Recogn 36:259–275

    Article  MATH  Google Scholar 

  35. Fernandez-Dols JM, Sierra B, Ruiz-Belda MA (1993) On the clarity of expressive and contextual information in the recognition of emotions: A methodological critique. European J. of Soc. Psychol. 23:195–202

    Article  Google Scholar 

  36. Fridlund A (1994) Human facial expression: an evolutionary view. Academic Press, New York

    Google Scholar 

  37. Gaebel W, Wolwer W (2004) Facial expressivity in the course of schizophrenia and depression. Eur Arch Psychiatry Clin Neurosci 254(5):335–342

    Article  Google Scholar 

  38. Gajsek R, Struc V, Mihelic F, Podlesek A, Komidar L, Socan G, Bajec B (2009) Multi-modal emotional database: AvID. Informatica 33:101–106

    Google Scholar 

  39. R. Gajšek, V. Štruc, B. Vesnicer, A. Podlesek, L. Komidar, and F. Mihelič (2009) Analysis and assessment of AvID: multi-modal emotional database, Proc. Int. Conf. Text, Speech and Dialogue, Pilsen, Czech Republic, pp. 266–273

  40. Gan Q, Wu C, Wang S (2015) Posed and Spontaneous Facial Expression Differentiation Using Deep Boltzmann Machines, Int. Conf. Affective Computing and Intelligent Interaction, vol. 643–648

  41. Gilbert DT, Krull DS (1988) Seeing less and knowing more: The benefits of perceptual ignorance. J Pers Soc Psychol 54:193–201

    Article  Google Scholar 

  42. Grafsgaard JF, Wiggins JB, Boyer KE, Wiebe EN, Lester JC (2013) Automatically recognizing facial expression: predicting engagement and frustration, Proc. Int. Conf. Educational Data Mining

  43. Grimm M, Kroschel K, Narayanan S (2008) The Vera am Mittag German audio-visual emotional speech database, IEEE Int. Conf. Multimedia and Expo, Hannover, pp. 865–868

  44. Gritti T, Shan C, Jeanne V, Braspenning R (2008) Local Features based Facial Expression Recognition with Face Registration Errors, in Proc. 8th IEEE Int. Conf. Auto. Face Gesture Recog., Amsterdam, pp. 1–8

  45. Hager JC, Ekman P (1983) The inner and outer meanings of facial expressions, in: J. T. Cacioppo and R. E. petty (Eds.), Soc. Psychophysiology, the Guilford Press, Newyork, pp. 287–307

  46. Hall JA, Bernieri FJ, Carney DR (2005) Nonverbal behavior and interpersonal sensitivity, In J. A. Harrigan, R. Rosenthal, & K. R. Scherer (Eds.), The new handbook of methods in nonverbal behavior research, Oxford: Oxford University Press, pp. 237–281

  47. Hammal Z, Couvreur L, Caplier A Rombaut M (2005) Facial Expressions Recognition Based on the Belief Theory: Comparison with Diferent Classifiers, Proc. 13th Int. Conf. Image Analysis and Processing, Italy

  48. Hammal Z, Kunz M, Arguin M, Gosselin F (2008) Spontaneous Pain Expression Recognition in Video Sequences, Proc. Int. Conf. Visions of Computer Science, pp. 191–210

  49. Happy SL, Patnaik P, Routray A, Guha R (2017) The Indian Spontaneous Expression Database for Emotion Recognition. IEEE Trans. Affective Computing 8(1):131–142

    Article  Google Scholar 

  50. He M, Wang S, Liu Z, Chen X (2013) Analyses of the Differences between Posed and Spontaneous Facial Expressions, Humaine Association Conference on Affective Computing and Intelligent Interaction (ACII), Geneva, Switzerland, pp. 79–8

  51. Hess U, Kleck RE (1990) Differentiating emotion elicited and deliberate emotional facial expressions. European J of Soc Psychol 20:369–385

    Article  Google Scholar 

  52. Hess U, Kleck RE (2005) Differentiating emotion elicited and deliberate emotional facial expressions, in P. Ekman & E. L. Rosenberg (Eds.), What the face reveals: Basic and applied studies of spontaneous expression using the facial action coding system (FACS) (2nd ed.) New York: Oxford University Press, pp. 271–286

  53. M. E. Hoque, M. Courgeon, J.-C. Martin, B. Mutlu, and R. W. Picard (2013) MACH: My automated conversation coacH, Proc. ACM Int. Jnt. Conf. Pervasive and ubiquitous computing, pp. 697–706

  54. Hussain A, Khan MS, Nazir M, Iqbal MA (2012) Survey of various feature extraction and classification techniques for facial expression recognition, Proc.11th WSEAS international conference on Electronics, Hardware, Wireless and Optical Communications, and proceedings of the 11th WSEAS international conference on Signal Processing, Robotics and Automation, and proceedings of the 4th WSEAS international conference on Nanotechnology, Stevens Point, Wisconsin, USA, pp. 138–142

  55. Iwase M, Ouchi Y, Okada H, Yokoyama C, Nobezawa S, Yoshikawa E et al (2002) Neural substrates of human facial expression of pleasant emotion induced by comic films: a PET Study. Neuroimage 17(2):758–768

    Article  Google Scholar 

  56. Jakobs E, Manstead ASR, Fischer AH (1999) Social motives and emotional feelings as determinants of facial displays: the case of smiling. Personal Soc Psychol Bull 25:424–435

    Article  Google Scholar 

  57. Joho H, Jose JM, Valenti R, Sebe N (2009) Exploiting facial expressions for affective video summarisation, in Proc. ACM Int. Conf. Image and Video Retrieval, New York, USA

  58. Jones V (2001) Robust real time object detection, Proc. 2nd Int. Workshop Statistical and Computational Theories of Vision

  59. el Kaliouby R, Teeters A (2007) Eliciting, Capturing and Tagging Spontaneous Facial Affect in Autism Spectrum Disorder, Proc. 9th Int. Conf. Multimodal interfaces, NY, USA, pp. 46–53

  60. Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions. Advances in Visual Computing:368–377

  61. Kirsh SJ, Mounts JR (2007) Violent video game play impacts facial emotion recognition. Aggress Behav 33(4):353–358

    Article  Google Scholar 

  62. Kleck RE, Vaughen RC, Cartwright-Smith J, Vaughan KB, Colby CZ, Lanzetta JT (1976) Effects of being observed on expressive, subjective and physiological responses in painful stimuli. J Pers Soc Psychol 34:1211–1218

    Article  Google Scholar 

  63. Knapp ML, Hall J (2005) Nonverbal communication in human interaction. Holt, Rinehart and Winston, New York

  64. Ko BC (2018) A Brief Review of Facial Emotion Recognition Based on Visual Information. Sensors 18:401

    Article  Google Scholar 

  65. Kohlera CG, Martina EA, Milonovaa M, Wangb P, Verma R, Brensingera CM, Bilkera W, Gura RE, Gura RC (2008) Dynamic evoked facial expressions of emotions in schizophrenia. Schizophr Res 105(1–3):30–39

    Article  Google Scholar 

  66. Korb S, Grandjean D, Scherer K (2008) Investigating the production of emotional facial expressions: a combined electroencephalographic (EEG) and electromyographic (EMG) approach, IEEE Int. Conf. Automatic Face & Gesture Recognition, Amsterdam, pp. 1–6

  67. Kring AM, Neale JM (1996) Do schizophrenic patients show a disjunctive relationship among expressive, experiential, and psychophysiological components of emotion? J Abnorm Psychol 105(2):249–257

    Article  Google Scholar 

  68. Kring AM, Kerr SL, Smith DA, Neale JM (1993) Flat affect in schizophrenia does not reflect diminished subjective experience of emotion. J Abnorm Psychology 102(4):507–517

    Article  Google Scholar 

  69. Krumhuber EG, Skora L, Küster D, Fou L (2017) A Review of Dynamic Datasets for Facial Expression Research. Emot Rev 9(3):280–292

    Article  Google Scholar 

  70. Kumar S (2015) Facial expression recognition: A review, Proc. Nat. Conf. Cloud Computing and Big Data, Shanghai, China, pp. 159–162

  71. Kunz M, Scharmann S, Hemmeter U, Schepelmann K, Lautenbacher S (2007) The facial expression of pain in patients with dementia. Pain 133(1–3):221–228

    Article  Google Scholar 

  72. Lee KK, Xu Y (2003) Real-time Estimation of Facial Expression Intensity Proc. IEEE Int. Conf. Robotics & Automation, pp. 2567–2572

  73. Lehr VT, Zeskind PS, Ofenstein JP, Cepeda E, Warrier I, Aranda JV (2007) Neonatal facial coding system scores and spectral characteristics of infant crying during newborn circumcision. Clin J Pain 23(5):417–424

    Article  Google Scholar 

  74. Li Y, Mavadati SM, Mahoor MH, Zhao Y, Ji Q (2015) Measuring the intensity of spontaneous facial action units with dynamic Bayesian network. Pattern Recogn 48:3417–3427

    Article  Google Scholar 

  75. Littlewort G, Whitehill J, Wu T, Fasel I, Frank M, Movellan J, Bartlett M, (2011)“The computer expression recognition toolbox (CERT),” Int. Conf. Automatic Face & Gesture Recognition and Workshops, Available: https://doi.org/10.1109/fg.2011.5771414

  76. Loconsole C, Chiaradia D, Bevilacqua V, A. Frisoli (2014) Real-Time Emotion Recognition: An Improved Hybrid Approach for Classification Performance, Proc. 10th Inter. Conf. ICIC, Taiyuan, China, pp. 320–331

  77. Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: The unbc-mcmaster shoulder pain expression archive database. Proc IEEE Int Conf Automatic Face & Gesture Recognition and Workshops:57–64

  78. Mahlke S, Minge M, Thüring M (2006) Measuring multiple components of emotions in interactive contexts, in Proc. Extended Abstracts on Human Factors in Computing, pp. 1061–1066

  79. Malatesta C, Izard C (1984) The facial expression of emotion in young, middle-aged, and older adults. In: Malatesta C, Izard C (eds) Emotion in Adult Development. Sage, Beverly Hills, CA

    Google Scholar 

  80. Mandal MK, Pandey R, Prasad AB (1998) Facial Expressions of Emotions and Schizophrenia: A Review. Schizophr Bull 24(1):399–412

    Article  Google Scholar 

  81. Marrero-Fernández P, Montoya-Padrón A, Jaume-i-Capó A, Rubio JMB (2014) Evaluating the Research in Automatic Emotion Recognition. IETE Tech Rev 31(3):220–232

    Article  Google Scholar 

  82. Martin CC, Borod JC, Alpert M, Brozgold A, Welkowitz J (1990) Spontaneous Expression of Facial Emotion in Schizophrenic and Right brain-Damaged Patients. J Commun Disord 23:287–301

    Article  Google Scholar 

  83. Martin C, Werner U, Gross H-M (2008) A Real-time Facial Expression Recognition System based on Active Appearance Models using Gray Images and Edge Images, Proc. 8th IEEE Int. Conf. Automatic Face & Gesture Recognition, Amsterdam, pp. 1–6

  84. Mavadati SM, Mahoor MH, Bartlett K, Trinh P, Cohn JF (2013) DISFA: A Spontaneous Facial Action Intensity Database. IEEE Trans. Affective Computing 4(2):151–160

    Article  Google Scholar 

  85. McDuff D, El Kaliouby R, Senechal T, Amr M, Cohn JF, Picard R (2013) Affectiva-MIT Facial Expression Dataset (AM-FED): Naturalistic and Spontaneous Facial Expressions Collected “In-the-Wild”, IEEE Conf. Computer Vision and Pattern Recognition Workshops, pp. 881–888

  86. McKeown G, Valstar M, Cowie R, Pantic M, Schröder M The SEMAINE Database: Annotated Multimodal Records of Emotionally Colored Conversations between a Person and a Limited Agent. IEEE Trans Affect Comput 3(1):5–17

  87. Mehrabian A (1968) Communication without words. Psychol Today 2(4):53–56

    Google Scholar 

  88. Michel P and Kaliouby R (2003) Real time facial expression recognition in video using support vector machines, in Proc. 5th Int. Conf. Multimodal interfaces, Vancouver, BC, Canada, pp. 258–264

  89. Moridis CN, Economides AA (2012) Affective Learning: Empathetic Agents with Emotional Facial and Tone of Voice Expressions. IEEE Trans on Affective Computing 3(3):260–272

    Article  Google Scholar 

  90. Murthy GRS, Jadon RS (2009) Effectiveness of Eigenspaces for Facial Expressions Recognition. Int J Computer Theory and Engineering 1(5):1793–8201

    Google Scholar 

  91. Naim I, Tanveer MI, Gildea D, Hoque ME (2015) Automated Prediction and Analysis of Job Interview Performance: The Role of What You Say and How You Say It, Proc. IEEE Int. Conf. and Workshops on Automatic Face and Gesture Recognition (FG), Ljubljana, Slovenia

  92. K. Nosu and T. Kurokawa, Facial Tracking for an Emotion-Diagnosis Robot to Support e-Learning,” Proc. Int. Conf. Mach. Learning and Cyber, pp. 3811–3816, 2006.

  93. Nosu K, Kurokawa T, Horita H, Ohhazama Y, Takeda H (2007) Real Time Emotion-Diagnosis of Video Game Players from their Facial Expressions and its Applications to Voice Feed-Backing to Game Players, Int. Conf. Mach. Learn. and Cyber., Hong Kong, pp. 2208–2212

  94. O’Toole AJ, Harms J, Snow SL, Hurst DR, Pappas MR, Ayyad JH, Abdi H (2005) A video database of moving faces and people. IEEE Trans Pattern Anal Mach Intell 27(5):812–816

    Article  Google Scholar 

  95. Ouellet S (2014) Real-time emotion recognition for gaming using deep convolutional network features, CoRR abs/1408.3750

  96. Pantic M, Rothkrantz LJM (2000) Automatic Analysis of Facial Expressions: The State of the Art. IEEE Trans Pattern Anal Mach Intell 22(12):1424–1445

    Article  Google Scholar 

  97. Pantic M, Rothkrantz LJM (2000) Expert System for Automatic Analysis of Facial Expression. J Image and Vision Computing 18(11):881–905

    Article  Google Scholar 

  98. Petridis S, Martinez B, Pantic M (2013) The MAHNOB Laughter database. Image Vis Comput 31:186–202

    Article  Google Scholar 

  99. F. Ringeval, A. Sonderegger, J. Sauer and D. Lalanne (2013) Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions, Proc. IEEE Int. Conf. on Automatic Face and Gesture Recognition (FG)

  100. Ryu YS, Oh SY (2002) Automatic extraction of eye and mouth fields from a face image using eigenfeatures and ensemble networks. Appl Intell 17:171–185

    Article  MATH  Google Scholar 

  101. Sariyanidi E, Gunes H, Cavallaro A (2015) Automatic Analysis of Facial Affect: A Survey of Registration, Representation, and Recognition. IEEE Trans on Pattern Analysis And Mach Intel 37(6):1113–1133

    Article  Google Scholar 

  102. Scherer KR, Ceschi G (1997) Lost Luggage: A Field Study of Emotion-Antecedent Appraisal. Motiv Emot 21(3):211–235

    Article  Google Scholar 

  103. Sebe N, Lew MS, Cohen I, Sun Y, Gevers T, Huang TS (2004) Authentic Facial Expression Analysis, Proc. IEEE Int. Conf. Automatic Face and Gesture Recognition, pp. 517–522

  104. Seckington M (2011) Using Dynamic Bayesian Networks for Posed versus Spontaneous Facial Expression Recognition Master Thesis, Department of Computer Science, Delft University of Technology

  105. Sikka K, Dhall A, Bartlett M (2015) Exemplar Hidden Markov Models for Classification of Facial Expressions in Videos, IEEE Conf. Computer Vision and Pattern Recognition Workshops, Boston, MA, pp. 18–25

  106. Sneddon I, McRorie M, McKeown G, Hanratty J (2012) The Belfast Induced Natural Emotion Database. IEEE Trans. on Affective Computing 3(1):32–41

    Article  Google Scholar 

  107. Suk M, Prabhakaran B (2014) Real-time Mobile Facial Expression Recognition System – A Case Study, Proc. Computer Vision and Pattern Recognition, Columbus, OH, pp. 132–137

  108. Sun Y, Sebe N, Lew MS, Gevers T (2004) Authentic Emotion Detection in Real-time Video, in: Proc. Computer Vision in Human-Computer Interaction, ECCV 2004 Workshop on HCI, Prague, Czech Republic

  109. Sun Y, Akansu AN, Cicon JE (2014) The power of fear: Facial emotion analysis of CEOs to forecast firm performance, Proc. IEEE 15th International Conf. Information Reuse and Integration, Redwood City, CA, pp. 695–702

  110. Sung J, Kim D (2009) Real-time facial expression using STAAM and layered GDA classifier. Image and Vis Comput 27(9):1313–1325

    Article  Google Scholar 

  111. Tan CT, Rosser D, Bakkes S, Pisan Y (2012) A feasibility study in using facial expressions analysis to evaluate player experiences, in Proc. 8th Australasian Conf. Interactive Entertainment: Playing the System, 5

  112. Tcherkassof A, Dupre D, Meillon B, Mandran N, Dubois M, Adam J (2013) DynEmo: A video database of natural facial expressions of emotions. Int J Multimedia & Its Applications 5(5):61–80

    Article  Google Scholar 

  113. Teijeiro-Mosquera L, Biel J-I, Alba-Castro JL, Gatica-Perez D (2015) What Your Face Vlogs About: Expressions of Emotion and Big-Five Traits Impressions in YouTube. IEEE Trans On Affective Computing 6(2):193–205

    Article  Google Scholar 

  114. Tian Y, Cohn JF, Kanade T (2005) Facial expression recognition, in Handbook of face recognition, S. Z. Li, and A. K. Jain (Eds.), New York: Springer, pp. 247–276

  115. Turan C, Lam K-M, He X (2015) Facial expression recognition with emotion-based feature fusion, Proc. Annual Summit and Conf. on Signal and Information Processing Association, Hong Kong, China

  116. Tzirakis P, Trigeorgis G, Nicolaou MA, Schuller BW, Zafeiriou S (2017) End-to-End Multimodal Emotion Recognition Using Deep Neural Networks. IEEE Journal of Selected Topics in Signal Processing 11(8):1301–1309

    Article  Google Scholar 

  117. Valstar MF, Pantic M, Ambadar Z, Cohn JF (2006) Spontaneous vs. posed facial behavior: automatic analysis of brow actions, Proc. 8th Int. Conf. Multimodal interfaces, pp. 162–170

  118. Wallhoff F, Schuller B, Hawellek M, Rigoll G (2006) Efficient recognition of authentic dynamic facial expressions on the feedtum database, IEEE Int. Conf. Multimedia and Expo, Toronto, Ont., pp. 493–496

  119. Wang S, Liu Z, Wang Z, Wu G, Shen P, He S, Wang X (2013) Analyses of a multi-modal spontaneous facial expression database. IEEE Trans Affective Computing 4(1):34–46

    Article  Google Scholar 

  120. Wang S, Wu C, He M, Wang J, Ji Q (2015) Posed and spontaneous expression recognition through modeling their spatial patterns. Mach Vis Appl 26:219–231

    Article  Google Scholar 

  121. Wanlnlhoff F (2006) Facial Expressions and Emotion Database, Available at: http://www.mmk.ei.tum.de/~waf/fgnet/feedtum.html, Technische Universität München

  122. Wild B, Rodden FA, Rapp A, Erb M, Grodd W, Ruch W (2006) Humor and smiling: cortical regions selective for cognitive, affective, and volitional components. Neurology 66(6):887–893

    Article  Google Scholar 

  123. Wu C, Wang S (2016) Posed and Spontaneous Expression Recognition through Restricted Boltzmann Machine, Proc. 22nd Int. Conf. MultiMedia Modeling, Miami, USA, pp. 127–137

  124. Xue Z, Li SZ, Teoh EK (2003) Bayesian shape model for facial feature extraction and recognition. Pattern Recogn 36:2819–2833

    Article  MATH  Google Scholar 

  125. Yannakakis GN, Hallam J (2005) A scheme for creating digital entertainment with substance, Proc. Reasoning, Representation, and Learning in Computer Games, Int. Jnt. Conf. Artificial Intelligence, pp. 119–124

  126. Yeasin M, Bullot B, Sharma R (2006) Recognition of Facial Expressions and Measurement of Levels of Interest from Video. IEEE Trans Multimedia 8(3):500–508

    Article  Google Scholar 

  127. Zaman B, Shrimpton-Smith T (2006) The face reader: measuring instant fun of use, in Proc. 4th Nordic conf. Human-computer interaction: changing roles, pp. 457–460

  128. Zeng Z, Hu Y, Roisman GI, Wen Z, Fu Y, Huang TS (2007) Audio-Visual Spontaneous Emotion Recognition. Artificial Intelligence for Human Computing, Lecture Notes in Computer Science:72–90

  129. Zeng Z, Pantic M, Roisman GI, Huang TS (2009) A Survey of Affect Recognition Methods: Audio,Visual, and Spontaneous Expressions. IEEE Trans on Pattern Anal And Mach Intel 31(1):39–58

    Article  Google Scholar 

  130. Zhalehpour S, Onder O, Akhtar Z, Erdem CE (2017) BAUM-1: A Spontaneous Audio-Visual Face Database of Affective and Mental States. IEEE Trans Affect Comput 8(03):300–313

    Article  Google Scholar 

  131. Zhan C, Li W, Ogunbona P, Safaei F (2008) A real-time facial expression recognition system for online games. Int J Computer Games Techn 10:1–7

    Article  Google Scholar 

  132. Zhang Z (1998) Comparison between Geometry-Based and Gabor-wavelet-based Facial Expression Recognition Using Multi-layer Perception, Proc. Int. Conf. Auto. Face Gesture Recog., Nara, pp. 454–459

  133. Zhang Y, Ji Q (May 2005) Active and dynamic information fusion for facial expression understanding from image sequences. IEEE Trans Pattern Analysis and Machine Intelligence 27(5):699–714

    Article  Google Scholar 

  134. Zhang Z, Lyons M, Schuster M, Akamatsu S (1998) Comparison between Geometry-based and Gabor wavelets-based facial expression recognition using multilayer perception, Proc. Int. Conf. Automatic Face and Gesture Recognition, pp. 454–459

  135. Zhang L, Tjondronegoro D, Chandran V (2012) Discovering the Best Feature Extraction and Selection Algorithms For Spontaneous Facial Expression Recognition, Proc. IEEE Int. Conf. Multimedia and Expo (ICME), Melbourne, VIC, pp. 1027–1032

  136. Zhang W, Zhang Y, Mab L, Guan J, Gong S (2015) Multimodal learning for facial expression recognition. Pattern Recogn 48:3191–3202

    Article  Google Scholar 

  137. Zhao X, Zhang S (2016) A Review on Facial Expression Recognition: Feature Extraction and Classification. IETE Tech Rev 33(5):505–517

    Article  MathSciNet  Google Scholar 

  138. Zheng D, Zhao Y, Wang J (2004) Features extraction using a Gabor filter family, Proc. Sixth IASTED International Conference Signal and Image Processing, Hawaii, USA

  139. Zhou X, Huang X, Wang Y (2004) Real-time Facial Expression Recognition in the Interactive Game Based on Embedded Hidden Markov Model, Proc. Int. Conf. Comp. Graphics, Imaging and Visualization, pp. 144–148

Download references

Acknowledgments

The first author is grateful to Department of Science and Technology (DST), Government of India for providing her Junior Research Fellowship (JRF) under DST-INSPIRE fellowship program (No. IF131067).

Author information

Authors and Affiliations

  1. Computer Science & Engineering Department, Tripura University (A Central University), Suryamaninagar, 799022, India

    Priya Saha

  2. Computer Science & Engineering Department, Jadavpur University, Kolkata, 700032, India

    Debotosh Bhattacharjee & Mita Nasipuri

  3. Physics Department, Tripura University (A Central University), Suryamaninagar, 799022, India

    Barin Kumar De

Authors
  1. Priya Saha
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Debotosh Bhattacharjee
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Barin Kumar De
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Mita Nasipuri
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Priya Saha.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Saha, P., Bhattacharjee, D., De, B.K. et al. A Survey on Image Acquisition Protocols for Non-posed Facial Expression Recognition Systems. Multimed Tools Appl 78, 23329–23368 (2019). https://doi.org/10.1007/s11042-019-7596-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-019-7596-2

Keywords

Search

Navigation