research-article

Mix Emotion Recognition from Facial Expression using SVM-CRF Sequence Classifier

Authors:

Dewi Yanti Liliana,

Chan Basaruddin,

M. Rahmat WidyantoAuthors Info & Claims

ICACS '17: Proceedings of the 1st International Conference on Algorithms, Computing and Systems

Pages 27 - 31

https://doi.org/10.1145/3127942.3127958

Published: 10 August 2017 Publication History

Abstract

Recently, emotion recognition has gained increasing attention in various applications related to Social Signal Processing (SSP) and human affect. The existing research is mainly focused on six basic emotions (happy, sad, fear, disgust, angry, and surprise). However human expresses many kind of emotions, including mix emotion which has not been explored due to its complexity. We model 12 types of mix emotion recognition from facial expression in a sequence of images using two-stages learning which combines Support Vector Machines (SVM) and Conditional Random Fields (CRF) as sequence classifiers. SVM classifies each image frame and produce emotion label output, subsequently it becomes the input for CRF which yields the mix emotion label of the corresponding observation sequence. We evaluate our proposed model on modified image frames of Cohn Kanade+ dataset, and on our own made mix emotion dataset. We also compare our model with the original CRF model, and our model shows a superior performance result.

References

[1]

Sumathi, C. P., Santhanam, T., and Mahadevi, M. 2012. Automatic Facial Expression Analysis A Survey. Int. J. Comput. Sci. Eng. Surv., vol. 3, no. 6 (Dec. 2012), 47--59.

[2]

Ekman, P. 1992. An argument for basic emotions, J. Cogn. Emot., vol. 6, no. 3-4 (May 1992), 169--200.

[3]

Grühn, D., Lumley M. A., Diehl M., and Labouvie-Vief G. 2013. Time-Based Indicators of Emotional Complexity: Interrelations and Correlates. Emot. Wash. DC, vol. 13, no. 2 (Apr. 2013), 226--237.

[4]

Berrios R., Totterdell P., and Kellett S. 2015. Eliciting mixed emotions: a meta-analysis comparing models, types, and measures. J. Front. Psychol., vol. 6, 428.

[5]

Izard C. E. 1992. Basic emotions, relations among emotions, and emotion-cognition relations. J. Psychol. Rev., vol. 99, no. 3, 561--565.

[6]

Valstar M. F., and Pantic M. 2012. Fully Automatic Recognition of the Temporal Phases of Facial Actions. IEEE Trans. Syst. Man Cybern. Part B Cybern., vol. 42, no. 1, (Feb. 2012), 28--43.

Digital Library

[7]

Zraqou J., Alkhadour W., and Al-Nu'aimi, A. 2013. An efficient approach for recognizing and tracking spontaneous facial expressions. Second Int. Conf. E-Learn. E-Technol. Educ. ICEEE, (23-25 September 2013) 304--307.

[8]

Van der Maaten, L., and Hendriks, E. 2012. Action unit classification using active appearance models and conditional random fields. Cognitive Processing, 13(Suppl 2), 507--518.

[9]

Siddiqi, M. H., R. Ali, Khan, A. M., Park, Y., and Lee, S. 2015. Human Facial Expression Recognition Using Stepwise Linear Discriminant Analysis and Hidden Conditional Random Fields. IEEE Trans Image Process, 24(4), (April 2015), 1386--98.

Digital Library

[10]

Lafferty, J., McCallum, A., and Pereira, F. 2001). Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the eighteenth international conference on machine learning, ICML Vol. 1, (June 2001), 282--289.

Digital Library

[11]

Sha, F. and Pereira, F., 2003, May. Shallow parsing with conditional random fields. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1 (pp. 134--141).

Digital Library

[12]

McCallum A., and Li. W, 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003 - Volume 4 (CONLL '03), Vol. 4. Association for Computational Linguistics, Stroudsburg, PA, USA, 188--191.

Digital Library

[13]

Hoefel, G., and Elkan, C. 2008. Learning a two-stage SVM/CRF sequence classifier. In Proceedings of the 17th ACM conference on Information and knowledge management (CIKM '08). ACM, New York, NY, USA, 271--278.

Digital Library

[14]

Du, S., Tao, Y. and Martinez, A.M., 2014. Compound facial expressions of emotion. Proceedings of the National Academy of Sciences, 111(15), E1454-E1462.

[15]

Cootes, T.F., Edwards, G. J., and Taylor C. J. 2001. Active Appearance Models. IEEE Trans. Pattern Anal. Mach. Intell. 23, 6 (June 2001), 681--685.

Digital Library

[16]

Tian, Y., Kanade, T., and Cohn, J. F. 2001. Recognizing Action Units for Facial Expression Analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23, 2 (February 2001), 97--115.

Digital Library

[17]

Bartlett, M.S., Littlewort, G., Fasel, I. and Movellan, J.R., 2003, June. Real Time Face Detection and Facial Expression Recognition: Development and Applications to Human Computer Interaction. In Computer Vision and Pattern Recognition Workshop, CVPRW'03. Vol. 5, pp. 53--53.

[18]

Tian, Y.L., Brown, L., Hampapur, A., Pankanti, S., Senior, A. and Bolle, R., 2003. Real world real-time automatic recognition of facial expressions. In In Proceedings of IEEE workshop on.

[19]

Li, X., Ruan, Q., Jin, Y., An, G., and Zhao, R. 2015. Fully automatic 3D facial expression recognition using polytypic multi-block local binary patterns. Signal Process. 108, C (March 2015), 297--308.

Digital Library

[20]

Brehm, J.W. and Miron, A.M., 2006. Can the simultaneous experience of opposing emotions really occur?. Motivation and Emotion, 30(1), pp. 13--30.

[21]

Smith RS, Windeatt T. Facial action unit recognition using multi-class classification. 2015. Neurocomputing 20, 150, (Feb. 2015), 440--8.

[22]

Ming, Z., Bugeau, A., Rouas, J.L. and Shochi, T., 2015, May. Facial action units intensity estimation by the fusion of features with multi-kernel support vector machine. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on (Vol. 6, pp. 1--6). IEEE.

[23]

Siddiqi, M.H., Ali, R., Khan, A.M., Park, Y.T. and Lee, S., 2015. Human facial expression recognition using stepwise linear discriminant analysis and hidden conditional random fields. IEEE Transactions on Image Processing, 24(4), pp. 1386--1398.

Digital Library

[24]

Liliana, D.Y., Widyanto, M.R. and Basaruddin, T., 2016, October. Human emotion recognition based on active appearance model and semi-supervised fuzzy C-means. In Advanced Computer Science and Information Systems (ICACSIS), 2016 International Conference on, 439--445.

[25]

Valstar, M.F. and Pantic, M., 2012. Fully automatic recognition of the temporal phases of facial actions. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 42(1), pp. 28--43.

Digital Library

[26]

Wu Y., and Ji, Q. 2015. Discriminative Deep Face Shape Model for Facial Point Detection. Int. J. Comput. Vision 113, 1 (May 2015), 37--53.

Digital Library

[27]

Heili, A., Chen, C. and Odobez, J.M., 2011, November. Detection-based multi-human tracking using a CRF model. In Computer Vision Workshops (ICCV Workshops), 2011 IEEE International Conference on (pp. 1673--1680).

[28]

Yang, B., Huang, C., and Nevatia, R. 2011. Learning affinities and dependencies for multi-target tracking using a CRF model. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (CVPR '11). IEEE Computer Society, Washington, DC, USA, 1233--1240.

Digital Library

Cited By

Rana SChaudhary RGupta MGarg P(2023)Exploring Different Techniques for Emotion Detection Through Face Recognition2023 International Conference on Advanced Computing & Communication Technologies (ICACCTech)10.1109/ICACCTech61146.2023.00128(779-786)Online publication date: 23-Dec-2023
https://doi.org/10.1109/ICACCTech61146.2023.00128
Borgalli RSurve S(2023)Review on learning framework for facial expression recognitionThe Imaging Science Journal10.1080/13682199.2023.217252670:7(483-521)Online publication date: 15-Mar-2023
https://doi.org/10.1080/13682199.2023.2172526
Sarvakar KSenkamalavalli RRaghavendra SSantosh Kumar JManjunath RJaiswal S(2023)Facial emotion recognition using convolutional neural networksMaterials Today: Proceedings10.1016/j.matpr.2021.07.29780(3560-3564)Online publication date: 2023
https://doi.org/10.1016/j.matpr.2021.07.297
Show More Cited By

Index Terms

Mix Emotion Recognition from Facial Expression using SVM-CRF Sequence Classifier
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Object recognition

Recommendations

Human-Computer Interaction Using Emotion Recognition from Facial Expression
EMS '11: Proceedings of the 2011 UKSim 5th European Symposium on Computer Modeling and Simulation

This paper describes emotion recognition system based on facial expression. A fully automatic facial expression recognition system is based on three steps: face detection, facial characteristic extraction and facial expression classification. We have ...
Natural facial expression recognition using differential-AAM and manifold learning

This paper proposes a novel natural facial expression recognition method that recognizes a sequence of dynamic facial expression images using the differential active appearance model (AAM) and manifold learning as follows. First, the differential-AAM ...
Emotion recognition using facial expression by fusing key points descriptor and texture features

Emotions have a great significance in human-to-human and in human-to-computer communication and interaction. In this paper, an effective and novel approach to recognize the emotions using facial expressions by the fusion of duplex features is proposed. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICACS '17: Proceedings of the 1st International Conference on Algorithms, Computing and Systems

August 2017

117 pages

ISBN:9781450352840

DOI:10.1145/3127942

Copyright © 2017 ACM.

© 2017 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of a national government. As such, the Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Universitas Indonesia

Conference

ICACS '17

ICACS '17: 2017 International Conference on Algorithms, Computing and Systems

August 10 - 13, 2017

Jeju Island, Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
214
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)2

Reflects downloads up to 07 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rana SChaudhary RGupta MGarg P(2023)Exploring Different Techniques for Emotion Detection Through Face Recognition2023 International Conference on Advanced Computing & Communication Technologies (ICACCTech)10.1109/ICACCTech61146.2023.00128(779-786)Online publication date: 23-Dec-2023
https://doi.org/10.1109/ICACCTech61146.2023.00128
Borgalli RSurve S(2023)Review on learning framework for facial expression recognitionThe Imaging Science Journal10.1080/13682199.2023.217252670:7(483-521)Online publication date: 15-Mar-2023
https://doi.org/10.1080/13682199.2023.2172526
Sarvakar KSenkamalavalli RRaghavendra SSantosh Kumar JManjunath RJaiswal S(2023)Facial emotion recognition using convolutional neural networksMaterials Today: Proceedings10.1016/j.matpr.2021.07.29780(3560-3564)Online publication date: 2023
https://doi.org/10.1016/j.matpr.2021.07.297
Li ZLi XXie HWang FLeng MLi QTao X(2023)A novel dropout mechanism with label extension schema toward text emotion classificationInformation Processing & Management10.1016/j.ipm.2022.10317360:2(103173)Online publication date: Mar-2023
https://doi.org/10.1016/j.ipm.2022.103173
Ramesh Chandra GGolla RGangireddy TNagula ABala D(2023)Emoji Creation from Facial Expression Detection Using CNNProceedings of Third International Conference on Advances in Computer Engineering and Communication Systems10.1007/978-981-19-9228-5_27(303-312)Online publication date: 18-Mar-2023
https://doi.org/10.1007/978-981-19-9228-5_27
Chhabra GOnyema EKumar SGoutham MMandapati SIwendi C(2022)Human Emotions Recognition, Analysis and Transformation by the Bioenergy Field in Smart Grid Using Image ProcessingElectronics10.3390/electronics1123405911:23(4059)Online publication date: 6-Dec-2022
https://doi.org/10.3390/electronics11234059
G SD SS MK T(2022)Facial Expression Recognition for Compound Emotions using Mobile Net Architecture2022 International Conference on Artificial Intelligence and Data Engineering (AIDE)10.1109/AIDE57180.2022.10060734(187-190)Online publication date: 22-Dec-2022
https://doi.org/10.1109/AIDE57180.2022.10060734
Rawal NStock-Homburg R(2022)Facial Emotion Expressions in Human–Robot Interaction: A SurveyInternational Journal of Social Robotics10.1007/s12369-022-00867-014:7(1583-1604)Online publication date: 24-Jun-2022
https://doi.org/10.1007/s12369-022-00867-0
Kamińska DAktas KRizhinashvili DKuklyanov DSham AEscalera SNasrollahi KMoeslund TAnbarjafari G(2021)Two-Stage Recognition and beyond for Compound Facial Emotion RecognitionElectronics10.3390/electronics1022284710:22(2847)Online publication date: 19-Nov-2021
https://doi.org/10.3390/electronics10222847
Navdeep PSharma NArora M(2021)Computational Learning Based Facial Emotions Recognition: A Review2021 Sixth International Conference on Image Information Processing (ICIIP)10.1109/ICIIP53038.2021.9702629(379-384)Online publication date: 26-Nov-2021
https://doi.org/10.1109/ICIIP53038.2021.9702629
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten