Skip to main content

Advertisement

SpringerLink
Log in

Optimized Anfis Model with Hybrid Metaheuristic Algorithms for Facial Emotion Recognition

  • Published:
International Journal of Fuzzy Systems Aims and scope Submit manuscript

Abstract

Emotion recognition from facial images is an important and active area of research. Facial features are widely used in computer vision for emotion interpretation, cognitive science, and social interaction. To obtain accurate analysis of facial expressions (happy, angry, sad, surprised, disgusted, fearful, and neutral), a complex method based on human–computer interaction and data is required. It is still difficult to develop an effective and computationally simple mechanism for feature selection and emotion classification. In this paper, an emotion recognition model using adaptive neuro-fuzzy inference system optimized with particle swarm optimization is proposed. The proposed model was compared with many classification algorithms (ANNs, SVMs, and k-Nearest Neighbor (k-NN) and their subcomponents). The confusion matrix was used to evaluate the performance of these classifiers. The proposed model was evaluated using the MUG database. The model achieved a prediction accuracy of 99.6%.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

References

  1. Ekman, P., Rosenberg, E.L.: What the face reveals: basic and applied studies of spontaneous expression using the facial action coding system (FACS). Oxford University Press, Oxford (2012). https://doi.org/10.1093/acprof:oso/9780195179644.001.0001

    Book  Google Scholar 

  2. Pantic, M., Rothkrantz, L.Ü.M.: Automatic analysis of facial expressions: the state of the art. IEEE Trans. Pattern Anal. Mach. Intell. 22(12), 1424–1445 (2000). https://doi.org/10.1109/34.895976

    Article  Google Scholar 

  3. Abdurrahim, S.H., Samad, S.A., Huddin, A.B.: Review on the effects of age, gender, and race demographics on automatic face recognition. Vis. Comput. 34, 1617–1630 (2018). https://doi.org/10.1007/s00371-017-1428-z

    Article  Google Scholar 

  4. Murugappan, M., Mutawa, A.: Facial geometric feature extraction based emotional expression classification using machine learning algorithms. PLOS ONE (2021). https://doi.org/10.1371/journal.pone.0247131

    Article  Google Scholar 

  5. Hu, T., de Silva, L.C., Sengupta, K.: A hybrid approach of NN and HMM for facial emotion classification. Pattern Recognit. Lett. 23(11), 1303–1310 (2002). https://doi.org/10.1016/s0167-8655(02)00079-x

    Article  MATH  Google Scholar 

  6. Lien, J.J.J., Kanade, T., Cohn, J.F., Li, C.C.: Detection, tracking, and classification of action units in facial expression. Robot. Auton. Syst. 31(3), 131–146 (2000). https://doi.org/10.1016/s0921-8890(99)00103-7

    Article  Google Scholar 

  7. Boughida, A., Kouahla, M.N., Lafifi, Y.: A novel approach for facial expression recognition based on gabor filters and genetic algorithm. Evol. Syst. 13(2), 331–345 (2022). https://doi.org/10.1007/S12530-021-09393-2/TABLES/12

    Article  Google Scholar 

  8. Hussein, H.I., Dino, H.I., Mstafa, R.J., Hassan, M.M.: Person-independent facial expression recognition based on the fusion of HOG descriptor and cuttlefish algorithm. Multimed. Tools Appl. 81(8), 11563–11586 (2022). https://doi.org/10.1007/S11042-022-12438-6/TABLES/5

    Article  Google Scholar 

  9. Ge, H., Zhu, Z., Dai, Y., Wang, B., Wu, X.: Facial expression recognition based on deep learning. Comput. Methods Programs Biomed. 215, 106621 (2022). https://doi.org/10.1016/J.CMPB.2022.106621

    Article  Google Scholar 

  10. Fang, H., et al.: Facial expression recognition in dynamic sequences: an integrated approach. Pattern Recogn. 47(3), 1271–1281 (2014). https://doi.org/10.1016/J.PATCOG.2013.09.023

    Article  Google Scholar 

  11. Lien, J.J, Cohn, J.F, Kanade, T, Li, C.C.: “Automated facial expression recognition based on FACS action units.” In: Proceedings—3rd IEEE International Conference on Automatic Face and Gesture Recognition, FG 1998, pp. 309–395 (1998). https://doi.org/10.1109/AFGR.1998.670980.

  12. Tian, Y.L., Kanade, T., Conn, J.F.: Recognizing action units for facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 23(2), 97 (2001). https://doi.org/10.1109/34.908962

    Article  Google Scholar 

  13. Tian, Y., Kanade, T., Colin, J.F.: “Recognizing action units for facial expression analysis.” pp. 32–66, (2002). https://doi.org/10.1142/9789812778543_0002.

  14. Donate, G., Bartlett, M.S., Hager, J.C., Ekman, P., Sejnowski, T.J.: Classifying facial actions. IEEE Trans. Pattern Anal. Mach. Intell. 21(10), 974–989 (1999). https://doi.org/10.1109/34.799905

    Article  Google Scholar 

  15. Essa, I.A., Pentland, A.P.: Coding, analysis, interpretation, and recognition of facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 19(7), 757–763 (1997). https://doi.org/10.1109/34.598232

    Article  Google Scholar 

  16. Fasel, B., Luettin, J.: Automatic facial expression analysis: a survey. Pattern Recogn. 36(1), 259–275 (2003). https://doi.org/10.1016/S0031-3203(02)00052-3

    Article  MATH  Google Scholar 

  17. Su, Y.S., Suen, H.Y., Hung, K.E.: Predicting behavioral competencies automatically from facial expressions in real-time video-recorded interviews. J. Real-Time Image Proc. 18(4), 1011–1021 (2021). https://doi.org/10.1007/S11554-021-01071-5/TABLES/5

    Article  Google Scholar 

  18. Revina, I.M., Emmanuel, W.R.S.: A survey on human face expression recognition techniques. J. King Saud Univ.—Comput. Inf. Sci. 33(6), 619–628 (2021). https://doi.org/10.1016/J.JKSUCI.2018.09.002

    Article  Google Scholar 

  19. Dornaika, F., Moujahid, A., Raducanu, B.: Facial expression recognition using tracked facial actions: classifier performance analysis. Eng. Appl. Artif. Intell. 26(1), 467–477 (2013). https://doi.org/10.1016/j.engappai.2012.09.002

    Article  Google Scholar 

  20. Loconsole, C., Miranda, C.R., Augusto, G., Frisoli, A., Orvalho, V.: Real-time emotion recognition: novel method for geometrical facial features extraction. VISAPP 2014 - Proc. 9th Int. Conf. Comp. Vision Theory. Appl. 1, 378–385 (2014). https://doi.org/10.5220/0004738903780385.

  21. Ekman, P., Friesen, W.V.: Constants across cultures in the face and emotion. J. Pers. Soc. Psychol. (1971). https://doi.org/10.1037/h0030377

    Article  Google Scholar 

  22. Jones, A.C., Gutierrez, R., Ludlow, A.K.: Emotion production of facial expressions: a comparison of deaf and hearing children. J. Commun. Disord. 92, 106113 (2021). https://doi.org/10.1016/j.jcomdis.2021.106113

    Article  Google Scholar 

  23. Clark, E.A., et al.: The facial action coding system for characterization of human affective response to consumer product-based stimuli: a systematic review. Front Psychol. 11, 1–21 (2020). https://doi.org/10.3389/fpsyg.2020.00920

    Article  Google Scholar 

  24. A. C. Network (2021) “Deep-emotion: facial expression recognition using” pp. 1–16 (2021).

  25. Rabhi, Y., Mrabet, M., Fnaiech, F.: A facial expression controlled wheelchair for people with disabilities. Comput. Methods Programs Biomed. 165, 89–105 (2018). https://doi.org/10.1016/j.cmpb.2018.08.013

    Article  Google Scholar 

  26. Dirik, M., Castillo, O., Kocamaz, A.F.: Emotion recognition based on interval type-2 fuzzy logic from facial expression. J. Soft Comput. Artif. Intell. 1(1), 1–17 (2020)

    Google Scholar 

  27. Laskar, B.Z., Ashutosh, Majumder, S.: Artificial neural networks and gene expression programing based age estimation using facial features. J. King Saud Univ.—Comput. Inf. Sci. 27(4), 458–467 (2015). https://doi.org/10.1016/j.jksuci.2014.06.017

    Article  Google Scholar 

  28. Ekman, P.: Facial expression and emotion. Am. Psychol. 48(4), 384–392 (1993). https://doi.org/10.1037/0003-066X.48.4.384

    Article  Google Scholar 

  29. Valstar, M., Pantic, M.: “Fully automatic facial action unit detection and temporal analysis,” In: Proceedings of the IEEE computer society conference on computer vision and pattern recognition. (2006).https://doi.org/10.1109/CVPRW.2006.85.

  30. Basser, H., et al.: Hybrid ANFIS-PSO approach for predicting optimum parameters of a protective spur dike. Appl. Soft Comput. 30, 642–649 (2015). https://doi.org/10.1016/j.asoc.2015.02.011

    Article  Google Scholar 

  31. Zanganeh, M.: Improvement of the ANFIS-based wave predictor models by the particle Swarm optimization. J. Ocean Eng. Sci. 5(1), 84–99 (2020). https://doi.org/10.1016/j.joes.2019.09.002

    Article  Google Scholar 

  32. Shamshirband, S., Hadipoor, M., Baghban, A., Mosavi, A., Bukor, J., Várkonyi-Kóczy, A.R.: Developing an ANFIS-PSO model to predict mercury emissions in combustion flue gases. Mathematics (2019). https://doi.org/10.3390/math7100965

    Article  Google Scholar 

  33. Jang, J.S.R.: ANFIS: adaptive-network-based fuzzy inference system. IEEE Trans. Syst. Man Cybern. 23(3), 665–685 (1993). https://doi.org/10.1109/21.256541

    Article  Google Scholar 

  34. Rathnayake, N., Dang, T.L., Hoshino, Y.: A novel optimization algorithm: cascaded adaptive neuro-fuzzy inference system. Int. J. Fuzzy Syst. 23(7), 1955–1971 (2021). https://doi.org/10.1007/S40815-021-01076-Z/FIGURES/20

    Article  Google Scholar 

  35. AbuHassan, A., Alshayeb, M., Ghouti, L.: Detection of design smells using adaptive neuro-fuzzy approaches. Int. J. Fuzzy Syst. (2022). https://doi.org/10.1007/S40815-022-01248-5/FIGURES/8

    Article  Google Scholar 

  36. Iqbal, M., Raza, S.A.: Artificial neural network based emotion classification and recognition from speech. Int. J. Adv. Comput. Sci. Appl. 11(12), 434–444 (2020)

    Google Scholar 

  37. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). https://doi.org/10.1007/bf00994018

    Article  MATH  Google Scholar 

  38. Valstar, M.F., Patras, I., Pantic, M., (2005) “Facial action unit detection using probabilistic actively learned support vector machines on tracked facial point data.” In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. https://doi.org/10.1109/CVPR.2005.457.

  39. Guo, X.: “A KNN classifier for face recognition.” In: 2021 IEEE 3rd International Conference on Communications, Information System and Computer Engineering, CISCE 2021: 292–297 (2021). https://doi.org/10.1109/CISCE52179.2021.9445908.

  40. Zebari, R., Abdulazeez, A., Zeebaree, D., Zebari, D., Saeed, J.: A comprehensive review of dimensionality reduction techniques for feature selection and feature extraction. J. Appl. Sci. Technol. Trends 1(2), 56–70 (2020). https://doi.org/10.38094/jastt1224

    Article  Google Scholar 

  41. Ghazouani, H.: A genetic programming-based feature selection and fusion for facial expression recognition. Appl. Soft Comput. 103, 107173 (2021). https://doi.org/10.1016/j.asoc.2021.107173

    Article  Google Scholar 

  42. Thanaraj, P., Alex, K., Joseph, N.: Emotion classification from speech signal based on empirical mode decomposition and non-linear features speech emotion recognition. Complex Intell. Syst. (2021). https://doi.org/10.1007/s40747-021-00295-z

    Article  Google Scholar 

  43. Aifanti, N., Papachristou, C., Delopoulos, A.: “The MUG Facial Expression Database,” In Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, Desenzano del Garda, Italy. (2010). https://mug.ee.auth.gr/fed/, Accessed 06 Jul 2019.

  44. Yaseen, Z.M., et al.: Novel hybrid data-intelligence model for forecasting monthly rainfall with uncertainty analysis. Water 11(3), 502 (2019). https://doi.org/10.3390/W11030502

    Article  Google Scholar 

  45. Ehteram, M., et al.: Performance improvement for infiltration rate prediction using hybridized adaptive neuro-fuzzy inferences system (ANFIS) with optimization algorithms. Ain Shams Eng. J. 12(2), 1665–1676 (2021). https://doi.org/10.1016/j.asej.2020.08.019

    Article  Google Scholar 

  46. Mahdevari, S., Bagher, M.: A hybrid PSO-ANFIS model for predicting unstable zones in underground roadways. Tunn. Undergr. Space Technol. Inc. Trenchless Technol. Res. (2021). https://doi.org/10.1016/j.tust.2021.104167

    Article  Google Scholar 

  47. Abualigah, L., Yousri, D., Abd Elaziz, M., Ewees, A.A., Al-qaness, M.A.A., Gandomi, A.H.: Aquila optimizer: a novel meta-heuristic optimization algorithm. Comput. Ind. Eng. (2021). https://doi.org/10.1016/J.CIE.2021.107250

    Article  Google Scholar 

  48. Abualigah, L., Elaziz, M.A., Sumari, P., Geem, Z.W., Gandomi, A.H.: Reptile search algorithm (RSA): a nature-inspired meta-heuristic optimizer. Expert Syst. Appl. 191, 116158 (2022). https://doi.org/10.1016/J.ESWA.2021.116158

    Article  Google Scholar 

  49. Oyelade, O.N., Ezugwu, A.E.S., Mohamed, T.I.A., Abualigah, L.: Ebola optimization search algorithm: a new nature-inspired metaheuristic optimization algorithm. IEEE Access 10, 16150–16177 (2022). https://doi.org/10.1109/ACCESS.2022.3147821

    Article  Google Scholar 

  50. Agushaka, J.O., Ezugwu, A.E., Abualigah, L.: Dwarf mongoose optimization algorithm. Comput. Methods Appl. Mech. Eng. 391, 114570 (2022). https://doi.org/10.1016/J.CMA.2022.114570

    Article  MathSciNet  MATH  Google Scholar 

  51. Caraveo, C., Valdez, F., Castillo, O.: A new meta-heuristics of optimization with dynamic adaptation of parameters using type-2 fuzzy logic for trajectory control of a mobile robot. Algorithms (2017). https://doi.org/10.3390/a10030085

    Article  Google Scholar 

  52. Oliveira, M.V., Schirru, R.: Applying particle swarm optimization algorithm for tuning a neuro-fuzzy inference system for sensor monitoring. Prog. Nucl. Energy 51(1), 177–183 (2009). https://doi.org/10.1016/j.pnucene.2008.03.007

    Article  Google Scholar 

  53. Gaxiola, F., et al.: PSO with dynamic adaptation of parameters for optimization in neural networks with interval type-2 fuzzy numbers weights. Axioms 8(1), 14 (2019). https://doi.org/10.3390/axioms8010014

    Article  Google Scholar 

  54. Kardani, N., Bardhan, A., Kim, D., Samui, P., Zhou, A.: Modelling the energy performance of residential buildings using advanced computational frameworks based on RVM, GMDH, ANFIS-BBO and ANFIS-IPSO. J. Build. Eng. 35, 102105 (2021). https://doi.org/10.1016/j.jobe.2020.102105

    Article  Google Scholar 

  55. Ehteram, M., et al.: Hybridization of artificial intelligence models with nature inspired optimization algorithms for lake water level prediction and uncertainty analysis. Alex. Eng. J. 60(2), 2193–2208 (2021). https://doi.org/10.1016/j.aej.2020.12.034

    Article  Google Scholar 

  56. Cho, H.C., Choi, S.H., Han, S.J., Lee, S.H., Kim, H.Y., Kim, K.S.: Effective compressive strengths of corner and edge concrete columns based on an adaptive neuro-fuzzy inference system. Appl. Sci. (Switzerland) (2020). https://doi.org/10.3390/app10103475

    Article  Google Scholar 

  57. Armaghani, D.J., Asteris, P.G.: A comparative study of ANN and ANFIS models for the prediction of cement-based mortar materials compressive strength. Neural Comput. Appl. (2021). https://doi.org/10.1007/s00521-020-05244-4

    Article  Google Scholar 

  58. Kennedy, J., Eberhart, R., (1995) “Particle swarm optimisation.” In: Proc. of the IEEE Int. conference on neural networks 4: 1942–1948 (1995). https://doi.org/10.1007/978-3-030-61111-8_2.

  59. Mostafaei, M.: ANFIS models for prediction of biodiesel fuels cetane number using desirability function. Fuel 216, 665–672 (2018). https://doi.org/10.1016/j.fuel.2017.12.025

    Article  Google Scholar 

  60. Elzain, H.E., et al.: ANFIS-MOA models for the assessment of groundwater contamination vulnerability in a nitrate contaminated area. J. Environ. Manag. (2021). https://doi.org/10.1016/j.jenvman.2021.112162

    Article  Google Scholar 

  61. Deng, X., Liu, Q., Deng, Y., Mahadevan, S.: An improved method to construct basic probability assignment based on the confusion matrix for classification problem. Inf. Sci. 340–341, 250–261 (2016). https://doi.org/10.1016/j.ins.2016.01.033

    Article  Google Scholar 

  62. Chicco, D., Jurman, G.: The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics 21(1), 61613 (2020). https://doi.org/10.1186/s12864-019-6413-7

    Article  Google Scholar 

  63. Powers, D.M.W.: Evaluation: from precision, recall and f-measure to roc, informedness, markedness & correlation. J. Mach. Learn. Technol. 2(1), 37–63 (2011)

    MathSciNet  Google Scholar 

  64. Fawcett, T.: An Introduction to ROC analysis. Pattern Recogn. Lett. 27(8), 861–874 (2006). https://doi.org/10.1016/j.patrec.2005.10.010

    Article  MathSciNet  Google Scholar 

Download references

Funding

No financial support was received for this work.

Author information

Authors and Affiliations

  1. Department of Computer Engineering, Faculty of Engineering, Şırnak University, Şırnak, Turkey

    Mahmut Dirik

Authors
  1. Mahmut Dirik
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mahmut Dirik.

Ethics declarations

Conflict of interest

The authors declared no conflict of interest.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dirik, M. Optimized Anfis Model with Hybrid Metaheuristic Algorithms for Facial Emotion Recognition. Int. J. Fuzzy Syst. 25, 485–496 (2023). https://doi.org/10.1007/s40815-022-01402-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s40815-022-01402-z

Keywords

Search

Navigation