Skip to main content
SpringerLink
Log in

Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Detection of emotion using facial expression is a growing field of research. Facial expression detection is also helpful to identify the behavior of a person when a man interacts with the computer. In this work, facial expression recognition with respect to the changes in the facial geometry is proposed. First, the image is enhanced by means of discrete wavelet transform and fuzzy combination. Then, the facial geometry is found using the modified eyemap and mouthmap algorithm after finding the landmarks. Finally, the area and angle of the constructed triangles are found and classified using neural network with the help of tensorflow central processing unit version. Results show that the proposed algorithm is efficient in finding the facial emotion.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14

Similar content being viewed by others

References

  1. Agarwal, S., Santra, B., Mukherjee, D.P.: Anubhav: recognizing emotions through facial expression. Vis. Comput. 34(2), 177–191 (2018)

    Article  Google Scholar 

  2. Alugupally, N., Samal, A., Marx, D., Bhatia, S.: Analysis of landmarks in recognition of face expressions. Pattern Recognit. Image Anal. 21(4), 681–693 (2011)

    Article  Google Scholar 

  3. Buciu, I., Kotropoulos, C., Pitas, I.: Comparison of ica approaches for facial expression recognition. Signal Image Video Process. 3(4), 345–361 (2009)

    Article  Google Scholar 

  4. Chan, T.F., Vese, L.A.: Active contours without edges. IEEE Trans. Image Process. 10(2), 266–277 (2001)

    Article  Google Scholar 

  5. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Washington (1978)

    Google Scholar 

  6. Gogić, I., Manhart, M., Pandžić, I.S., Ahlberg, J.: Fast facial expression recognition using local binary features and shallow neural networks. Vis. Comput. 1–16 (2018)

  7. Happy, S., Routray, A.: Automatic facial expression recognition using features of salient facial patches. IEEE Trans. Affect. Comput. 6(1), 1–12 (2015)

    Article  Google Scholar 

  8. Hsu, R.L., Abdel-Mottaleb, M., Jain, A.K.: Face detection in color images. IEEE Trans. Pattern Anal. Mach. Intell. 24(5), 696–706 (2002)

    Article  Google Scholar 

  9. Huang, C.L., Huang, Y.M.: Facial expression recognition using model-based feature extraction and action parameters classification. J. Vis. Commun. Image Represent. 8(3), 278–290 (1997)

    Article  Google Scholar 

  10. Ilbeygi, M., Shah-Hosseini, H.: A novel fuzzy facial expression recognition system based on facial feature extraction from color face images. Eng. Appl. Artif. Intell. 25(1), 130–146 (2012)

    Article  Google Scholar 

  11. Jackway, P.T., Deriche, M.: Scale-space properties of the multiscale morphological dilation–erosion. IEEE Trans. Pattern Anal. Mach. Intell. 18(1), 38–51 (1996)

    Article  Google Scholar 

  12. Jain, V., Mavridou, E., Crowley, J.L., Lux, A.: Facial expression analysis and the affect space. Pattern Recognit. Image Anal. 25(3), 430–436 (2015)

    Article  Google Scholar 

  13. Kanade, T., Cohn, J.F., Tian, Y.: Comprehensive database for facial expression analysis. In: Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings, pp. 46–53. IEEE (2000)

  14. Karthigayan, M., Juhari, M.R.M., Nagarajan, R., Sugisaka, M., Yaacob, S., Mamat, M.R., Desa, H.: Development of a personified face emotion recognition technique using fitness function. Artif. Life Robot. 11(2), 197–203 (2007)

    Article  Google Scholar 

  15. Kim, D.: Facial expression recognition using ASM-based post-processing technique. Pattern Recognit. Image Anal. 26(3), 576–581 (2016)

    Article  Google Scholar 

  16. Lajevardi, S.M., Hussain, Z.M.: Automatic facial expression recognition: feature extraction and selection. Signal Image Video Process. 6(1), 159–169 (2012)

    Article  Google Scholar 

  17. Lekdioui, K., Messoussi, R., Ruichek, Y., Chaabi, Y., Touahni, R.: Facial decomposition for expression recognition using texture/shape descriptors and SVM classifier. Signal Process. Image Commun. 58, 300–312 (2017)

    Article  Google Scholar 

  18. Liu, N., Zhang, B., Zong, Y., Liu, L., Chen, J., Zhao, G., Zhu, L.: Super wide regression network for unsupervised cross-database facial expression recognition. In: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1897–1901. IEEE (2018)

  19. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended Cohn–Kanade dataset (CK+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)

  20. Lundqvist, D., Flykt, A., Öhman, A.: The Karolinska Directed Emotional Faces-KDEF. CD ROM from Department of Clinical Neuroscience, Psychology section, Karolinska Institutet, ISBN 91-630-7164-9 (1998)

  21. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J.: Coding facial expressions with Gabor wavelets. In: Third IEEE International Conference on Automatic Face and Gesture Recognition, 1998. Proceedings, pp. 200–205. IEEE (1998)

  22. Mayer, C., Eggers, M., Radig, B.: Cross-database evaluation for facial expression recognition. Pattern Recognit. Image Anal. 24(1), 124–132 (2014)

    Article  Google Scholar 

  23. Mlakar, U., Potočnik, B.: Automated facial expression recognition based on histograms of oriented gradient feature vector differences. Signal Image Video Process. 9(1), 245–253 (2015)

    Article  Google Scholar 

  24. Panda, S.P.: Image contrast enhancement in spatial domain using fuzzy logic based interpolation method. In: 2016 IEEE Students’ Conference on Electrical, Electronics and Computer Science (SCEECS), pp. 1–4. IEEE (2016)

  25. Ruiz-Garcia, A., Palade, V., Elshaw, M., Almakky, I.: Deep learning for illumination invariant facial expression recognition. In: 2018 International Joint Conference on Neural Networks (IJCNN), pp. 1–6. IEEE (2018)

  26. Silva, C., Schnitman, L., Oliveira, L.: Detection of facial landmarks using local-based information. In: The 19th Edition of the Brazilian Conference on Automation-CBA 2012, Campina Grande, PB, Brazil (oral presentation), September 3 (2012)

  27. Sun, Z., Hu, Z., Wang, M., Zhao, S.: Individual-free representation-based classification for facial expression recognition. Signal Image Video Process. 11(4), 597–604 (2017)

    Article  Google Scholar 

  28. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001. CVPR 2001, vol. 1, pp. I–I. IEEE (2001)

  29. Wong, J.J., Cho, S.Y.: A face emotion tree structure representation with probabilistic recursive neural network modeling. Neural Comput. Appl. 19(1), 33–54 (2010)

    Article  Google Scholar 

  30. Yaddaden, Y., Adda, M., Bouzouane, A., Gaboury, S., Bouchard, B.: User action and facial expression recognition for error detection systemin an ambient assisted environment. Expert. Syst. Appl. 112, 173–189 (2018)

    Article  Google Scholar 

  31. Yang, H., Ciftci, U., Yin, L.: Facial expression recognition by de-expression residue learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2177 (2018)

  32. Yu, Z., Liu, Q., Liu, G.: Deeper cascaded peak-piloted network for weak expression recognition. Vis. Comput. 34(12), 1691–1699 (2018)

    Article  Google Scholar 

  33. Zhao, G., Huang, X., Taini, M., Li, S.Z., PietikäInen, M.: Facial expression recognition from near-infrared videos. Image Vis. Comput. 29(9), 607–619 (2011)

    Article  Google Scholar 

Download references

Funding

This work was funded by the Department of Science and Technology—Promotion of University Research and Scientific Excellence (DST-PURSE) Phase II Program, India.

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, College of Engineering, Guindy, Anna University, Chennai, India

    Allen Joseph & P. Geetha

Authors
  1. Allen Joseph
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. P. Geetha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Allen Joseph.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Joseph, A., Geetha, P. Facial emotion detection using modified eyemap–mouthmap algorithm on an enhanced image and classification with tensorflow. Vis Comput 36, 529–539 (2020). https://doi.org/10.1007/s00371-019-01628-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-019-01628-3

Keywords

Search

Navigation