Skip to main content
SpringerLink
Log in

Smile intensity recognition in real time videos: fuzzy system approach

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Facial emotion is a significant way of understanding or interpreting one’s inner thoughts. Real time video at any instant exhibits the emotion which serves as the input to the emotion recognition system. Many literatures propose different strategies in identifying the emotions by working on different features in the facial components, including geometrical, appearance and motion features. This paper considers the geometrical features as a prime component in deciding the intensity of the smile expressed in the real time videos of the AM-FED (Affectiva-MIT Facial Expression Dataset). Geometrical features considered in the work are the normalized Euclidean distance between the contributing LandMarkPoints (LMPs) of the eyes and the lip portions of the face. Fuzzy logic is applied to the system to effectively classify the intensity of the emotion, i.e., happiness or smile as Maximum, Moderate, Less and neutral. Being a Land mark based assessment, evaluating the normalized values of the Euclidean distance between LMPs for each frame of the video and then mapping the values of all the frames in a range helps the fuzzy decision making stage to relate the mapped values to the smile intensity of each frame. The average recognition rate obtained is 86.54%. The system contributes a less complex but nearly accurate smile intensity recognition model when compared to other computation intensive decision making models, with a practical significance in the customer or client’s mood/satisfactory identification in the online marketing and/or communication of the intensity of the smile of the conversing person to a visually challenged person.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Baltrusaitis T et al (2011) Real-time inference of mental states from facial expressions and upper body gestures. In: Proc. of the IEEE Int'l Conference on Automatic Face and Gesture Recognition and Workshops, p 909–914. https://doi.org/10.1109/FG.2011.5771372

  2. Chakraborty A et al Emotion recognition from facial expressions and its control using fuzzy logic. IEEE Trans Syst Man Cybern A Syst Hum 39(4):726–743. https://doi.org/10.1109/TSMCA.2009.2014645

  3. Chen S, Tian YL, Liu Q, Metaxas DN (2013) Recognizing expressions from face and body gesture by temporal normalized motion and appearance features. Image Vis Comput 31:175–185. https://doi.org/10.1109/CVPRW.2011.5981880

    Article  Google Scholar 

  4. Chen J, Ou Q, Chi Z, Fu H (2017) Smile detection in the wild with deep convolutional neural networks. Mach Vis Appl 28(1–2):173–183. https://doi.org/10.1007/s00138-016-0817

    Article  Google Scholar 

  5. Dornaika F, Moujahid A, Raducanu B (2013) Facial expression recognition using tracked facial actions: classifier performance analysis. Eng Appl Artif Intell 26:467–477. https://doi.org/10.1016/j.engappai.2012.09.002

    Article  Google Scholar 

  6. Ekman P, Friesen W (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press

  7. Esau N et al (2007) Real-time facial expression recognition using a fuzzy emotion model. Fuzzy Systems Conference. FUZZ-IEEE 2007. IEEE International, p 1–6. https://doi.org/10.1109/FUZZY.2007.4295451

  8. Freire-Obregón D, Castrillón-Santana M (2015) An evolutive approach for smile recognition in video sequences. Int J Patt Recogn Artif Intell. https://doi.org/10.1142/S0218001415500068

  9. Guo G, Guo R, Li X (2013) Facial expression recognition influenced by human aging. IEEE Trans Affect Comput 4:291–298. https://doi.org/10.1109/T-AFFC.2013.13.(9)

    Article  Google Scholar 

  10. Hablani R, Chaudari N, Tanwani S (2013) Recognition of facial expressions using local binary patterns of important facial parts. International Journal of Image Processing 7:163–170

  11. Happy SL, Aurobinda R (2015) Automatic facial expression recognition using features of salient face patches. IEEE Trans Affect Comput 6(1):1–11. https://doi.org/10.1109/TAFFC.2014.2386334

    Article  Google Scholar 

  12. Kauser N, Sharma J Facial expression recognition using LBP template of facial parts and multilayer neural network. In: proc. of International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud) (I-SMAC), p 445–449. https://doi.org/10.1109/I-SMAC.2017.8058389

  13. Kudiri KM, Said AMd, Nayan MY Human emotion detection through speech and facial expressions. International Conference on Computer and Information Sciences (ICCOINS), p 351–356. https://doi.org/10.1109/ICCOINS.2016.7783240

  14. Li P, Phung SL, Bouzerdom A, Tivive FHC (2010) Automatic recognition of smiling and neutral facial expressions. Digital Image Computing: Techniques and Applications, p 581–586. https://doi.org/10.1109/DICTA.2010.103

  15. Liu Z et al (2017) A facial expression emotion recognition based human-robot interaction system. IEEE/CAA J. Autom. Sinica 4(4):668–676. https://doi.org/10.1109/JAS.2017.7510622

    Article  Google Scholar 

  16. Liu Y et al Facial expression recognition with PCA and LBP features extracting from active facial patches. IEEE International Conference on Real-time Computing and Robotics (RCAR), p 368–373. https://doi.org/10.1109/RCAR.2016.7784056

  17. Mamdani EH, Assilian S (1975) An experiment in linguistic synthesis with a fuzzy logic controller. Int J Man Mach Stud 7(1):1–13. https://doi.org/10.1016/S0020-7373(75)80002-2

    Article  MATH  Google Scholar 

  18. Matlab 2016 Help Documentation-mapminmax algorithm in NeuralNetwork Toolbox

  19. McDuff D, el Kaliouby R, Demirdjian D, Picard R (2013) Predicting online media effectiveness based on smile responses gathered over the internet. 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. https://doi.org/10.1109/FG.2013.6553750

  20. McDuff D, el Kaliouby R, Senechal T, Amr M, Cohn J, Picard R (2013) Affectiva-MIT Facial Expression Dataset (AMFED): naturalistic and spontaneous facial expressions collected “In-the-Wild”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops

  21. McDuff D et al Automatic measurement of ad preferences from facial responses gathered over the internet. Image Vis Comput:630–640. https://doi.org/10.1016/j.imavis.2014.01.004

  22. Nicolle J, K Bailly, Chetouani M (2015) Facial action unit intensity prediction via hard multi-task metric learning for Kernel regression. 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. https://doi.org/10.1109/FG.2015.7284868

  23. Patil JV, Bailke P Real time facial expression recognition using RealSense camera and ANN. In: Proc of International conference on Inventive Computation Technologies (ICICT), p 1–6. https://doi.org/10.1109/INVENTIVE.2016.7824820

  24. Peihe W, Dekai Z Convexity of level sets of minimal graph on space form with nonnegative curvature. J Differ Equ 262:5534–5564. https://doi.org/10.1016/j.jde.2017.02.010

  25. Peihe W, Lingling Z Some geometrical properties of convex level sets of minimal graph on 2-dimensional Riemannian mani-folds. Nonlinear Anal Theory Methods Appl 130(1):1–13. https://doi.org/10.1016/j.na.2015.09.021

  26. Qiao Y, Zeng K, Xu L (2016) A smartphone-based driver fatigue detection using fusion of multiple real-time facial features. In: proc of Consumer Communications & Networking Conference (CCNC), 2016 13th IEEE Annual, p 230–235. https://doi.org/10.1109/CCNC.2016.7444761

  27. Riu B et al Local directional ternary pattern for facial expression recognition. IEEE Trans Image Process 26(12):6006–6018. https://doi.org/10.1109/TIP.2017.2726010

  28. Shimada K et al Appearance-based smile intensity estimation by cascaded support vector machines. Pattern Recogn Lett:13–21. https://doi.org/10.1016/j.patrec.2014.10.004

  29. Whitehill J, Littlewort G, Fasel I, Bartlett M (2009) Toward practical smile detection. IEEE Trans Pattern Anal Mach Intell 31(11):2106–2111

    Article  Google Scholar 

  30. Yang Z et al (2018) “Intermediate data caching optimization for multistage and parallel big data frameworks,” ARXiv

  31. Zhang L, Tjondronegoro D (2011) Facial expression recognition using facial movement features. IEEE Trans Affect Comput 2:219–229. https://doi.org/10.1109/T-AFFC.2011.13

    Article  Google Scholar 

  32. Zhang K et al. (2016) Gender and smile classification using deep convolutional neural networks. IEEE Conference on Computer Vision and Pattern Recognition Workshops, p 34–38. https://doi.org/10.1109/CVPRW.2016.97

  33. Zimmermann HJ Fuzzy set theory-and its applications, Fourth Edition. Springer Science+Business Media, LLC. https://doi.org/10.1007/978-94-010-0646-0

  34. Zisheng, Jun-Ichi, Masahide (2009) Facial-component-based bag of words and PHOG descriptor for facial expression recognition. In: Proc. of IEEE Int'l conference on Systems, Man and Cybernetics, p 1353–1358. https://doi.org/10.1109/ICSMC.2009.5346254

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, Francis Xavier Engineering College, Tirunelveli, Tamilnadu, India

    Vinola C.

  2. Department of Computer Science and Engineering, Velammal Engineering College, Chennai, Tamilnadu, India

    Vimala Devi K.

Authors
  1. Vinola C.
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vimala Devi K.
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Vinola C..

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vinola C., Vimala Devi K. Smile intensity recognition in real time videos: fuzzy system approach. Multimed Tools Appl 78, 15033–15052 (2019). https://doi.org/10.1007/s11042-018-6890-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-018-6890-8

Keywords

Search

Navigation