Skip to main content

Advertisement

Springer Nature Link
Log in

An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

To address the variabilities of the number and position of extracted feature points for the traditional scale-invariant feature transform (SIFT) method, an improved SIFT algorithm is proposed for robust emotion recognition. Specifically, shape decomposition is first performed on the detected facial images by defining a weight vector. Then, a feature point constraint algorithm is developed to determine the optimum position of the feature points that can effectively represent the expression change regions. On this basis, the SIFT descriptors are applied to extract the regional gradient information as feature parameters. Finally, the support vector machine classifier combined with the principal component analysis method is used to reduce the feature dimensions and facial expression recognition. Experiments have been performed under different conditions, i.e., varied illuminations, face poses and facial moisture levels, using 15 participants. In the cases of frontal face and 5-degree face rotation views, the average recognition accuracies are 98.52% and 94.47% (no additional light sources), as well as 96.97% and 95.40% (two additional light sources), respectively. In addition, as an effective supplement to the problem of changes in illumination, the average recognition ratios are 96.23% and 96.20% under dry and wet face conditions, respectively. The experimental results reveal the robust performance of the proposed method in facial expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

References

  1. Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141

    Article  Google Scholar 

  2. Mohammadi Z, Frounchi J, Amiri M (2017) Wavelet-based emotion recognition system using EEG signal. Neural Comput Appl 28(8):1985–1990

    Article  Google Scholar 

  3. Aspinall P, Mavros P, Coyne R, Roe J (2015) The urban brain: analysing outdoor physical activity with mobile EEG. Br J Sports Med 49(4):272–276

    Article  Google Scholar 

  4. Tantawi MM, Revett K, Salem A-B, Tolba MF (2015) A wavelet feature extraction method for electrocardiogram (ECG)-based biometric recognition. Signal Image Video Process 9(6):1271–1280

    Article  Google Scholar 

  5. Chen Y, Yang Z, Gong H, Wang SJ (2018) Recognition of sketching from surface electromyography. Neural Comput Appl 30(9):2725–2737

    Article  Google Scholar 

  6. Gruebler A, Suzuki K (2010) Measurement of distal EMG signals using a wearable device for reading facial expressions. In: 2010 annual international conference of the IEEE engineering in medicine and biology. IEEE, pp 4594–4597

  7. Di Rienzo M, Rizzo F, Parati G, Brambilla G, Ferratini M, Castiglioni P (2005) MagIC system: a new textile-based wearable device for biological signal monitoring. Applicability in daily life and clinical setting. In: Conference proceedings IEEE engineering in medicine and biology society, pp 7167–7169

  8. Einighammer H, Gilenko M (2009) Method and device for recognition of natural skin during contact-free biometric identification of a person. Google Patents

  9. Giannakakis G, Pediaditis M, Manousos D, Kazantzaki E, Chiarugi F, Simos PG, Marias K, Tsiknakis M (2017) Stress and anxiety detection using facial cues from videos. Signal Process Control 31:89–101

    Article  Google Scholar 

  10. Cohen I, Sebe N, Garg A, Chen LS, Huang TS (2003) Facial expression recognition from video sequences: temporal and static modeling. Comput Vis Image Underst 91(1–2):160–187

    Article  Google Scholar 

  11. Wilson PI, Fernandez J (2006) Facial feature detection using Haar classifiers. J Comput Sci Coll 21(4):127–133

    Google Scholar 

  12. Zhang H, Wu QJ, Chow TW, Zhao M (2012) A two-dimensional neighborhood preserving projection for appearance-based face recognition. Pattern Recognit 45(5):1866–1876

    Article  Google Scholar 

  13. Saeed S, Mahmood MK, Khan YD (2018) An exposition of facial expression recognition techniques. Neural Comput Appl 29(9):425–443

    Article  Google Scholar 

  14. Gross R, Brajovic V (2003) An image preprocessing algorithm for illumination invariant face recognition. In: International conference on audio-and video-based biometric person authentication. Springer, Berlin, pp 10–18

  15. Dizdaroğlu B, Ataer-Cansizoglu E, Kalpathy-Cramer J, Keck K, Chiang MF, Erdogmus D (2014) Structure-based level set method for automatic retinal vasculature segmentation. EURASIP J Image Video Process 2014(1):39

    Article  Google Scholar 

  16. Xu F, Wang Z (2017) A facial expression recognition method based on cubic spline interpolation and HOG features. In: 2017 IEEE international conference on robotics and biomimetics (ROBIO). IEEE, pp 1300–1305

  17. Wang L, Li R, Wang K (2013) Automatic facial expression recognition using SVM based on AAMs. In: 2013 5th international conference on intelligent human-machine systems and cybernetics. IEEE, pp 330–333

  18. Wang X, Liu X, Lu L, Shen Z (2014) A new facial expression recognition method based on geometric alignment and l bp features. In: 2014 IEEE 17th international conference on computational science and engineering. IEEE, pp 1734–1737

  19. Luo Y, Zhang L, Chen Y, Jiang W (2017) Facial expression recognition algorithm based on reverse co-salient regions (RCSR) features. In: 2017 4th international conference on information science and control engineering (ICISCE). IEEE, pp 326–329

  20. Cornejo JYR, Pedrini H (2018) Emotion recognition from occluded facial expressions using weber local descriptor. In: 2018 25th international conference on systems, signals and image processing (IWSSIP). IEEE, pp 1–5

  21. Zhang H, Ji Y, Huang W, Liu L (2018) Sitcom-star-based clothing retrieval for video advertising: a deep learning framework. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3579-x

    Article  Google Scholar 

  22. Zhang H, Cao X, Ho JK, Chow TW (2017) Object-level video advertising: an optimization framework. IEEE Trans Ind Inform 13(2):520–531

    Article  Google Scholar 

  23. Chaki J, Dey N, Shi F, Sherratt RS (2019) Pattern mining approaches used in sensor-based biometric recognition: a review. IEEE Sens J 19(10):3569–3580

    Article  Google Scholar 

  24. Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628

    Article  Google Scholar 

  25. Zhang K, Huang Y, Du Y, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203

    Article  MathSciNet  Google Scholar 

  26. Meng Z, Liu P, Cai J, Han S, Tong Y (2017) Identity-aware convolutional neural network for facial expression recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, pp 558–565

  27. Uçar A, Demir Y, Güzeliş C (2016) A new facial expression recognition based on curvelet transform and online sequential extreme learning machine initialized with spherical clustering. Neural Comput Appl 27(1):131–142

    Article  Google Scholar 

  28. Lowe DG (1999) Object recognition from local scale-invariant features. In: iccv, vol 2, pp 1150–1157

  29. Viola P, Jones MJ (2004) Robust real-time face detection. Int J Comput Vis 57(2):137–154

    Article  Google Scholar 

  30. Gower JC (1975) Generalized procrustes analysis. Psychometrika 40(1):33–51

    Article  MathSciNet  Google Scholar 

  31. Saragih JM, Lucey S, Cohn JF (2009) Face alignment through subspace constrained mean-shifts. In: 2009 IEEE 12th international conference on computer vision. IEEE, pp 1034–1041

  32. Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110

    Article  Google Scholar 

  33. Brown M, Lowe DG (2007) Automatic panoramic image stitching using invariant features. Int J Comput Vis 74(1):59–73

    Article  Google Scholar 

  34. Wold S, Esbensen K, Geladi P (1987) Principal component analysis. Chemom Intell Lab Syst 2(1–3):37–52

    Article  Google Scholar 

  35. Rosipal R, Girolami M, Trejo LJ, Cichocki A (2001) Kernel PCA for feature extraction and de-noising in nonlinear regression. Neural Comput Appl 10(3):231–243

    Article  Google Scholar 

  36. Yildirim A, Halici U (2013) Analysis of dimension reduction by PCA and AdaBoost on spelling paradigm EEG data. In: 2013 6th international conference on biomedical engineering and informatics. IEEE, pp 192–196

  37. Zhang C, Shao Y, Tan J, Deng N (2013) Mixed-norm linear support vector machine. Neural Comput Appl 23(7–8):2159–2166

    Article  Google Scholar 

  38. Zhang J, Wang S (2016) A fast leave-one-out cross-validation for SVM-like family. Neural Comput Appl 27(6):1717–1730

    Article  Google Scholar 

  39. Kohavi R (1995) A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai, vol 2. Montreal, Canada, pp 1137–1145

  40. Lyons MJ, Akamatsu S, Kamachi M, Gyoba J, Budynek J (1998) The Japanese female facial expression (JAFFE) database. In: Proceedings of third international conference on automatic face and gesture recognition, pp 14–16

  41. Kanade T, Cohn JF, Tian Y (2000) Comprehensive database for facial expression analysis. In: Proceedings fourth IEEE international conference on automatic face and gesture recognition (Cat. No. PR00580). IEEE, pp 46–53

  42. Lu Y, Zhang C, Zhou B-Y, Gao X-P, Lv Z (2018) A dual model approach to EOG-based human activity recognition. Biomed Signal Process Control 45:50–57

    Article  Google Scholar 

  43. Bradley AP (1997) The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern Recognit 30(7):1145–1159

    Article  Google Scholar 

  44. Hand DJ, Till RJ (2001) A simple generalisation of the area under the ROC curve for multiple class classification problems. Mach Learn 45(2):171–186

    Article  Google Scholar 

  45. Friesen E, Ekman P (1978) Facial action coding system: a technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto, CA

    Google Scholar 

  46. Jiang B, Valstar MF, Pantic M (2011) Action unit detection using sparse appearance descriptors in space-time video volumes. In: Face and gesture 2011. IEEE, pp 314–321

  47. Dharavath K, Talukdar F, Laskar R, Dey N (2017) Face recognition under dry and wet face conditions. In: Dey N, Santhi V (eds) Intelligent techniques in signal processing for multimedia security. Springer, Berlin, pp 253–271

    Chapter  Google Scholar 

  48. Munasinghe M (2018) Facial expression recognition using facial landmarks and random forest classifier. In: 2018 IEEE/ACIS 17th international conference on computer and information science (ICIS). IEEE, pp 423–427

  49. Alsubari A, Satange D, Ramteke R (2017) Facial expression recognition using wavelet transform and local binary pattern. In: 2017 2nd international conference for convergence in technology (I2CT). IEEE, pp 338–342

Download references

Acknowledgements

The authors would like to thank the volunteers who participated in this study and anonymous reviewers for comments on a draft of this article. The research work is supported by the Open fund for Discipline Construction, Institute of Physical Science and Information Technology, Anhui University; the Anhui Provincial Natural Science Research Project of Colleges and Universities Fund under Grant KJ2018A0008, KJ2017A012; the Natural Science Foundation of Anhui Province under Grant 1908085MF203; and the Open fund for Shaanxi Key Laboratory of Network and System Security under Grant NSSOF1900101.

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Anhui University, Hefei, 230601, China

    Yong Shi, Zhao Lv & Chao Zhang

  2. Institute of Physical Science and Information Technology, Anhui University, Hefei, 230601, China

    Zhao Lv

  3. School of Computer Science and Technology, Georgia Institute of Technology, 801 Atlantic Dr NW, Atlanta, GA, 30332, USA

    Ning Bi

  4. Shanxi Key Laboratory of Network and System Security, Xidian University, Xi’an, 710071, China

    Chao Zhang

Authors
  1. Yong Shi
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Zhao Lv
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Ning Bi
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Chao Zhang
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Zhao Lv.

Ethics declarations

Conflict of interest

Author Yong Shi declares that he has no conflict of interest. Author Chao Zhang declares that he has no conflict of interest. Author Ning Bi declares that he has no conflict of interest. Author Zhao Lv declares that he has no conflict of interest.

Ethical approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Informed consent

Informed consent was obtained from all individual participants included in the study.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, Y., Lv, Z., Bi, N. et al. An improved SIFT algorithm for robust emotion recognition under various face poses and illuminations. Neural Comput & Applic 32, 9267–9281 (2020). https://doi.org/10.1007/s00521-019-04437-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04437-w

Keywords