Skip to main content
Log in

BRIEF-based face descriptor: an application to automatic facial expression recognition (AFER)

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

This paper presents a new face descriptor based on binary robust independent elementary features (BRIEF) (Calonder et al. in IEEE Trans Pattern Anal Mach Intell 34(7):1281–1298, 2012; in: European conference on computer vision, Springer, pp 778–792, 2010). The most important properties of BRIEF are the independence of the descriptor length from the matching window size and the possibility of using a subsample of pixel pairs located at arbitrary positions in the matching window. Furthermore, BRIEF is computationally simple and gives a compact representation. The BRIEF descriptor can be used to generate discriminative features globally from an image. However, when BRIEF is used to generate features from a region of an image with no explicit shape such as a face in an image, BRIEF must be used locally to ensure that each pixel in the region is evaluated locally to capture the local properties. In this paper, the performance of BRIEF feature is evaluated in the task of AFER. Using three different facial expression databases, we demonstrate that BRIEF provides satisfactory, encouraging and comparable performance to the performance of alternative face representation tested on the same data, and it is efficient in terms of memory usage and execution time.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. Comput. Vis. 2004, 469–481 (2004)

    MATH  Google Scholar 

  2. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)

    Article  Google Scholar 

  3. Algaraawi, N., Morris, T.: Study on aging effect on facial expression recognition. In: World Congress on Engineering, London, UK (2016)

  4. Alshamsi, H., Meng, H., Li, M.: Real time facial expression recognition app development on mobile phones. In: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 1750–1755. IEEE (2016)

  5. Calonder, M., Lepetit, V., Ozuysal, M., Trzcinski, T., Strecha, C., Fua, P.: Brief: computing a local binary descriptor very fast. IEEE Trans. Pattern Analy. Mach. Intell. 34(7), 1281–1298 (2012)

    Article  Google Scholar 

  6. Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: binary robust independent elementary features. In: European Conference on Computer Vision, pp. 778–792. Springer (2010)

  7. Chang, C.C., Lin, C.J.: Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2(3), 27 (2011)

    Google Scholar 

  8. Cruz, A., Bhanu, B., Yang, S.: A psychologically-inspired match-score fusion model for video-based facial expression recognition. In: Affective Computing and Intelligent Interaction, pp. 341–350. Springer (2011)

  9. Dahmane, M., Meunier, J.: Continuous emotion recognition using Gabor energy filters. In: Affective Computing and Intelligent Interaction, pp. 351–358. Springer (2011)

  10. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 1, pp. 886–893. IEEE (2005)

  11. Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), E1454–E1462 (2014)

    Article  Google Scholar 

  12. Ebner, N.C., Riediger, M., Lindenberger, U.: Faces: a database of facial expressions in young, middle-aged, and older women and men: development and validation. Behav. Res. Methods 42(1), 351–362 (2010)

    Article  Google Scholar 

  13. Galvez-Lopez, D., Tardos, J.D.: Real-time loop detection with bags of binary words. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 51–58. IEEE (2011)

  14. Gálvez-López, D., Tardos, J.D.: Bags of binary words for fast place recognition in image sequences. IEEE Trans. Robot. 28(5), 1188–1197 (2012)

    Article  Google Scholar 

  15. Glodek, M., Tschechne, S., Layher, G., Schels, M., Brosch, T., Scherer, S., Kächele, M., Schmidt, M., Neumann, H., Palm, G., et al.: Multiple classifier systems for the classification of audio-visual emotional states. In: Affective Computing and Intelligent Interaction, pp. 359–368. Springer (2011)

  16. Guo, G., Guo, R., Li, X.: Facial expression recognition influenced by human aging. IEEE Trans. Affect. Comput. 4(3), 291–298 (2013)

    Article  Google Scholar 

  17. Heinly, J., Dunn, E., Frahm, J.M.: Comparative evaluation of binary features. In: Computer Vision—ECCV 2012, pp. 759–773. Springer (2012)

  18. Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space–time video volumes. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 314–321. IEEE (2011)

  19. Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories, pp. 2169–2178. IEEE (2006)

  20. Leutenegger, S., Chli, M., Siegwart, R.Y.: Brisk: Binary robust invariant scalable keypoints. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2548–2555. IEEE (2011)

  21. Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020). https://doi.org/10.1109/TAFFC.2020.2981446

  22. Li, X., Xiaopeng, H., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikainen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. (2017). https://doi.org/10.1109/TAFFC.2017.2667642

  23. Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (cert). In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 298–305. IEEE (2011)

  24. Liu, M., Li, S., Shan, S., Chen, X.: Au-aware deep networks for facial expression recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2013)

  25. Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)

    Article  Google Scholar 

  26. Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)

  27. Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999)

    Article  Google Scholar 

  28. Menchetti, G., Chen, Z., Wilkie, D.J., Ansari, R., Yardimci, Y., Çetin, A.E.: Pain detection from facial videos using two-stage deep learning. In: 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5. IEEE (2019)

  29. Mohammad, S., Morris, D., Thacker, N.: Texture analysis for the segmentation of optic disc in retinal images. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 4265–4270. IEEE (2013)

  30. Mohammad, S., Morris, T.: Binary robust independent elementary feature features for texture segmentation. Adv. Sci. Lett. 23(6), 5178–5182 (2017)

    Article  Google Scholar 

  31. Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)

    Article  Google Scholar 

  32. Ojansivu, V., Heikkilä, J.: Blur insensitive texture classification using local phase quantization. In: International Conference on Image and Signal Processing, pp. 236–243. Springer (2008)

  33. Ouellet, S.: Real-time emotion recognition for gaming using deep convolutional network features. arXiv preprint arXiv:1408.3750 (2014)

  34. Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: an efficient alternative to sift or surf. In: 2011 IEEE international conference on Computer Vision (ICCV), pp. 2564–2571. IEEE (2011)

  35. Sandbach, G., Zafeiriou, S., Pantic, M.: Markov random field structures for facial action unit intensity estimation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 738–745 (2013)

  36. Sariyanidi, E., Gunes, H., Gökmen, M., Cavallaro, A.: Local zernike moment representation for facial affect recognition. In: BMVC (2013)

  37. Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)

    Article  Google Scholar 

  38. Sikka, K., Wu, T., Susskind, J., Bartlett, M.: Exploring bag of words architectures in the facial expression domain. In: Computer Vision—ECCV 2012. Workshops and Demonstrations, pp. 250–259. Springer (2012)

  39. Tong, Y., Chen, J., Ji, Q.: A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE Trans. Pattern Anal. Mach. Intell. 32(2), 258–273 (2010)

    Article  Google Scholar 

  40. Tong, Y., Liao, W., Ji, Q.: Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), 1683–1699 (2007)

    Article  Google Scholar 

  41. Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., Pantic, M.: Avec 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pp. 3–10. ACM (2013)

  42. Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The first facial expression recognition and analysis challenge. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 921–926. IEEE (2011)

  43. Vapnik, V.N., Vapnik, V.: Statistical Learning Theory, vol. 1. Wiley, New York (1998)

    MATH  Google Scholar 

  44. Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)

    Article  Google Scholar 

  45. Vukadinovic, D., Pantic, M.: Fully automatic facial feature point detection using Gabor feature based boosted classifiers. In: 2005 IEEE International Conference on Systems, Man and Cybernetics, vol. 2, pp. 1692–1698. IEEE (2005)

  46. Yang, S., Bhanu, B.: Facial expression recognition using emotion avatar image. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 866–871. IEEE (2011)

  47. Zhang, T.: Facial expression recognition based on deep learning: a survey. In: International Conference on Intelligent and Interactive Systems and Applications, pp. 345–352. Springer (2017)

  48. Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Nora Al-Garaawi.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 4540 KB)

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Al-Garaawi, N., Wu, Q. & Morris, T. BRIEF-based face descriptor: an application to automatic facial expression recognition (AFER). SIViP 15, 371–379 (2021). https://doi.org/10.1007/s11760-020-01759-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-020-01759-4

Keywords

Navigation