Abstract
This paper presents a new face descriptor based on binary robust independent elementary features (BRIEF) (Calonder et al. in IEEE Trans Pattern Anal Mach Intell 34(7):1281–1298, 2012; in: European conference on computer vision, Springer, pp 778–792, 2010). The most important properties of BRIEF are the independence of the descriptor length from the matching window size and the possibility of using a subsample of pixel pairs located at arbitrary positions in the matching window. Furthermore, BRIEF is computationally simple and gives a compact representation. The BRIEF descriptor can be used to generate discriminative features globally from an image. However, when BRIEF is used to generate features from a region of an image with no explicit shape such as a face in an image, BRIEF must be used locally to ensure that each pixel in the region is evaluated locally to capture the local properties. In this paper, the performance of BRIEF feature is evaluated in the task of AFER. Using three different facial expression databases, we demonstrate that BRIEF provides satisfactory, encouraging and comparable performance to the performance of alternative face representation tested on the same data, and it is efficient in terms of memory usage and execution time.
Similar content being viewed by others
References
Ahonen, T., Hadid, A., Pietikäinen, M.: Face recognition with local binary patterns. Comput. Vis. 2004, 469–481 (2004)
Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)
Algaraawi, N., Morris, T.: Study on aging effect on facial expression recognition. In: World Congress on Engineering, London, UK (2016)
Alshamsi, H., Meng, H., Li, M.: Real time facial expression recognition app development on mobile phones. In: 2016 12th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD), pp. 1750–1755. IEEE (2016)
Calonder, M., Lepetit, V., Ozuysal, M., Trzcinski, T., Strecha, C., Fua, P.: Brief: computing a local binary descriptor very fast. IEEE Trans. Pattern Analy. Mach. Intell. 34(7), 1281–1298 (2012)
Calonder, M., Lepetit, V., Strecha, C., Fua, P.: Brief: binary robust independent elementary features. In: European Conference on Computer Vision, pp. 778–792. Springer (2010)
Chang, C.C., Lin, C.J.: Libsvm: a library for support vector machines. ACM Trans. Intell. Syst. Technol. (TIST) 2(3), 27 (2011)
Cruz, A., Bhanu, B., Yang, S.: A psychologically-inspired match-score fusion model for video-based facial expression recognition. In: Affective Computing and Intelligent Interaction, pp. 341–350. Springer (2011)
Dahmane, M., Meunier, J.: Continuous emotion recognition using Gabor energy filters. In: Affective Computing and Intelligent Interaction, pp. 351–358. Springer (2011)
Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2005. CVPR 2005, vol. 1, pp. 886–893. IEEE (2005)
Du, S., Tao, Y., Martinez, A.M.: Compound facial expressions of emotion. Proc. Natl. Acad. Sci. 111(15), E1454–E1462 (2014)
Ebner, N.C., Riediger, M., Lindenberger, U.: Faces: a database of facial expressions in young, middle-aged, and older women and men: development and validation. Behav. Res. Methods 42(1), 351–362 (2010)
Galvez-Lopez, D., Tardos, J.D.: Real-time loop detection with bags of binary words. In: 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 51–58. IEEE (2011)
Gálvez-López, D., Tardos, J.D.: Bags of binary words for fast place recognition in image sequences. IEEE Trans. Robot. 28(5), 1188–1197 (2012)
Glodek, M., Tschechne, S., Layher, G., Schels, M., Brosch, T., Scherer, S., Kächele, M., Schmidt, M., Neumann, H., Palm, G., et al.: Multiple classifier systems for the classification of audio-visual emotional states. In: Affective Computing and Intelligent Interaction, pp. 359–368. Springer (2011)
Guo, G., Guo, R., Li, X.: Facial expression recognition influenced by human aging. IEEE Trans. Affect. Comput. 4(3), 291–298 (2013)
Heinly, J., Dunn, E., Frahm, J.M.: Comparative evaluation of binary features. In: Computer Vision—ECCV 2012, pp. 759–773. Springer (2012)
Jiang, B., Valstar, M.F., Pantic, M.: Action unit detection using sparse appearance descriptors in space–time video volumes. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 314–321. IEEE (2011)
Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: spatial pyramid matching for recognizing natural scene categories, pp. 2169–2178. IEEE (2006)
Leutenegger, S., Chli, M., Siegwart, R.Y.: Brisk: Binary robust invariant scalable keypoints. In: 2011 IEEE International Conference on Computer Vision (ICCV), pp. 2548–2555. IEEE (2011)
Li, S., Deng, W.: Deep facial expression recognition: a survey. IEEE Trans. Affect. Comput. (2020). https://doi.org/10.1109/TAFFC.2020.2981446
Li, X., Xiaopeng, H., Moilanen, A., Huang, X., Pfister, T., Zhao, G., Pietikainen, M.: Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods. IEEE Trans. Affect. Comput. (2017). https://doi.org/10.1109/TAFFC.2017.2667642
Littlewort, G., Whitehill, J., Wu, T., Fasel, I., Frank, M., Movellan, J., Bartlett, M.: The computer expression recognition toolbox (cert). In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 298–305. IEEE (2011)
Liu, M., Li, S., Shan, S., Chen, X.: Au-aware deep networks for facial expression recognition. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–6. IEEE (2013)
Liu, M., Li, S., Shan, S., Chen, X.: Au-inspired deep networks for facial expression feature learning. Neurocomputing 159, 126–136 (2015)
Lucey, P., Cohn, J.F., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I.: The extended cohn-kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 94–101. IEEE (2010)
Lyons, M.J., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21(12), 1357–1362 (1999)
Menchetti, G., Chen, Z., Wilkie, D.J., Ansari, R., Yardimci, Y., Çetin, A.E.: Pain detection from facial videos using two-stage deep learning. In: 2019 IEEE Global Conference on Signal and Information Processing (GlobalSIP), pp. 1–5. IEEE (2019)
Mohammad, S., Morris, D., Thacker, N.: Texture analysis for the segmentation of optic disc in retinal images. In: 2013 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 4265–4270. IEEE (2013)
Mohammad, S., Morris, T.: Binary robust independent elementary feature features for texture segmentation. Adv. Sci. Lett. 23(6), 5178–5182 (2017)
Ojala, T., Pietikainen, M., Maenpaa, T.: Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Trans. Pattern Anal. Mach. Intell. 24(7), 971–987 (2002)
Ojansivu, V., Heikkilä, J.: Blur insensitive texture classification using local phase quantization. In: International Conference on Image and Signal Processing, pp. 236–243. Springer (2008)
Ouellet, S.: Real-time emotion recognition for gaming using deep convolutional network features. arXiv preprint arXiv:1408.3750 (2014)
Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: an efficient alternative to sift or surf. In: 2011 IEEE international conference on Computer Vision (ICCV), pp. 2564–2571. IEEE (2011)
Sandbach, G., Zafeiriou, S., Pantic, M.: Markov random field structures for facial action unit intensity estimation. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 738–745 (2013)
Sariyanidi, E., Gunes, H., Gökmen, M., Cavallaro, A.: Local zernike moment representation for facial affect recognition. In: BMVC (2013)
Shan, C., Gong, S., McOwan, P.W.: Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis. Comput. 27(6), 803–816 (2009)
Sikka, K., Wu, T., Susskind, J., Bartlett, M.: Exploring bag of words architectures in the facial expression domain. In: Computer Vision—ECCV 2012. Workshops and Demonstrations, pp. 250–259. Springer (2012)
Tong, Y., Chen, J., Ji, Q.: A unified probabilistic framework for spontaneous facial action modeling and understanding. IEEE Trans. Pattern Anal. Mach. Intell. 32(2), 258–273 (2010)
Tong, Y., Liao, W., Ji, Q.: Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Trans. Pattern Anal. Mach. Intell. 29(10), 1683–1699 (2007)
Valstar, M., Schuller, B., Smith, K., Eyben, F., Jiang, B., Bilakhia, S., Schnieder, S., Cowie, R., Pantic, M.: Avec 2013: the continuous audio/visual emotion and depression recognition challenge. In: Proceedings of the 3rd ACM international workshop on Audio/visual emotion challenge, pp. 3–10. ACM (2013)
Valstar, M.F., Jiang, B., Mehu, M., Pantic, M., Scherer, K.: The first facial expression recognition and analysis challenge. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 921–926. IEEE (2011)
Vapnik, V.N., Vapnik, V.: Statistical Learning Theory, vol. 1. Wiley, New York (1998)
Viola, P., Jones, M.J.: Robust real-time face detection. Int. J. Comput. Vis. 57(2), 137–154 (2004)
Vukadinovic, D., Pantic, M.: Fully automatic facial feature point detection using Gabor feature based boosted classifiers. In: 2005 IEEE International Conference on Systems, Man and Cybernetics, vol. 2, pp. 1692–1698. IEEE (2005)
Yang, S., Bhanu, B.: Facial expression recognition using emotion avatar image. In: 2011 IEEE International Conference on Automatic Face and Gesture Recognition and Workshops (FG 2011), pp. 866–871. IEEE (2011)
Zhang, T.: Facial expression recognition based on deep learning: a survey. In: International Conference on Intelligent and Interactive Systems and Applications, pp. 345–352. Springer (2017)
Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Trans. Pattern Anal. Mach. Intell. 29(6), 915–928 (2007)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Al-Garaawi, N., Wu, Q. & Morris, T. BRIEF-based face descriptor: an application to automatic facial expression recognition (AFER). SIViP 15, 371–379 (2021). https://doi.org/10.1007/s11760-020-01759-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11760-020-01759-4