Skip to main content
SpringerLink
Log in

A spatio-temporal wavelet-based descriptor for dynamic 3D facial expression retrieval and recognition

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Human emotions are often expressed by facial expressions and are generated by facial muscle movements. In recent years, the analysis of facial expressions has emerged as an active research area due to its various applications such as human–computer interaction, human behavior understanding, biometrics, emotion recognition, computer graphics, driver fatigue detection, and psychology. A novel analysis of dynamic 3D facial expressions using the positional information of automatically detected facial landmarks and the wavelet transformation is presented, which results in the proposed spatio-temporal descriptor. This descriptor is employed within the current paper in a retrieval scheme for dynamic 3D facial expression datasets and is thoroughly evaluated. Experiments have been conducted using the six prototypical expressions of the publicly available BU-4DFE dataset as well as the eight expressions included in the newly released publicly available BP4D-Spontaneous dataset. The obtained retrieval results outperform the retrieval results of the state-of-the-art methodologies. Furthermore, the retrieval results are exploited to achieve unsupervised dynamic 3D facial expression recognition. The aforementioned unsupervised procedure achieves better recognition accuracy compared to supervised dynamic 3D facial expression recognition state-of-the-art techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Berretti, S., Del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis. Comput. 29(12), 1333–1350 (2013)

    Article  Google Scholar 

  2. Bovik, A.C.: Handbook of Image and Video Processing (Communications, Networking and Multimedia). Academic Press, Orlando (2005)

    Google Scholar 

  3. Canavan, S.J., Sun, Y., Zhang, X., Yin, L.: A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR Workshops, pp. 14–19 (2012)

  4. Ciregan, D., Meier, U., Schmidhuber, J.: Multi-column deep neural networks for image classification. In: Computer Vision and Pattern Recognition (CVPR), pp. 3642–3649 (2012)

  5. Danelakis, A., Theoharis, T., Pratikakis, I.: Geotopo: Dynamic 3D facial expression retrieval using topological and geometric information. In: Proc. 3D Object Retrieval 2014 Workshop, pp. 1–8 (2014)

  6. Danelakis, A., Theoharis, T., Pratikakis, I.: A robust spatio-temporal scheme for dynamic 3D facial expression retrieval. Vis. Comput. 32(2), 257–269 (2015). doi:10.1007/s00371-015-1142-7

    Article  Google Scholar 

  7. Danelakis, A., Theoharis, T., Pratikakis, I.: A spatio-temporal descriptor for dynamic 3D facial expression retrieval and recognition. In: Proc. 3D Object Retrieval 2015 Workshop, pp. 63–70 (2015)

  8. Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimedia Tools Appl. 74(15), 5577–5615 (2015)

    Article  Google Scholar 

  9. Danelakis, A., Theoharis, T., Pratikakis, I., Perakis, P.: An effective methodology for dynamic 3D facial expression retrieval. Pattern Recognit. 52, 174–185 (2016). doi:10.1016/j.patcog.2015.10.012

    Article  Google Scholar 

  10. Dapogny, A., Bailly, K., Dubuisson, S.: Dynamic facial expression recognition by joint static and multi-time gap transition classification. In: International Conference on Automatic Face and Gesture Recognition, Ljubljana, pp. 1–6 (2015)

  11. Daubechies, I.: Ten Lectures on Wavelets. Society for Industrial and Applied Mathematics, Philadelphia (1992)

    Book  MATH  Google Scholar 

  12. Drira, H., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: 3D dynamic expression recognition based on a novel deformation vector field and random forest. In: ICPR ’12, pp. 1104–1107 (2012)

  13. Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  14. Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012)

    Article  Google Scholar 

  15. Fang, T., Zhao, X., Shah, S.K., Kakadiaris, I.A.: 4D facial expression recognition. In: ICCV ’11, pp. 1594–1601 (2011)

  16. Jeni, L.A., Lórincz, A., Nagy, T., Palotai, Z., Sebók, J., Szabó, Z., Takács, D.: 3D shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis. Comput. 30(10), 785–795 (2012)

    Article  Google Scholar 

  17. Mallat, S.: A Wavelet Tour of Signal Processing, 3rd edn. The Sparse Way, 3rd edn. Academic Press, New York (2008)

    Google Scholar 

  18. Navarro, R., Tabernero, A.: Gaussian wavelet transform: two alternative fast implementations for images. Multidimens. Syst. Signal Process. 2(4), 421–436 (1991)

    Article  Google Scholar 

  19. Perakis, P., Passalis, G., Theoharis, T., Kakadiaris, I.A.: 3D facial landmark detection under large yaw and expression variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1552–1564 (2013)

    Article  Google Scholar 

  20. Perakis, P., Theoharis, T., Kakadiaris, I.A.: Feature fusion for facial landmark detection. Pattern Recognit. 47(9), 2783–2793 (2014)

    Article  Google Scholar 

  21. Quiroga, R.Q., Sakowitz, O.W., Basar, E., Schürmann, M.: Wavelet transform in the analysis of the frequency composition of evoked potentials. Brain Res. Protoc. 8(1), 16–24 (2001)

    Article  Google Scholar 

  22. Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Elsevier Image Vis. Comput. 30(10), 762–773 (2012)

    Article  Google Scholar 

  23. Sandbach, G., Zafeiriou, S., Pantic, M., Yin, L.: Static and dynamic 3D facial expression recognition: a comprehensive survey. Image Vis. Comput. 30(10), 683–697 (2012)

    Article  Google Scholar 

  24. Shilane, P., Min, P., Kazhdan, M., Funkhouser, T.: The Princeton shape benchmark. In: Shape Modeling International, pp. 167–178 (2004)

  25. Sun, Y., Chen, X., Rosato, M.J., Yin, L.: Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans. Syst. Man Cybern. A 40(3), 461–474 (2010)

    Article  Google Scholar 

  26. Sun, Y., Reale, M., Yin, L.: Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08, pp. 1–8 (2008)

  27. Sun, Y., Yin, L.: Facial expression recognition based on 3D dynamic range model sequences. In: Springer Proc. ECCV ’08: Part II, pp. 58–71 (2008)

  28. Tsalakanidou, F., Malassiotis, S.: Robust facial action recognition from real-time 3D streams. In: CVPR ’09, pp. 4–11 (2009)

  29. Tsalakanidou, F., Malassiotis, S.: Real-time 2D+3D facial action and expression recognition. Elsevier Pattern Recognit. 43(5), 1763–1775 (2010)

    Article  Google Scholar 

  30. Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: IEEE Proc. FG ’08, pp. 1–6 (2008)

  31. Yin, L., Wei, X., Longo, P., Bhuvanesh, A.: Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proc. ICPR ’06, pp. 1248–1251 (2006)

  32. Zhang, X., Reale, M., Yin, L.: Nebula feature: a space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13 (2013)

  33. Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.: A high-resolution spontaneous 3D dynamic facial expression database. In: IEEE FG ’13 (2013)

  34. Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.: BP4D Spontaneous: a high-resolution spontaneous 3D dynamic facial expression database. Image Vis. Comput. 32(10), 692–706 (2014)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Informatics and Telecommunications, University of Athens, Athens, Greece

    Antonios Danelakis & Theoharis Theoharis

  2. IDI, Norwegian University of Science and Technology (NTNU), Trondheim, Norway

    Theoharis Theoharis

  3. Department of Electrical and Computer Engineering, Democritus University of Thrace, Xanthi, Greece

    Ioannis Pratikakis

Authors
  1. Antonios Danelakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theoharis Theoharis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Pratikakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonios Danelakis.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Danelakis, A., Theoharis, T. & Pratikakis, I. A spatio-temporal wavelet-based descriptor for dynamic 3D facial expression retrieval and recognition. Vis Comput 32, 1001–1011 (2016). https://doi.org/10.1007/s00371-016-1243-y

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-016-1243-y

Keywords

Search

Navigation