Skip to main content
Springer Nature Link
Log in

A robust spatio-temporal scheme for dynamic 3D facial expression retrieval

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

The problem of facial expression recognition in dynamic sequences of 3D face scans has received a significant amount of attention in the recent past whereas the problem of retrieval in this type of data has not. A novel retrieval scheme for such data is introduced in this paper. It is the first spatio-temporal retrieval scheme ever used for retrieval in dynamic sequences of 3D face scans. The proposed scheme automatically detects specific facial landmarks and uses them to create a spatio-temporal descriptor. At first, geometric as well as topological information of the 3D face scans is captured by using the detected landmarks. In the sequel, the aforementioned spatial information is filtered by using wavelet transformation, resulting to our final spatio-temporal descriptor. Our descriptor is invariant to the number of the 3D face scans of a facial expression sequence. The proposed retrieval scheme exploits the Square of Euclidean distance in order to compare descriptors corresponding to different 3D facial sequences. A detailed evaluation of the introduced retrieval scheme is presented showing that it outperforms previous state-of-the-art retrieval schemes. Experiments have been conducted using the six prototypical expressions of the standard data set \(\textit{BU}-4\textit{DFE}\). Finally, a majority voting methodology based on the retrieval results is used to achieve unsupervised dynamic 3D facial expression recognition. The achieved classification accuracy outperforms the state-of-the-art supervised dynamic 3D facial expression recognition techniques.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Notes

  1. 4D will refer to 3D+ time (dynamic 3D); each element of such a sequence is a 3D frame.

References

  1. Berretti, S., Del Bimbo, A., Pala, P.: Automatic facial expression recognition in real-time from dynamic sequences of 3D face scans. Vis. Comput. 29(12), 1333–1350 (2013)

    Article  Google Scholar 

  2. Bovik, A.C.: Handbook of image and video processing (communications, networking and multimedia). Academic Press Inc., Orlando (2005)

    Google Scholar 

  3. Canavan, S.J., Sun, Y., Zhang, X., Yin, L.: A dynamic curvature based approach for facial activity analysis in 3D space. In: CVPR Workshops, pp. 14–19 (2012)

  4. Danelakis, A., Theoharis, T., Pratikakis, I.: Geotopo: dynamic 3D facial expression retrieval using topological and geometric information. In: Proceedings of the 3D Object Retrieval 2014 Workshop, pp. 1–8 (2014)

  5. Danelakis, A., Theoharis, T., Pratikakis, I.: A survey on facial expression recognition in 3D video sequences. Multimed. Tools. Appl. pp. 1–39 (2014)

  6. Daubechies, I.: Ten lectures on wavelets. Society for Industrial and Applied Mathematics, Philadelphia (1992)

    Book  MATH  Google Scholar 

  7. Drira, H., Ben Amor, B., Daoudi, M., Srivastava, A., Berretti, S.: 3D Dynamic expression recognition based on a novel deformation vector field and random forest. In: ICPR ’12, pp. 1104–1107 (2012)

  8. Ekman, P., Friesen, W.: Facial action coding system: A technique for the measurement of facial movement. Consulting Psychologists Press, Palo Alto (1978)

    Google Scholar 

  9. Fang, T., Zhao, X., Ocegueda, O., Shah, S.K., Kakadiaris, I.A.: 3D/4D facial expression analysis: an advanced annotated face model approach. Image Vis. Comput. 30(10), 738–749 (2012)

    Article  Google Scholar 

  10. Fang, T., Zhao, X., Shah, S.K., Kakadiaris, I.A.: 4D Facial expression recognition. In: ICCV ’11, pp. 1594–1601 (2011)

  11. Gebal, K., Bærentzen, J.A., Aanæs, H., Larsen, R.: Shape analysis using the auto diffusion function. In: Proceedings of the Symposium on Geometry Processing, SGP ’09, pp. 1405–1413 (2009)

  12. Haar, F., Veltkamp, R.: 3D Face model fitting for recognition. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 652–664 (2008)

  13. Jeni, L.A., Lórincz, A., Nagy, T., Palotai, Z., Sebók, J., Szabó, Z., Takács, D.: 3D Shape estimation in video sequences provides high precision evaluation of facial expressions. Image Vis. Comput. 30(10), 785–795 (2012)

    Article  Google Scholar 

  14. Matuszewski, B., Quan, W., Shark, L., McLoughlin, A., Lightbody, C., Emsley, H., Watkins, C.: Hi4D-ADSIP 3D dynamic facial articulation database. Elsevier Image Vis. Comput. 30(10), 713–727 (2012)

    Article  Google Scholar 

  15. Passalis, G., Theoharis, T., Kakadiaris, I.A.: PTK: a novel depth buffer-based shape descriptor for three-dimensional object retrieval. Vis. Comput. 23(1), 5–14 (2007)

    Article  Google Scholar 

  16. Perakis, P., Passalis, G., Theoharis, T., Kakadiaris, I.A.: 3D Facial landmark detection under large yaw and expression variations. IEEE Trans. Pattern Anal. Mach. Intell. 35(7), 1552–1564 (2013)

    Article  Google Scholar 

  17. Perakis, P., Theoharis, T., Kakadiaris, I.A.: Feature fusion for facial landmark detection. Pattern Recognit. 47(9), 2783–2793 (2014)

    Article  Google Scholar 

  18. Quiroga, R.Q., Sakowitz, O.W., Basar, E., Schürmann, M.: Wavelet transform in the analysis of the frequency composition of evoked potentials. Brain Res. Protoc. 8(1), 16–24 (2001)

    Article  Google Scholar 

  19. Sandbach, G., Zafeiriou, S., Pantic, M., Rueckert, D.: Recognition of 3D facial expression dynamics. Elsevier Image Vis. Comput. 30(10), 762–773 (2012)

    Article  Google Scholar 

  20. Sfikas, K., Theoharis, T., Pratikakis, I.: Non-rigid 3D object retrieval using topological information guided by conformal factors. Vis. Comput. 28(9), 943–955 (2012)

    Article  Google Scholar 

  21. Sfikas, K., Theoharis, T., Pratikakis, I.: 3D Object retrieval via range image queries in a bag-of-visual-words context. Vis. Comput. 29(12), 1351–1361 (2013)

    Article  Google Scholar 

  22. Sun, J., Ovsjanikov, M., Guibas, L.: A concise and provably informative multi-scale signature based on heat diffusion. In: Proceedings of the Symposium on Geometry Processing, SGP ’09, pp. 1383–1392. Eurographics Association (2009)

  23. Sun, Y., Chen, X., Rosato, M.J., Yin, L.: Tracking vertex flow and model adaptation for three-dimensional spatiotemporal face analysis. IEEE Trans. Syst. Man Cybern. Part A 40(3), 461–474 (2010)

    Article  Google Scholar 

  24. Sun, Y., Reale, M., Yin, L.: Recognizing partial facial action units based on 3D dynamic range data for facial expression recognition. In: FG ’08, pp. 1–8 (2008)

  25. Sun, Y., Yin, L.: Facial expression recognition based on 3D dynamic range model sequences. In: Springer Proceedings of the ECCV ’08: Part II, pp. 58–71 (2008)

  26. Tsalakanidou, F., Malassiotis, S.: Robust facial action recognition from real-time 3D streams. In: CVPR ’09, pp. 4–11 (2009)

  27. Tsalakanidou, F., Malassiotis, S.: Real-time 2D + 3D facial action and expression recognition. Elsevier Pattern Recognit. 43(5), 1763–1775 (2010)

    Article  Google Scholar 

  28. Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: IEEE Proceedings of the FG ’08, pp. 1–6 (2008)

  29. Yin, L., Wei, X., Longo, P., Bhuvanesh, A.: Analyzing facial expressions using intensity-variant 3D data for human computer interaction. In: Proceedings of the ICPR ’06, pp. 1248–1251 (2006)

  30. Zhang, X., Reale, M., Yin, L.: Nebula feature: A space-time feature for posed and spontaneous 4D facial behavior analysis. In: IEEE FG ’13 (2013)

  31. Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P.: A high-resolution spontaneous 3D dynamic facial expression database. In: IEEE FG ’13 (2013)

Download references

Acknowledgments

This research has been co-financed by the European Union (European Social Fund—ESF) and Greek national funds through the Operational Program “Education and Lifelong Learning” of the National Strategic Reference Framework (NSRF)—Research Funding Program: THALES-3DOR (MIS 379516).

In addition, special thanks should be dedicated to Takis Perakis, researcher of the Norwegian University of Science and Technology, for his precious help with the facial landmark detector.

Author information

Authors and Affiliations

  1. Department of Informatics and Telecommunications, University of Athens, Athens, Greece

    Antonios Danelakis & Theoharis Theoharis

  2. Department of Computer and Information Science, Norwegian University of Science and Technology, Trondheim, Norway

    Theoharis Theoharis

  3. Department of Electrical and Computer Engineering, Democritus University of Thrace, 67100, Xanthi, Greece

    Ioannis Pratikakis

Authors
  1. Antonios Danelakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Theoharis Theoharis
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ioannis Pratikakis
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Antonios Danelakis.

Ethics declarations

Conflict of Interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Danelakis, A., Theoharis, T. & Pratikakis, I. A robust spatio-temporal scheme for dynamic 3D facial expression retrieval. Vis Comput 32, 257–269 (2016). https://doi.org/10.1007/s00371-015-1142-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-015-1142-7

Keywords