Skip to main content
SpringerLink
Log in

MimicME: A Large Scale Diverse 4D Database for Facial Expression Analysis

  • Conference paper
  • First Online:
Book cover Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13668))

Included in the following conference series:

Abstract

Recently, Deep Neural Networks (DNNs) have been shown to outperform traditional methods in many disciplines such as computer vision, speech recognition and natural language processing. A prerequisite for the successful application of DNNs is the big number of data. Even though various facial datasets exist for the case of 2D images, there is a remarkable absence of datasets when we have to deal with 3D faces. The available facial datasets are limited either in terms of expressions or in the number of subjects. This lack of large datasets hinders the exploitation of the great advances that DNNs can provide. In this paper, we overcome these limitations by introducing MimicMe, a novel large-scale database of dynamic high-resolution 3D faces. MimicMe contains recordings of 4, 700 subjects with a great diversity on age, gender and ethnicity. The recordings are in the form of 4D videos of subjects displaying a multitude of facial behaviours, resulting to over 280, 000 3D meshes in total. We have also manually annotated a big portion of these meshes with 3D facial landmarks and they have been categorized in the corresponding expressions. We have also built very powerful blendshapes for parameterising facial behaviour. MimicMe will be made publicly available upon publication and we envision that it will be extremely valuable to researchers working in many problems of face modelling and analysis, including 3D/4D face and facial expression recognition\(^{\dagger }\). We conduct several experiments and demonstrate the usefulness of the database for various applications. (\(^{\dagger }\)https://github.com/apapaion/mimicme)

A. Papaioannou, B. Gecer, S. Cheng, G. Chrysos, J. Deng, E. Fotiadou, C. Kampouris, D. Kollias, S. Moschoglou, K. Songsri-In, S. Ploumpis, G. Trigeorgis, P. Tzirakis, E. Ververas, Y. Zhou were with Imperial College London during this work.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://3dmd.com/.

  2. 2.

    https://github.com/menpo/landmarker.io.

References

  1. Abrevaya, V.F., Wuhrer, S., Boyer, E.: Multilinear autoencoder for 3D face model learning. In: WACV 2018-IEEE Winter Conference on Applications of Computer Vision (2018)

    Google Scholar 

  2. Amberg, B., Knothe, R., Vetter, T.: Expression invariant 3D face recognition with a morphable model. In: 8th IEEE International Conference on Automatic Face & Gesture Recognition, 2008. FG 2008, pp. 1–6. IEEE (2008)

    Google Scholar 

  3. Amberg, B., Romdhani, S., Vetter, T.: Optimal step nonrigid ICP algorithms for surface registration. In: IEEE Conference on Computer Vision and Pattern Recognition, 2007, CVPR2007, pp. 1–8. IEEE (2007)

    Google Scholar 

  4. Blanz, V., Basso, C., Poggio, T., Vetter, T.: Reanimating faces in images and video. In: Computer Graphics Forum, vol. a22, pp. 641–650. Wiley Online Library (2003)

    Google Scholar 

  5. Blanz, V., Vetter, T.: A morphable model for the synthesis of 3D faces. In: Proceedings of the 26th Annual Conference on computer Graphics and Interactive Techniques, pp. 187–194. ACM Press/Addison-Wesley Publishing Co. (1999)

    Google Scholar 

  6. Bolkart, T., Wuhrer, S.: 3d faces in motion: fully automatic registration and statistical analysis. Comput. Vis. IDmage Underst. 131, 100–115 (2015)

    Article  Google Scholar 

  7. Bolkart, T., Wuhrer, S.: A robust multilinear model learning framework for 3d faces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4911–4919 (2016)

    Google Scholar 

  8. Booth, J., Roussos, A., Ponniah, A., Dunaway, D., Zafeiriou, S.: Large scale 3D morphable models. Int. J. Comput. Vision 126(2–4), 233–254 (2018)

    Article  MathSciNet  Google Scholar 

  9. Booth, J., Roussos, A., Zafeiriou, S., Ponniah, A., Dunaway, D.: A 3D morphable model learnt from 10,000 faces. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5543–5552 (2016)

    Google Scholar 

  10. Bouritsas, G., Bokhnyak, S., Ploumpis, S., Bronstein, M., Zafeiriou, S.: Neural 3D morphable models: spiral convolutional networks for 3D shape representation learning and generation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 7213–7222 (2019)

    Google Scholar 

  11. Brunton, A., Salazar, A., Bolkart, T., Wuhrer, S.: Review of statistical shape spaces for 3d data with comparative analysis for human faces. Comput. Vis. Image Underst. 128, 1–17 (2014)

    Article  Google Scholar 

  12. Cao, C., Weng, Y., Zhou, S., Tong, Y., Zhou, K.: Facewarehouse: a 3d facial expression database for visual computing. IEEE Trans. Visual Comput. Graphics 20(3), 413–425 (2014)

    Article  Google Scholar 

  13. Cheng, S., et al.: MeshGAN: non-linear 3D morphable models of faces. arXiv preprint arXiv:1903.10384 (2019)

  14. Cheng, S., Kotsia, I., Pantic, M., Zafeiriou, S.: 4DFAB: a large scale 4d database for facial expression analysis and biometric applications. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018

    Google Scholar 

  15. Chung, J.S., Senior, A., Vinyals, O., Zisserman, A.: Lip reading sentences in the wild. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3444–3453. IEEE (2017)

    Google Scholar 

  16. Cosker, D., Krumhuber, E., Hilton, A.: A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling. In: 2011 International Conference on Computer Vision, pp. 2296–2303 (2011). https://doi.org/10.1109/ICCV.2011.6126510

  17. Cudeiro, D., Bolkart, T., Laidlaw, C., Ranjan, A., Black, M.J.: Capture, learning, and synthesis of 3d speaking styles. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10101–10111 (2019)

    Google Scholar 

  18. Dai, H., Pears, N., Smith, W., Duncan, C.: A 3D morphable model of craniofacial shape and texture variation. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 3104–3112. IEEE (2017)

    Google Scholar 

  19. Egger, B.: 3d morphable face models-past, present, and future. ACM Trans. on Grap. 39(5), 1–38 (2020)

    Article  Google Scholar 

  20. Ferrari, C., Lisanti, G., Berretti, S., Del Bimbo, A.: Dictionary learning based 3D morphable model construction for face recognition with varying expression and pose. In: International Conference on 3D Vision (3DV), pp. 509–517. IEEE (2015)

    Google Scholar 

  21. Gecer, B., Deng, J., Zafeiriou, S.: OSTeC: one-shot texture completion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7628–7638 (2021)

    Google Scholar 

  22. Gecer, B., Lattas, A., Ploumpis, S., Deng, J., Papaioannou, A., Moschoglou, S., Zafeiriou, S.: Synthesizing Coupled 3D Face Modalities by Trunk-Branch Generative Adversarial Networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12374, pp. 415–433. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58526-6_25

    Chapter  Google Scholar 

  23. Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.: GanFit: GEnerative adversarial network fitting for high fidelity 3d face reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1155–1164 (2019)

    Google Scholar 

  24. Gecer, B., Ploumpis, S., Kotsia, I., Zafeiriou, S.P.: Fast-GANFit: gnerative adversarial network for high fidelity 3D face reconstruction. IEEE Trans Pattern Anal. Mach. Intell. 44, 4879–4893 (2021)

    Google Scholar 

  25. Gilani, S.Z., Mian, A., Shafait, F., Reid, I.: Dense 3d face correspondence. IEEE Trans. Pattern Anal. Mach. Intell. 40(7), 1584–1598 (2017)

    Article  Google Scholar 

  26. bibitemch27gong19 Gong, S., Chen, L., Bronstein, M., Zafeiriou, S.: SpiralNet++: a fast and highly efficient mesh convolution operator. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 0–0 (2019)

    Google Scholar 

  27. Guo, Y., Cai, J., Jiang, B., Zheng, J., et al.: Cnn-based real-time dense face reconstruction with inverse-rendered photo-realistic face images. IEEE Trans. Pattern Anal. Mach. Intell. 41(6), 1294–1307 (2018)

    Article  Google Scholar 

  28. Ichim, A.E., Kadleček, P., Kavan, L., Pauly, M.: Phace: physics-based face modeling and animation. ACM Transactions on Graphics (TOG) 36(4), 1–14 (2017)

    Article  Google Scholar 

  29. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of stylegan. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110–8119 (2020)

    Google Scholar 

  30. Knoops, P.G., et al.: A machine learning framework for automated diagnosis and computer-assisted planning in plastic and reconstructive surgery. Sci. Rep.D 9(1), 1–12 (2019)

    Google Scholar 

  31. Koppen, P., et al.: Gaussian mixture 3d morphable face model. Pattern Recogn. 74, 617–628 (2018)

    Article  Google Scholar 

  32. Li, T., Bolkart, T., Black, M.J., Li, H., Romero, J.: Learning a model of facial shape and expression from 4d scans. ACM Trans. Graph. 36(6), 194 (2017)

    Article  Google Scholar 

  33. Lüthi, M., Gerig, T., Jud, C., Vetter, T.: Gaussian process morphable models. IEEE Trans. Pattern Anal. Mach. Intell. 40, 1860–1873 (2017)

    Google Scholar 

  34. Marshall, A.D., Rosin, P.L., Vandeventer, J., Aubrey, A.: 4D Cardiff conversation database (4D CCDB): a 4D database of natural, dyadic conversations. Audit. Vis. Speech Process. \(\{\)AVSP\(\}\)2015, 157–162 (2015)

    Google Scholar 

  35. Moschoglou, S., Ploumpis, S., Nicolaou, M.A., Papaioannou, A., Zafeiriou, S.: 3dfacegan: Adversarial nets for 3d face representation, generation, and translation. Int. J. Comput. Vision 128, 2534–2551 (2020)

    Article  Google Scholar 

  36. Myronenko, A., Song, X.: Point set registration: coherent point drift. IEEE Trans. Pattern Anal. Mach. Intell. 32(12), 2262–2275 (2010)

    Article  Google Scholar 

  37. Neumann, T., Varanasi, K., Wenger, S., Wacker, M., Magnor, M., Theobalt, C.: Sparse localized deformation components. ACM Trans. Graph. 32(6), 179 (2013)

    Article  Google Scholar 

  38. O’Sullivan, E., et al.: The 3D skull 0–4 years: a validated, generative, statistical shape model. Bone Rep. 15 (2021)

    Google Scholar 

  39. O’Sullivan, E., et al.: Convolutional mesh autoencoders for the 3-dimensional identification of FGFR-related craniosynostosis. Sci. Rep. 12(1), 1–8 (2022)

    Google Scholar 

  40. Patel, A., Smith, W.A.: 3D morphable face models revisited. In: IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009, pp. 1327–1334. IEEE (2009)

    Google Scholar 

  41. Ranjan, A., Bolkart, T., Sanyal, S., Black, M.J.: Generating 3d faces using convolutional mesh autoencoders. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11207, pp. 725–741. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01219-9_43

    Chapter  Google Scholar 

  42. Savran, A., et al.: Bosphorus database for 3D face analysis. In: BIOID, pp. 47–56 (2008)

    Google Scholar 

  43. Slossberg, R., Shamai, G., Kimmel, R.: High quality facial surface and texture synthesis via generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, vol. 11131, pp. 498–513. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11015-4_36

    Chapter  Google Scholar 

  44. Staal, F.C., Ponniah, A.J., Angullia, F., Ruff, C., Koudstaal, M.J., Dunaway, D.: Describing crouzon and pfeiffer syndrome based on principal component analysis. J. Cranio-Maxillof. Surg. 43(4), 528–536 (2015). https://doi.org/10.1016/j.jcms.2015.02.005, http://www.sciencedirect.com/science/article/pii/S101051821500027X

  45. Thies, J., Zollhofer, M., Stamminger, M., Theobalt, C., Nießner, M.: Face2Face: real-time face capture and reenactment of RGB videos. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2387–2395 (2016)

    Google Scholar 

  46. Tran, L., Liu, X.: Nonlinear 3D face morphable model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7346–7355 (2018)

    Google Scholar 

  47. Tzirakis, P., Papaioannou, A., Lattas, A., Tarasiou, M., Schuller, B., Zafeiriou, S.: Synthesising 3D facial motion from in-the-wild-speech. In: 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020)(FG), pp. 627–634 (2020)

    Google Scholar 

  48. Vlasic, D., Brand, M., Pfister, H., Popović, J.: Face transfer with multilinear models. ACM Trans. Graph. 24(3), 426–433 (2005)

    Article  Google Scholar 

  49. Wang, M., Panagakis, Y., Snape, P., Zafeiriou, S.: Learning the multilinear structure of visual data. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4592–4600 (2017)

    Google Scholar 

  50. Yang, H., et al.: Facescape: a large-scale high quality 3D face dataset and detailed riggable 3D face prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 601–610 (2020)

    Google Scholar 

  51. Ye, Y., Song, Z., Guo, J., Qiao, Y.: Siat-3dfe: A high-resolution 3d facial expression dataset. IEEE Access 8, 48205–48211 (2020)

    Article  Google Scholar 

  52. Yin, L., Chen, X., Sun, Y., Worm, T., Reale, M.: A high-resolution 3D dynamic facial expression database. In: 2008 8th IEEE International Conference on Automatic Face Gesture Recognition, pp. 1–6 (2008). https://doi.org/10.1109/AFGR.2008.4813324

  53. Yin, L., Wei, X., Sun, Y., Wang, J., Rosato, M.J.: A 3D facial expression database for facial behavior research. In: 7th International Conference on Automatic Face and Gesture Recognition (FGR 2006), pp. 211–216. IEEE (2006)

    Google Scholar 

  54. Zhang, J., Fisher, R.B.: 3d visual passcode: Speech-driven 3d facial dynamics for behaviometrics. Signal Process. 160, 164–177 (2019)

    Article  Google Scholar 

  55. Zhang, K., Zhang, Z., Li, Z., Qiao, Y.: Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process. Lett. 23(10), 1499–1503 (2016)

    Article  Google Scholar 

  56. Zhang, X., Yin, L., Cohn, J.F., Canavan, S., Reale, M., Horowitz, A., Liu, P., Girard, J.M.: Bp4d-spontaneous: a high-resolution spontaneous 3d dynamic facial expression database. Image Vis. Comput. 32(10), 692–706 (2014)

    Article  Google Scholar 

  57. Zhu, X., Lei, Z., Liu, X., Shi, H., Li, S.Z.: Face alignment across large poses: a 3D solution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 146–155 (2016)

    Google Scholar 

  58. Zollhöfer, M., et al.: State of the art on monocular 3D face reconstruction, tracking, and applications. In: Computer Graphics Forum, vol. 37, pp. 523–550. Wiley Online Library (2018)

    Google Scholar 

  59. Zulqarnain Gilani, S., Shafait, F., Mian, A.: Shape-based automatic detection of a large number of 3D facial landmarks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4639–4648 (2015)

    Google Scholar 

Download references

Acknowledgements

S. Zafeiriou and part of research was funded by the EPSRC Fellowship DEFORM: Large Scale Shape Analysis of Deformable Models of Humans (EP/S010203/1).

Author information

Authors and Affiliations

  1. University of Exeter, Exeter, UK

    Anastasios Roussos

  2. Institute of Computer Science, Foundation for Research and Technology Hellas, Heraklion, Greece

    Anastasios Roussos

  3. Imperial College London, London, UK

    Athanasios Papaioannou, Baris Gecer, Shiyang Cheng, Grigorios Chrysos, Jiankang Deng, Eftychia Fotiadou, Christos Kampouris, Dimitrios Kollias, Stylianos Moschoglou, Kritaphat Songsri-In, Stylianos Ploumpis, George Trigeorgis, Panagiotis Tzirakis, Evangelos Ververas, Yuxiang Zhou & Stefanos Zafeiriou

  4. Department of Plastic Surgery, Royal. Free Hospital, London, UK

    Allan Ponniah

Authors
  1. Athanasios Papaioannou
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Baris Gecer
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shiyang Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Grigorios Chrysos
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jiankang Deng
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Eftychia Fotiadou
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Christos Kampouris
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Dimitrios Kollias
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Stylianos Moschoglou
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Kritaphat Songsri-In
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Stylianos Ploumpis
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. George Trigeorgis
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. Panagiotis Tzirakis
    View author publications

    You can also search for this author in PubMed Google Scholar

  14. Evangelos Ververas
    View author publications

    You can also search for this author in PubMed Google Scholar

  15. Yuxiang Zhou
    View author publications

    You can also search for this author in PubMed Google Scholar

  16. Allan Ponniah
    View author publications

    You can also search for this author in PubMed Google Scholar

  17. Anastasios Roussos
    View author publications

    You can also search for this author in PubMed Google Scholar

  18. Stefanos Zafeiriou
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Athanasios Papaioannou .

Editor information

Editors and Affiliations

  1. Tel Aviv University, Tel Aviv, Israel

    Shai Avidan

  2. University College London, London, UK

    Gabriel Brostow

  3. Google AI, Accra, Ghana

    Moustapha Cissé

  4. University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Facebook (United States), Menlo Park, CA, USA

    Tal Hassner

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Papaioannou, A. et al. (2022). MimicME: A Large Scale Diverse 4D Database for Facial Expression Analysis. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13668. Springer, Cham. https://doi.org/10.1007/978-3-031-20074-8_27

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20074-8_27

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20073-1

  • Online ISBN: 978-3-031-20074-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation