skip to main content
10.1145/3379337.3415877acmconferencesArticle/Chapter ViewAbstractPublication PagesuistConference Proceedingsconference-collections
research-article

Interactive Exploration and Refinement of Facial Expression using Manifold Learning

Published: 20 October 2020 Publication History

Abstract

Posing expressive 3D faces is extremely challenging. Typical facial rigs have upwards of 30 controllable parameters, that while anatomically meaningful, are hard to use due to redundancy of expression, unrealistic configurations, and many semantic and stylistic correlations between the parameters. We propose a novel interface for rapid exploration and refinement of static facial expressions, based on a data-driven face manifold of natural expressions. Rapidly explored face configurations are interactively projected onto this manifold of meaningful expressions. These expressions can then be refined using a 2D embedding of nearby faces, both on and off the manifold. Our validation is fourfold: we show expressive face creation using various devices; we verify that our learnt manifold transcends its training face, to expressively control very different faces; we perform a crowd-sourced study to evaluate the quality of manifold face expressions; and we report on a usability study that shows our approach is an effective interactive tool to author facial expression.

Supplementary Material

VTT File (3379337.3415877.vtt)
ZIP File (ufp7277aux.mp4.zip)
Video
MP4 File (ufp7277pv.mp4)
Preview video
MP4 File (ufp7277vf.mp4)
Video figure
MP4 File (3379337.3415877.mp4)
Presentation Video

References

[1]
Rinat Abdrashitov, Alec Jacobson, and Karan Singh. 2019. A system for efficient 3D printed stop-motion face animation. ACM Transactions on Graphics (TOG) 39, 1 (2019), 1--11.
[2]
Stephen W Bailey, Dave Otte, Paul Dilorenzo, and James F O'Brien. 2018. Fast and deep deformation approximations. ACM Transactions on Graphics (TOG) 37, 4 (2018), 1--12.
[3]
Steve Bako, Thijs Vogels, Brian Mcwilliams, Mark Meyer, Jan NováK, Alex Harvill, Pradeep Sen, Tony Derose, and Fabrice Rousselle. 2017. Kernel-predicting Convolutional Networks for Denoising Monte Carlo Renderings. ACM Trans. Graph. 36, 4, Article 97 (July 2017), 14 pages.
[4]
Tadas Baltruvs aitis, Marwa Mahmoud, and Peter Robinson. 2015. Cross-dataset learning and person-specific normalisation for automatic action unit detection. In Automatic Face and Gesture Recognition (FG), 2015 11th IEEE International Conference and Workshops on, Vol. 6. IEEE, 1--6.
[5]
Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis-Philippe Morency. 2018. Openface 2.0: Facial behavior analysis toolkit. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 59--66.
[6]
Kiran S Bhat, Rony Goldenthal, Yuting Ye, Ronald Mallet, and Michael Koperwas. 2013. High fidelity facial animation capture and retargeting with contours. In Proceedings of the 12th ACM SIGGRAPH/eurographics symposium on computer animation. ACM, 7--14.
[7]
Sofien Bouaziz, Yangang Wang, and Mark Pauly. 2013. Online modeling for realtime facial animation. ACM Transactions on Graphics (ToG) 32, 4 (2013), 1--10.
[8]
Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian interactive optimization approach to procedural animation design. In Proceedings of the 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation. Eurographics Association, 103--112.
[9]
Chen Cao, Yanlin Weng, Stephen Lin, and Kun Zhou. 2013. 3D shape regression for real-time facial animation. ACM Transactions on Graphics (TOG) 32, 4 (2013), 41.
[10]
Chakravarty R. Alla Chaitanya, Anton S. Kaplanyan, Christoph Schied, Marco Salvi, Aaron Lefohn, Derek Nowrouzezahrai, and Timo Aila. 2017. Interactive Reconstruction of Monte Carlo Image Sequences Using a Recurrent Denoising Autoencoder. ACM Trans. Graph. 36, 4, Article 98 (July 2017), 12 pages.
[11]
Ya Chang, Changbo Hu, Rogerio Feris, and Matthew Turk. 2006. Manifold based analysis of facial expression. Image and Vision Computing 24, 6 (2006), 605--614.
[12]
Ya Chang, Changbo Hu, and Matthew Turk. 2003. Manifold of facial expression. In AMFG. 28--35.
[13]
Ya Chang, Changbo Hu, and Matthew Turk. 2004. Probabilistic expression analysis on manifolds. In Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004., Vol. 2. IEEE, II--II.
[14]
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321--357.
[15]
Girum G Demisse, Djamila Aouada, and Björn Ottersten. 2018. Deformation-Based 3D Facial Expression Representation. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 14, 1s (2018), 1--22.
[16]
Zhigang Deng, Pei-Ying Chiang, Pamela Fox, and Ulrich Neumann. 2006. Animating blendshape faces by cross-mapping motion capture data. In Proceedings of the 2006 symposium on Interactive 3D graphics and games. ACM, 43--48.
[17]
Rosenberg Ekman. 1997. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). Oxford University Press, USA.
[18]
Jennifer Fernquist, Tovi Grossman, and George Fitzmaurice. 2011. Sketch-sketch revolution: an engaging tutorial system for guided sketching and application learning. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 373--382.
[19]
Pablo Garrido, Levi Valgaerts, Chenglei Wu, and Christian Theobalt. 2013. Reconstructing detailed dynamic face geometry from monocular video. ACM Trans. Graph. 32, 6 (2013), 158--1.
[20]
Sarah Gibson, Paul Beardsley, Wheeler Ruml, Thomas Kang, Brian Mirtich, Joshua Seims, William Freeman, Jessica Hodgins, Hanspeter Pfister, Joe Marks, and others. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. (1997).
[21]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT press.
[22]
X. Han, C. Gao, and Y. Yu. 2017. DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling. ACM Transactions on Graphics 36, 4 (July 2017).
[23]
Daniel Holden. 2018. Robust Solving of Optical Motion Capture Data by Denoising. ACM Trans. Graph. 37, 4, Article 165 (July 2018), 12 pages.
[24]
Daniel Holden, Jun Saito, and Taku Komura. 2016. A deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG) 35, 4 (2016), 138.
[25]
Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. 2015. Learning motion manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 Technical Briefs. ACM, 18.
[26]
Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Jens Fursund, Iman Sadeghi, Carrie Sun, Yen-Chun Chen, and Hao Li. 2017. Avatar digitization from a single image for real-time rendering. ACM Transactions on Graphics (TOG) 36, 6 (2017), 195.
[27]
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014).
[28]
Kris N Kirby and Daniel Gerlanc. 2013. BootES: An R package for bootstrap confidence intervals on effect sizes. Behavior research methods 45, 4 (2013), 905--927.
[29]
Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered parameter analysis for visual design exploration. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 65--74.
[30]
Chris Landreth. 2013. Subconscious Password. National Film Board of Canada.
[31]
Manfred Lau, Jinxiang Chai, Ying-Qing Xu, and Heung-Yeung Shum. 2009. Face poser: Interactive modeling of 3D facial expressions using facial priors. ACM Transactions on Graphics (TOG) 29, 1 (2009), 3.
[32]
Neil D Lawrence. 2004. Gaussian process latent variable models for visualisation of high dimensional data. In Advances in neural information processing systems. 329--336.
[33]
Brian Lee, Savil Srivastava, Ranjitha Kumar, Ronen Brafman, and Scott R Klemmer. 2010. Designing with interactive example galleries. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 2257--2266.
[34]
John P Lewis, Ken Anjyo, Taehyun Rhee, Mengjie Zhang, Frederic H Pighin, and Zhigang Deng. 2014. Practice and Theory of Blendshape Facial Models. Eurographics (State of the Art Reports) 1, 8 (2014), 2.
[35]
John P Lewis and Ken-ichi Anjyo. 2010. Direct manipulation blendshapes. IEEE Computer Graphics and Applications 30, 4 (2010), 42--50.
[36]
John P Lewis, Jonathan Mooser, Zhigang Deng, and Ulrich Neumann. 2005. Reducing blendshape interference by selected motion attenuation. In Proceedings of the 2005 symposium on Interactive 3D graphics and games. 25--29.
[37]
Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. 2015. Facial performance sensing head-mounted display. ACM Transactions on Graphics (ToG) 34, 4 (2015), 47.
[38]
Hao Li, Jihun Yu, Yuting Ye, and Chris Bregler. 2013. Realtime facial animation with on-the-fly correctives. ACM Trans. Graph. 32, 4 (2013), 42--1.
[39]
Stuart Lloyd. 1982. Least squares quantization in PCM. IEEE transactions on information theory 28, 2 (1982), 129--137.
[40]
Karl F. MacDorman, Robert D. Green, Chin-Chang Ho, and Clinton T. Koch. 2009. Too real for comfort? Uncanny responses to computer generated faces. Computers in Human Behavior 25, 3 (2009).
[41]
Utkarsh Mall, G Roshan Lal, Siddhartha Chaudhuri, and Parag Chaudhuri. 2017. A deep recurrent framework for cleaning motion capture data. arXiv preprint arXiv:1712.03380 (2017).
[42]
Brais Martinez, Michel F Valstar, Bihan Jiang, and Maja Pantic. 2017. Automatic analysis of facial actions: A survey. IEEE transactions on affective computing (2017).
[43]
José Carlos Miranda, Xenxo Alvarez, Jo ao Orvalho, Diego Gutierrez, A Augusto Sousa, and Verónica Orvalho. 2011. Sketch express: facial expressions made easy. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling. ACM, 87--94.
[44]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NIPS-W.
[45]
Caifeng Shan, Shaogang Gong, and Peter W McOwan. 2005. Appearance manifold of facial expression. In International Workshop on Human-Computer Interaction. Springer, 221--230.
[46]
Ben Shneiderman. 2007. Creativity support tools: Accelerating discovery and innovation. Commun. ACM 50, 12 (2007), 20--32.
[47]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15, 1 (2014), 1929--1958.
[48]
Tanasai Sucontphunt, Zhenyao Mo, Ulrich Neumann, and Zhigang Deng. 2008. Interactive 3D facial expression posing through 2D portrait manipulation. In Proceedings of graphics interface 2008. Canadian Information Processing Society, 177--184.
[49]
Jerry O Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory modeling with collaborative design spaces. ACM Transactions on Graphics-TOG 28, 5 (2009), 167.
[50]
Nobuyuki Umetani. 2017. Exploring generative 3D shapes using autoencoder networks. In SIGGRAPH Asia 2017 Technical Briefs. 1--4.
[51]
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion. J. Mach. Learn. Res. 11 (Dec. 2010), 3371--3408. http://dl.acm.org/citation.cfm?id=1756006.1953039
[52]
Thibaut Weise, Sofien Bouaziz, Hao Li, and Mark Pauly. 2011. Realtime performance-based facial animation. In ACM transactions on graphics (TOG), Vol. 30. ACM, 77.
[53]
Lance Williams. 1990. Performance-driven facial animation. In ACM SIGGRAPH Computer Graphics, Vol. 24. ACM, 235--242.
[54]
Jun Xiao, Yinfu Feng, Mingming Ji, Xiaosong Yang, Jian J. Zhang, and Yueting Zhuang. 2015. Sparse Motion Bases Selection for Human Motion Denoising. Signal Process. 110, C (May 2015), 108--122.
[55]
Rui Xiao, Qijun Zhao, David Zhang, and Pengfei Shi. 2011. Facial expression recognition on multiple manifolds. Pattern Recognition 44, 1 (2011), 107--116.
[56]
Jun Xie, Aaron Hertzmann, Wilmot Li, and Holger Winnemöller. 2014. PortraitSketch: face sketching assistance for novices. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 407--417.
[57]
Junyuan Xie, Linli Xu, and Enhong Chen. 2012. Image denoising and inpainting with deep neural networks. In Advances in neural information processing systems. 341--349.
[58]
Raymond A Yeh, Chen Chen, Teck Yian Lim, Alexander G Schwing, Mark Hasegawa-Johnson, and Minh N Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5485--5493.
[59]
Eduard Zell, JP Lewis, Junyong Noh, Mario Botsch, and others. 2017. Facial retargeting with automatic range of motion alignment. ACM Transactions on Graphics (TOG) 36, 4 (2017), 154.
[60]
Károly Zsolnai-Fehér, Peter Wonka, and Michael Wimmer. 2018. Gaussian material synthesis. ACM Transactions on Graphics (TOG) 37, 4 (2018), 76.

Cited By

View all
  • (2024)EmoSpaceTime: Decoupling Emotion and Content through Contrastive Learning for Expressive 3D Speech AnimationProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696336(1-12)Online publication date: 21-Nov-2024
  • (2024)Natural Language Dataset Generation Framework for Visualizations Powered by Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642943(1-22)Online publication date: 11-May-2024
  • (2023)Large-scale Text-to-Image Generation Models for Visual Artists’ Creative WorksProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584078(919-933)Online publication date: 27-Mar-2023
  • Show More Cited By

Index Terms

  1. Interactive Exploration and Refinement of Facial Expression using Manifold Learning

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image ACM Conferences
      UIST '20: Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology
      October 2020
      1297 pages
      ISBN:9781450375146
      DOI:10.1145/3379337
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Sponsors

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 20 October 2020

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. 3d face modeling
      2. blendshape
      3. manifold learning

      Qualifiers

      • Research-article

      Funding Sources

      • Modeling Animation and Fabrication of 3D Human Faces NSERC CRD

      Conference

      UIST '20

      Acceptance Rates

      Overall Acceptance Rate 561 of 2,567 submissions, 22%

      Upcoming Conference

      UIST '25
      The 38th Annual ACM Symposium on User Interface Software and Technology
      September 28 - October 1, 2025
      Busan , Republic of Korea

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)28
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 16 Feb 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)EmoSpaceTime: Decoupling Emotion and Content through Contrastive Learning for Expressive 3D Speech AnimationProceedings of the 17th ACM SIGGRAPH Conference on Motion, Interaction, and Games10.1145/3677388.3696336(1-12)Online publication date: 21-Nov-2024
      • (2024)Natural Language Dataset Generation Framework for Visualizations Powered by Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642943(1-22)Online publication date: 11-May-2024
      • (2023)Large-scale Text-to-Image Generation Models for Visual Artists’ Creative WorksProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584078(919-933)Online publication date: 27-Mar-2023
      • (2022)Animatomy: an Animator-centric, Anatomically Inspired System for 3D Facial Modeling, Animation and TransferSIGGRAPH Asia 2022 Conference Papers10.1145/3550469.3555398(1-9)Online publication date: 29-Nov-2022
      • (2022)EAMM: One-Shot Emotional Talking Face via Audio-Based Emotion-Aware Motion ModelACM SIGGRAPH 2022 Conference Proceedings10.1145/3528233.3530745(1-10)Online publication date: 27-Jul-2022
      • (2022)We-toon: A Communication Support System between Writers and Artists in Collaborative Webtoon Sketch RevisionProceedings of the 35th Annual ACM Symposium on User Interface Software and Technology10.1145/3526113.3545612(1-14)Online publication date: 29-Oct-2022
      • (2022)Low-rank kernel decomposition for scalable manifold modeling2022 Joint 12th International Conference on Soft Computing and Intelligent Systems and 23rd International Symposium on Advanced Intelligent Systems (SCIS&ISIS)10.1109/SCISISIS55246.2022.10001865(1-6)Online publication date: 29-Nov-2022
      • (2021)DAG amendment for inverse control of parametric shapesACM Transactions on Graphics10.1145/3476576.347676040:4(1-14)Online publication date: Aug-2021
      • (2021)Optimizing UI layouts for deformable face-rig manipulationACM Transactions on Graphics10.1145/3476576.347675940:4(1-12)Online publication date: Aug-2021
      • (2021)The Intersection of Users, Roles, Interactions, and Technologies in Creativity Support ToolsProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3462050(1817-1833)Online publication date: 28-Jun-2021
      • Show More Cited By

      View Options

      Login options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Figures

      Tables

      Media

      Share

      Share

      Share this Publication link

      Share on social media