skip to main content
research-article

Persona: A Method for Facial Analysis in Video and Application in Entertainment

Published: 12 September 2018 Publication History

Abstract

This article proposes the Persona method. The goal of the prosposed method is to learn and classify the facial actions of actors in video sequences. Persona is based on standard action units. We use a database with main expressions mapped and pre-classified that allows the automatic learning and faces selection. The learning stage uses Support Vector Machine (SVM) classifiers to identify expressions from a set of feature points tracked in the input video. After that, labeled control 3D masks are built for each selected action unit or expression, which composes the Persona structure. The proposed method is almost automatic (little intervention is needed) and does not require markers on the actor’s face or motion capture devices. Many applications are possible based on the Persona structure such as expression recognition, customized avatar deformation, and mood analysis, as discussed in this article.

References

[1]
R. Bennetts, J. Kim, D. Burke, K. Brooks, S. Lucey, J. Saragih, and R. Robbins. 2013. The movement advantage in famous and unfamiliar faces: A comparison of point-light displays and shape-normalised avatar stimuli. Perception 42, 9 (2013), 950--970.
[2]
R. J. Bennetts, D. Burke, K. Brooks, J. Kim, S. Lucey, J. Saragih, and R. A. Robbins. 2011. Avatars vs point-light faces: Movement matching is better without a face. In Proceedings of the 38th Australasian Experimental Psychology Conference.
[3]
Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer-Verlag New York, Inc., Secaucus, NJ.
[4]
Yong Cao, Petros Faloutsos, and Frédéric Pighin. 2003. Unsupervised learning for speech motion editing. In Proceedings of Eurographics/SIGGRAPH Symposium on Computer Animation.
[5]
Wen-Sheng Chu, F. de la Torre, and J. F. Cohn. 2013. Selective transfer machine for personalized facial action unit detection. In 2013 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 3515--3522.
[6]
E. S. Chuang, F. Deshpande, and C. Bregler. 2002. Facial expression space learning. In Proceedings of the 10th Pacific Conference on Computer Graphics and Applications, 2002. 68--76.
[7]
T. F. Cootes, G. J. Edwards, and C. J. Taylor. 1998. Active appearence models. In European Conference on Computer Vision 1998, Vol. II. Springer, 484--498.
[8]
Ludovic Dutreve, Alexandre Meyer, and Saïda Bouakaz. 2008. Feature points based facial animation retargeting. In VRST’08: Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology. ACM, New York, NY, 197--200.
[9]
Paul Ekman. 1999. Facial expressions. In Handbook of Cognition and Emotion, Dalgleish and M. J. Power (Eds.). Wiley, NY, 301--320.
[10]
Paul Ekman, W. V. Friesen, and J. C. Hager. 2002. The Facial Action Coding System. Weidenfeld 8 Nicolson.
[11]
Klaus Förger and Tapio Takala. 2016. Animating with style: Defining expressive semantics of motion. The Visual Computer 32, 2 (2016), 191--203.
[12]
Pablo Garrido, Levi Valgaerts, Ole Rehmsen, Thorsten Thormaehlen, Patrick Perez, and Christian Theobalt. 2014. Automatic face reenactment. In Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR’14). IEEE Computer Society, Washington, DC, 4217--4224.
[13]
Isabel Gonzalez, Hichem Sahli, Valentin Enescu, and Werner Verhelst. 2011. Context-independent facial action unit recognition using shape and Gabor phase information. In Affective Computing and Intelligent Interaction, Sidney DMello, Arthur Graesser, Bjorn Schuller, and Jean-Claude Martin (Eds.). Lecture Notes in Computer Science, Vol. 6974. Springer Berlin, 548--557.
[14]
Jihun Hamm, Christian G. Kohler, Ruben C. Gur, and Ragini Verma. 2011. Automated facial action coding system for dynamic analysis of facial expressions in neuropsychiatric disorders. Journal of Neuroscience Methods 200, 2 (2011), 237--256.
[15]
M. Julius Hossain, M. AliAkber Dewan, Kiok Ahn, and Oksam Chae. 2012. A linear time algorithm of computing Hausdorff distance for content-based image analysis. Circuits, Systems, and Signal Processing 31, 1 (2012), 389--399.
[16]
Chun-houh Chen, Wolfgang K. Härdle, and Antony Unwin. 2008. Handbook of Data Visualization. American Psychological Association.
[17]
Ira Kemelmacher-Shlizerman, Aditya Sankar, Eli Shechtman, and Steven M. Seitz. 2010. Being John Malkovich. In ECCV (1) (Lecture Notes in Computer Science), Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.), Vol. 6311. Springer, 341--353. http://dblp.uni-trier.de/db/conf/eccv/eccv2010-1.html#Kemelmacher-ShlizermanSSS10
[18]
Gengdai Liu, Zhigeng Pan, and Zuoyan Lin. 2008. Style subspaces for character animation. Comput. Animat. Virtual Worlds 19, 3--4 (Sept. 2008), 199--209.
[19]
Xiaohan Ma, Binh Huy Le, and Zhigang Deng. 2009. Style learning and transferring for facial animation editing. In Eurographics/ ACM SIGGRAPH Symposium on Computer Animation, Eitan Grinspun and Jessica Hodgins (Eds.). ACM SIGGRAPH / Eurographics Association.
[20]
M. H. Mahoor, Mu Zhou, Kevin L. Veon, S. M. Mavadati, and J. F. Cohn. 2011. Facial action unit recognition with sparse representation. In 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011). 336--342.
[21]
A. Mohammadian, H. Aghaeinia, and F. Towhidkhah. 2016. Diverse videos synthesis using manifold-based parametric motion model for facial understanding. IET Image Processing 10, 4 (2016), 253--260.
[22]
Takahiro Okamoto, Takaaki Shiratori, M. Glisson, K. Yamane, Shunsuke Kudoh, and Katsushi Ikeuchi. 2014. Extraction of person-specific motion style based on a task model and imitation by humanoid robot. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, September 14--18, 2014.
[23]
Igor S. Pandzic and Robert Forchheimer (Eds.). 2003. MPEG-4 Facial Animation: The Standard, Implementation and Applications. John Wiley 8 Sons, Inc., New York, NY.
[24]
Ottorino Pianigiani. 2012. Vocabolario Etimologico Della Lingua Italiana. Societa editrice Dante Alighieri di Albrighi.
[25]
Rossana Baptista Queiroz, Adriana Braun, Juliano Lucas Moreira, Marcelo Cohen, Soraia Raupp Musse, Marcelo Resende Thielo, and Ramin Samadani. 2010. Reflecting user faces in avatars. In Proceedings of the 10th International Conference on Intelligent Virtual Agents (IVA’10). Springer-Verlag, Berlin, 420--426. http://dl.acm.org/citation.cfm?id=1889075.1889127
[26]
R. B. Queiroz, A. Braun, and S. R. Musse. 2015. An adaptive methodology for facial expression transfer. In 2015 14th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames). 11--23.
[27]
D. Roark, A. O’Toole, H. Abdi, and S. Barrett. 2006. Learning the moves: The effect of familiarity and facial motion on person recognition across large changes in viewing format. Perception 35 (2006), 761--773.
[28]
M. Sanchez and S. Maddock. 2003. Planar bones for MPEG-4 facial animation. In TPCG’03: Proceedings of the Theory and Practice of Computer Graphics 2003. IEEE Computer Society, Washington, DC, 81.
[29]
Arman Savran, Neşe Alyüz, Hamdi Dibeklioğlu, Oya Çeliktutan, Berk Gökberk, Bülent Sankur, and Lale Akarun. 2008. Bosphorus database for 3D face analysis. In Biometrics and Identity Management, Ben Schouten, Niels Christian Juul, Andrzej Drygajlo, and Massimo Tistarelli (Eds.). Springer-Verlag, Berlin, 47--56.
[30]
Arman Savran, Bulent Sankur, and M. Taha Bilge. 2012. Regression-based intensity estimation of facial action units. Image and Vision Computing 30, 10 (2012), 774--784.
[31]
T. Senechal, V. Rapp, H. Salam, R. Seguier, K. Bailly, and L. Prevost. 2012. Facial action recognition combining heterogeneous features via multikernel learning. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics 42, 4 (Aug 2012), 993--1005.
[32]
Graham W. Taylor and Geoffrey E. Hinton. 2009. Factored conditional restricted Boltzmann machines for modeling motion style. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML’09). ACM, New York, NY, 1025--1032.
[33]
J. Thies, M. Zollhöfer, M. Nießner, L. Valgaerts, M. Stamminger, and C. Theobalt. 2015. Real-time expression transfer for facial reenactment. ACM Transactions on Graphics (TOG) 34, 6 (2015).
[34]
Yan Tong, Wenhui Liao, and Qiang Ji. 2007. Facial action unit recognition by exploiting their dynamic and semantic relationships. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 10 (2007), 1683--1699.
[35]
Lorenzo Torresani, Peggy Hackney, and Christoph Bregler. 2006. Learning motion style synthesis from perceptual observations. In NIPS. 1393--1400.
[36]
Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popović. 2005. Face transfer with multilinear models. ACM Trans. Graph. 24, 3 (July 2005), 426--433.
[37]
Yang Wang, Xiaolei Huang, Chan-Su Lee, Song Zhang, Zhiguo Li, Dimitris Samaras, Dimitris Metaxas, Ahmed Elgammal, and Peisen Huang. 2004. High resolution acquisition, learning and transfer of dynamic 3-D facial expressions. Computer Graphics Forum (2004).
[38]
Tingfan Wu, Nicholas J. Butko, Paul Ruvolo, Jacob Whitehill, Marian S. Bartlett, and Javier R. Movellan 2012. Multilayer architectures for facial action unit recognition. IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics, 42, 4 (Aug 2012), 1027--1038.
[39]
Peng Yang, Qingshan Liu, and Dimitris N. Metaxas. 2007. Boosting coded dynamic features for facial action units and facial expression recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2007. CVPR’07. IEEE, 1--6.

Cited By

View all
  • (2019)Investigating Emotion Style in Human Faces and Avatars2019 18th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames)10.1109/SBGames.2019.00025(115-124)Online publication date: Oct-2019

Index Terms

  1. Persona: A Method for Facial Analysis in Video and Application in Entertainment

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Computers in Entertainment
      Computers in Entertainment   Volume 16, Issue 3
      Theoretical and Practical Computer Applications in Entertainment
      September 2018
      127 pages
      EISSN:1544-3574
      DOI:10.1145/3236468
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 12 September 2018
      Accepted: 01 January 2018
      Received: 01 June 2017
      Published in CIE Volume 16, Issue 3

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Facial animation
      2. Support Vector Machine (SVM)
      3. facial components detection

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      • Brazilian research agencies CAPES and CNPq

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)14
      • Downloads (Last 6 weeks)0
      Reflects downloads up to 17 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2019)Investigating Emotion Style in Human Faces and Avatars2019 18th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames)10.1109/SBGames.2019.00025(115-124)Online publication date: Oct-2019

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media