Work in Progress

An Evaluation of Other-Avatar Facial Animation Methods for Social VR

Authors:

Peter Kullmann,

Marc Erich LatoschikAuthors Info & Claims

CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

Article No.: 33, Pages 1 - 7

https://doi.org/10.1145/3544549.3585617

Published: 19 April 2023 Publication History

Abstract

We report a mixed-design study on the effect of facial animation method (static, synthesized, or tracked expressions) and its synchronization to speaker audio (in sync or delayed by the method’s inherent latency) on an avatar’s perceived naturalness and plausibility. We created a virtual human for an actor and recorded his spontaneous half-minute responses to conversation prompts. As a simulated immersive interaction, 44 participants unfamiliar with the actor observed and rated performances rendered with the avatar, each with the different facial animation methods. Half of them observed performances in sync and the others with the animation method’s latency. Results show audio synchronization did not influence ratings and static faces were rated less natural and less plausible than animated faces. Notably, synthesized expressions were rated as more natural and more plausible than tracked expressions. Moreover, ratings of verbal behavior naturalness differed in the same way. We discuss implications of these results for avatar-mediated communication.

Supplementary Material

MP4 File (3544549.3585617-talk-video.mp4)

Pre-recorded Video Presentation

Download
17.15 MB

MP4 File (3544549.3585617-video-preview.mp4)

Video Preview

Download
2.84 MB

References

[1]

Jascha Achenbach, Thomas Waltemate, Marc Erich Latoschik, and Mario Botsch. 2017. Fast Generation of Realistic Virtual Humans. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (Gothenburg, Sweden) (VRST ’17). Association for Computing Machinery, New York, NY, USA, Article 12, 10 pages. https://doi.org/10.1145/3139131.3139154

Digital Library

[2]

Nalini Ambady and Robert Rosenthal. 1992. Thin Slices of Expressive Behavior as Predictors of Interpersonal Consequences: A Meta-Analysis.Psychological bulletin 111, 2 (1992), 256. https://doi.org/10.1037/0033-2909.111.2.256

[3]

Jeremy N. Bailenson and Jim Blascovich. 2004. Avatars. In Encyclopedia of Human-Computer Interaction. Vol. 1. Berkshire Publishing Group, 64–6.

[4]

Andrea Bartl, Stephan Wenninger, Erik Wolf, Mario Botsch, and Marc Erich Latoschik. 2021. Affordable But Not Cheap: A Case Study of the Effects of Two 3D-Reconstruction Methods of Virtual Humans. Frontiers in Virtual Reality 2 (Sept. 2021), 694617. https://doi.org/10.3389/frvir.2021.694617

[5]

David Borland, Tabitha Peck, and Mel Slater. 2013. An Evaluation of Self-Avatar Eye Movement for Virtual Embodiment. IEEE Transactions on Visualization and Computer Graphics 19, 4 (April 2013), 591–596. https://doi.org/10.1109/TVCG.2013.24

Digital Library

[6]

Jeffrey F. Cohn and Paul Ekman. 2005. Measuring Facial Action. In The New Handbook of Methods in Nonverbal Behavior Research.Oxford University Press, New York, NY, US, 9–64.

[7]

Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael Black. 2019. Capture, Learning, and Synthesis of 3D Speaking Styles. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR). 10101–10111. http://voca.is.tue.mpg.de/

[8]

Andy P. Field, Jeremy Miles, and Zoë Field. 2012. Discovering Statistics Using R. Sage, London ; Thousand Oaks, Calif.

Digital Library

[9]

Alexandra Frischen, Andrew P. Bayliss, and Steven P. Tipper. 2007. Gaze Cueing of Attention: Visual Attention, Social Cognition, and Individual Differences.Psychological Bulletin 133, 4 (2007), 694–724. https://doi.org/10.1037/0033-2909.133.4.694

[10]

Mar Gonzalez-Franco, Anthony Steed, Steve Hoogendyk, and Eyal Ofek. 2020. Using Facial Animation to Increase the Enfacement Illusion and Avatar Self-Identification. IEEE Transactions on Visualization and Computer Graphics 26, 5 (May 2020), 2023–2029. https://doi.org/10.1109/TVCG.2020.2973075

[11]

Man Jin, Adam Polis, and Jonathan Hartzel. 2021. Algorithms for Minimization Randomization and the Implementation with an R Package. Communications in Statistics - Simulation and Computation 50, 10 (Oct. 2021), 3077–3087. https://doi.org/10.1080/03610918.2019.1619765

[12]

Robert S Kennedy, Norman E Lane, Kevin S Berbaum, and Michael G Lilienthal. 1993. Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. The international journal of aviation psychology 3, 3 (1993), 203–220. https://doi.org/10.1207/s15327108ijap0303_3

[13]

Chris L. Kleinke. 1986. Gaze and Eye Contact: A Research Review.Psychological Bulletin 100, 1 (1986), 78–100. https://doi.org/10.1037/0033-2909.100.1.78

[14]

Hiromi Kobayashi and Shiro Kohshima. 2001. Unique Morphology of the Human Eye and Its Adaptive Meaning: Comparative Studies on External Morphology of the Primate Eye. Journal of Human Evolution 40, 5 (May 2001), 419–435. https://doi.org/10.1006/jhev.2001.0468

[15]

Marc Erich Latoschik and Carolin Wienrich. 2022. Congruence and Plausibility, Not Presence: Pivotal Conditions for XR Experiences and Effects, a Novel Approach. Frontiers in Virtual Reality 3 (June 2022), 694433. https://doi.org/10.3389/frvir.2022.694433

[16]

Gilwoo Lee, Zhiwei Deng, Shugao Ma, Takaaki Shiratori, Siddhartha S. Srinivasa, and Yaser Sheikh. 2019. Talking With Hands 16.2M: A Large-Scale Dataset of Synchronized Body-Finger Motion and Audio for Conversational Motion Analysis and Synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 763–772. https://doi.org/10.1109/ICCV.2019.00085

[17]

Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, Pei-Lun Hsieh, Aaron Nicholls, and Chongyang Ma. 2015. Facial Performance Sensing Head-Mounted Display. ACM Transactions on Graphics 34, 4 (July 2015), 1–9. https://doi.org/10.1145/2766939

Digital Library

[18]

David Mal, Erik Wolf, Nina Dollinger, Mario Botsch, Carolin Wienrich, and Marc Erich Latoschik. 2022. Virtual Human Coherence and Plausibility – Towards a Validated Scale. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE. https://doi.org/10.1109/vrw55335.2022.00245

[19]

David Matsumoto and Hyisung C. Hwang. 2018. Microexpressions Differentiate Truths From Lies About Future Malicious Intent. Frontiers in Psychology 9 (Dec. 2018), 2545. https://doi.org/10.3389/fpsyg.2018.02545

[20]

Harry McGurk and John MacDonald. 1976. Hearing Lips and Seeing Voices. Nature 264, 5588 (1976), 746–748. https://doi.org/10.1038/264746a0

[21]

Timo Menzel, Mario Botsch, and Marc Erich Latoschik. 2022. Automated Blendshape Personalization for Faithful Face Animations Using Commodity Smartphones. In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology (Tsukuba, Japan) (VRST ’22). Association for Computing Machinery, New York, NY, USA, Article 22, 9 pages. https://doi.org/10.1145/3562939.3565622

Digital Library

[22]

Maria Murcia-Lopez, Tara Collingwoode-Williams, William Steptoe, Raz Schwartz, Timothy J. Loving, and Mel Slater. 2020. Evaluating Virtual Reality Experiences Through Participant Choices. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Atlanta, GA, USA, 747–755. https://doi.org/10.1109/VR46266.2020.00098

[23]

Geoff Norman. 2010. Likert Scales, Levels of Measurement and the “Laws” of Statistics. Advances in Health Sciences Education 15, 5 (Dec. 2010), 625–632. https://doi.org/10.1007/s10459-010-9222-y

[24]

Catherine Oh Kruzic, David Kruzic, Fernanda Herrera, and Jeremy Bailenson. 2020. Facial Expressions Contribute More than Body Movements to Conversational Outcomes in Avatar-Mediated Virtual Environments. Scientific Reports 10, 1 (Dec. 2020), 20626. https://doi.org/10.1038/s41598-020-76672-4

[25]

Kyle Olszewski, Joseph J. Lim, Shunsuke Saito, and Hao Li. 2016. High-Fidelity Facial and Speech Animation for VR HMDs. ACM Transactions on Graphics 35, 6 (Nov. 2016), 1–14. https://doi.org/10.1145/2980179.2980252

Digital Library

[26]

José Pinheiro, Douglas Bates, and R Core Team. 2022. nlme: Linear and Nonlinear Mixed Effects Models. https://CRAN.R-project.org/package=nlme R package version 3.1-161.

[27]

R Core Team. 2022. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

[28]

Alexander Richard, Colin Lea, Shugao Ma, Jurgen Gall, Fernando de la Torre, and Yaser Sheikh. 2021. Audio- and Gaze-Driven Facial Animation of Codec Avatars. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 41–50. https://doi.org/10.1109/WACV48630.2021.00009

[29]

Daniel Roth, Gary Bente, Peter Kullmann, David Mal, Chris Felix Purps, Kai Vogeley, and Marc Erich Latoschik. 2019. Technologies for Social Augmentations in User-Embodied Virtual Reality. In 25th ACM Symposium on Virtual Reality Software and Technology. ACM, Parramatta NSW Australia, 1–12. https://doi.org/10.1145/3359996.3364269

Digital Library

[30]

Daniel Roth, Carola Bloch, Anne-Kathrin Wilbers, Marc Erich Latoschik, Kai Kaspar, and Gary Bente. 2016. What You See Is What You Get: Channel Dominance in the Decoding of Affective Nonverbal Behavior Displayed by Avatars. In 66th Annual Conference of the International Communication Association. Fukuoka, Japan.

[31]

Daniel Roth, Peter Kullmann, Gary Bente, Dominik Gall, and Marc Erich Latoschik. 2018. Effects of Hybrid and Synthetic Social Gaze in Avatar-Mediated Interactions. In 2018 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, Munich, Germany, 103–108. https://doi.org/10.1109/ISMAR-Adjunct.2018.00044

[32]

Karen Ruhleder and Brigitte Jordan. 2001. Co-constructing non-mutual realities: Delay-generated trouble in distributed interaction. Computer Supported Cooperative Work (CSCW) 10, 1 (2001), 113–138. https://doi.org/10.1023/A:1011243905593

Digital Library

[33]

Lucas M. Seuren, Joseph Wherton, Trisha Greenhalgh, and Sara E. Shaw. 2021. Whose Turn Is It Anyway? Latency and the Organization of Turn-Taking in Video-Mediated Interaction. Journal of Pragmatics 172 (Jan. 2021), 63–78. https://doi.org/10.1016/j.pragma.2020.11.005

[34]

Richard Skarbez, Solene Neyret, Frederick P. Brooks, Mel Slater, and Mary C. Whitton. 2017. A Psychophysical Experiment Regarding Components of the Plausibility Illusion. IEEE Transactions on Visualization and Computer Graphics 23, 4 (April 2017), 1369–1378. https://doi.org/10.1109/TVCG.2017.2657158

Digital Library

[35]

Mel Slater, Domna Banakou, Alejandro Beacco, Jaime Gallego, Francisco Macia-Varela, and Ramon Oliva. 2022. A Separate Reality: An Update on Place Illusion and Plausibility in Virtual Reality. Frontiers in Virtual Reality 3 (June 2022), 914392. https://doi.org/10.3389/frvir.2022.914392

[36]

Jan-Philipp Stauffert, Florian Niebling, and Marc Erich Latoschik. 2020. Simultaneous Run-Time Measurement of Motion-to-Photon Latency and Latency Jitter. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, Atlanta, GA, USA, 636–644. https://doi.org/10.1109/VR46266.2020.00086

[37]

R. Steinmetz. Jan./1996. Human Perception of Jitter and Media Synchronization. IEEE Journal on Selected Areas in Communications 14, 1 (Jan./1996), 61–72. https://doi.org/10.1109/49.481694

Digital Library

[38]

Monica Villanueva Aylagas, Hector Anadon Leon, Mattias Teye, and Konrad Tollmar. 2022. Voice2Face: Audio-driven Facial and Tongue Rig Animations with cVAEs. Computer Graphics Forum (2022). https://doi.org/10.1111/cgf.14640

[39]

Matias Volonte, Eyal Ofek, Ken Jakubzak, Shawn Bruner, and Mar Gonzalez-Franco. 2022. HeadBox: A Facial Blendshape Animation Toolkit for the Microsoft Rocketbox Library. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, Christchurch, New Zealand, 39–42. https://doi.org/10.1109/VRW55335.2022.00015

[40]

Petra Wagner, Zofia Malisz, and Stefan Kopp. 2014. Gesture and Speech in Interaction: An Overview. Speech Communication 57 (Feb. 2014), 209–232. https://doi.org/10.1016/j.specom.2013.09.008

Digital Library

[41]

Audrey C Younkin and Philip J Corriveau. 2008. Determining the amount of audio-video synchronization errors perceptible to the average end-user. IEEE Transactions on Broadcasting 54, 3 (2008), 623–627. https://doi.org/10.1109/TBC.2008.2002102

Cited By

Mal DDöllinger NWolf EWenninger SBotsch MWienrich CLatoschik M(2024)Am I the odd one? Exploring (in)congruencies in the realism of avatars and virtual others in virtual realityFrontiers in Virtual Reality10.3389/frvir.2024.14170665Online publication date: 19-Jul-2024
https://doi.org/10.3389/frvir.2024.1417066
Etienne ERistorcelli MSaufnay SQuilez ACasanova RSchyns MOchs M(2024)A Systematic Review on the Socio-affective Perception of IVAs' Multi-modal behaviourProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673943(1-10)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673943
Mal DWolf EDöllinger NBotsch MWienrich CLatoschik M(2024)From 2D-Screens to VR: Exploring the Effect of Immersion on the Plausibility of Virtual HumansExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650773(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650773
Show More Cited By

Index Terms

An Evaluation of Other-Avatar Facial Animation Methods for Social VR
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Empirical studies in HCI
    2. Interaction paradigms
      1. Virtual reality

Recommendations

Dynamic Facial Expression Analysis and Synthesis With MPEG-4 Facial Animation Parameters

This paper describes a probabilistic framework for faithful reproduction of dynamic facial expressions on a synthetic face model with MPEG-4 facial animation parameters (FAPs) while achieving very low bitrate in data transmission. The framework consists ...
Semantic 3D motion retargeting for facial animation
APGV '06: Proceedings of the 3rd symposium on Applied perception in graphics and visualization

We present a system for realistic facial animation that decomposes facial motion capture data into semantically meaningful motion channels based on the Facial Action Coding System. A captured performance is retargeted onto a morphable 3D face model ...
High fidelity facial animation capture and retargeting with contours
SCA '13: Proceedings of the 12th ACM SIGGRAPH/Eurographics Symposium on Computer Animation

Human beings are naturally sensitive to subtle cues in facial expressions, especially in areas of the eyes and mouth. Current facial motion capture methods fail to accurately reproduce motions in those areas due to multiple limitations. In this paper, ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems

April 2023

3914 pages

ISBN:9781450394222

DOI:10.1145/3544549

Editors:
Albrecht Schmidt
LMU Munich, Germany
,
Kaisa Väänänen
Tampere University, Finland
,
Tesh Goyal
Google Research, USA
,
Per Ola Kristensson
University of Cambridge, UK
,
Anicia Peters
University of Namibia, Namibia

Copyright © 2023 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Check for updates

Author Tags

Qualifiers

Work in progress
Research
Refereed limited

Funding Sources

Bavarian State Ministry For Digital Affairs
German Federal Ministry of Education and Research

Conference

CHI '23

Sponsor:

SIGCHI

CHI '23: CHI Conference on Human Factors in Computing Systems

April 23 - 28, 2023

Hamburg, Germany

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
412
Total Downloads

Downloads (Last 12 months)193
Downloads (Last 6 weeks)34

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Mal DDöllinger NWolf EWenninger SBotsch MWienrich CLatoschik M(2024)Am I the odd one? Exploring (in)congruencies in the realism of avatars and virtual others in virtual realityFrontiers in Virtual Reality10.3389/frvir.2024.14170665Online publication date: 19-Jul-2024
https://doi.org/10.3389/frvir.2024.1417066
Etienne ERistorcelli MSaufnay SQuilez ACasanova RSchyns MOchs M(2024)A Systematic Review on the Socio-affective Perception of IVAs' Multi-modal behaviourProceedings of the 24th ACM International Conference on Intelligent Virtual Agents10.1145/3652988.3673943(1-10)Online publication date: 16-Sep-2024
https://dl.acm.org/doi/10.1145/3652988.3673943
Mal DWolf EDöllinger NBotsch MWienrich CLatoschik M(2024)From 2D-Screens to VR: Exploring the Effect of Immersion on the Plausibility of Virtual HumansExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3650773(1-8)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3650773
Combe TFribourg RDetto LNorm J(2024)Exploring the Influence of Virtual Avatar Heads in Mixed Reality on Social Presence, Performance and User Experience in Collaborative TasksIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2024.337205130:5(2206-2216)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1109/TVCG.2024.3372051
Macaluso RVisconti ACalandra DCiardo RBarresi GLamberti F(2024)Evaluating Therapist Representation Techniques in Mixed Reality-based Tele-rehabilitation Exergames2024 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)10.1109/ISMAR-Adjunct64951.2024.00066(288-294)Online publication date: 21-Oct-2024
https://doi.org/10.1109/ISMAR-Adjunct64951.2024.00066
Wei SFreeman DHarris VRovira A(2024)A randomised controlled test in virtual reality of the effects on paranoid thoughts of virtual humans’ facial animation and expressionScientific Reports10.1038/s41598-024-67534-414:1Online publication date: 24-Jul-2024
https://doi.org/10.1038/s41598-024-67534-4

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View full text|Download PDF

View Table of Conten