skip to main content
research-article

Taming Mona Lisa: Communicating gaze faithfully in 2D and 3D facial projections

Published: 13 January 2012 Publication History

Abstract

The perception of gaze plays a crucial role in human-human interaction. Gaze has been shown to matter for a number of aspects of communication and dialogue, especially for managing the flow of the dialogue and participant attention, for deictic referencing, and for the communication of attitude. When developing embodied conversational agents (ECAs) and talking heads, modeling and delivering accurate gaze targets is crucial. Traditionally, systems communicating through talking heads have been displayed to the human conversant using 2D displays, such as flat monitors. This approach introduces severe limitations for an accurate communication of gaze since 2D displays are associated with several powerful effects and illusions, most importantly the Mona Lisa gaze effect, where the gaze of the projected head appears to follow the observer regardless of viewing angle. We describe the Mona Lisa gaze effect and its consequences in the interaction loop, and propose a new approach for displaying talking heads using a 3D projection surface (a physical model of a human head) as an alternative to the traditional flat surface projection. We investigate and compare the accuracy of the perception of gaze direction and the Mona Lisa gaze effect in 2D and 3D projection surfaces in a five subject gaze perception experiment. The experiment confirms that a 3D projection surface completely eliminates the Mona Lisa gaze effect and delivers very accurate gaze direction that is independent of the observer's viewing angle. Based on the data collected in this experiment, we rephrase the formulation of the Mona Lisa gaze effect. The data, when reinterpreted, confirms the predictions of the new model for both 2D and 3D projection surfaces. Finally, we discuss the requirements on different spatially interactive systems in terms of gaze direction, and propose new applications and experiments for interaction in a human-ECA and a human-robot settings made possible by this technology.

References

[1]
Abele, A. 1986. Function of gaze in social interaction: Communication and monitoring. J. Nonverb. Behav. 10, 83--101.
[2]
Argyle, M. and Cook, M. 1976. Gaze and Mutual Gaze. Cambridge University Press.
[3]
Beskow, J. 2003. Talking heads: Models and applications for multimodal speech synthesis. Doctoral dissertation, KTH Royal Institute of Technology, Stockholm, Sweden.
[4]
Bente, G., Donaghy, W. C., and Suwelack, D. 1998. Sex differences in body movement and visual attention: An integrated analysis of movement and gaze in mixed-sex dyads. J. Nonverb. Behav. 22, 31--58.
[5]
Bilvi, M. and Pelachaud, C. 2003. Communicative and statistical eye gaze predictions. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS).
[6]
Bloom, K. and Erickson, M. 1971. The role of eye contact in social reinforcement of infant vocalizations. In Proceedings of the Biannual Meeting of the Society for Research in Child Development.
[7]
Bohus, D. and Horvitz, E. 2010. Facilitating multiparty dialog with gaze, gesture, and speech. In Proceedings of the 12th International Conference on Multimodal Interfaces and the 7th Workshop on Machine Learning for Multimodal Interaction.
[8]
Cuijpers, R. H., van der Pol, D., and Meesters, L. M. J. 2010. Mediated eye-contact is determined by relative pupil position within the sclera. In Proceedings of the European Conference on Visual Perception (ECVP'10). 129.
[9]
Delaunay, F., De Greeff, J., and Belpaeme, T. 2010. A study of a retro-projected robotic face and its effectiveness for gaze reading by humans. In Proceedings of the 5th ACM/IEEE International Conference on Human-Robot Interaction. 39--44.
[10]
Descartes, R. 1637. Dioptrics. In Discourse on Method, Optics, Geometry, and Meteorology, Hackett Publishing, Indianapolis, IN. 65--162.
[11]
Edlund, J. and Beskow, 1. 2009. MushyPeek: A framework for online investigation of audiovisual dialogue phenomena. Lang. Speech, 52, 2--3, 351--367.
[12]
Edlund, J. and Nordstrand, M. 2002. Turn-taking gestures and hour-glasses in a multi-modal dialogue system. In Proceedings of the ISCA Workshop on Multi-Modal Dialogue in Mobile Environments.
[13]
Frieschen, A., Bayliss, A., and Tipper, S. 2007. Gaze cueing of attention: Visual attention, social cognition and individual differences. Psych. Bull. 133, 694--724.
[14]
Gemmell, J., Zitnick, C., Kang, T., Toyama, K., and Seitz, S. 2000. Gaze-awareness for videoconferencing: A software approach. IEEE Multimedia 7, 4, 26--35.
[15]
Gregory, R. 1997. Eye and Brain: The Psychology of Seeing. Princeton University Press.
[16]
Gu, E. and Badler, N. 2006. Visual attention and eye gaze during multiparty conversations with distractions. In Proceedings of the International Conference on Intelligent Virtual Agents.
[17]
Kendon, A. 1967. Some functions of gaze direction in social interaction. Acta Psychologica 26, 22--63.
[18]
Kuenke, C. L. 1986. Gaze and eye contact: a research review. Psych. Bull. 100, 78--100.
[19]
Lance, B. and Marsella, S. 2008. A model of gaze for the purpose of emotional expression in virtual embodied agents. In Proceedings of the 7th International Conference on Autonomous Agents and Multiagent Systems. 199--206.
[20]
Lance, B. and Marsella, S. 2010. Glances, glares, and glowering: How should a virtual human express emotion through gaze? J. Auton. Agents Multiagent Syst. 20, 50--69.
[21]
Lincoln, P., Welch, G., Nashel, A., Ilie, A., State, A., and Fuchs, H. 2009. Animatronic shader lamps avatars. In Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality.
[22]
Nordenberg, M., Svanfeldt, G., and Wik, P. 2005. Artificial gaze: Perception experiment of eye gaze in synthetic faces. In Proceedings of the 2nd Nordic Conference on Multimodal Communication.
[23]
Pelachaud, C. and Bilvi, M., 2003. Modelling gaze behavior for conversational agents. In Proceedings of the International Conference on Intelligent Virtual Agents. 15--17.
[24]
Poggi, I. and Pelachaud, C. 2000. Emotional meaning and expression in performative faces. In Affective Interactions: Towards a New Generation of Computer Interfaces, Paiva, A., Ed., Lecture Notes in Computer Science vol. 1814, 182--195.
[25]
Raskar, R., Welch, G., Low, K.-L., and Bandyopadhyay, D. 2001. Shader lamps: Animating realobjects with image-based illumination. In Proceedings of the 12th Eurographics Workshop on Rendering Techniques. 89--102.
[26]
Smith, A. M. 1996. Ptolemy's Theory of Visual Perception: An English translation of the Optics with Introduction and Commentary. American Philosophical Society, Philadelphia, PA.
[27]
Takeuchi, A. and Nagao, K. 1993. Communicative facial displays as a new conversational modality. In Proceedings of the Conference on Human Factors in Computing Systems.
[28]
Todorović, D. 2006. Geometrical basis of perception of gaze direction. Vis. Resear. 45, 21, 3549--3S62.
[29]
Vertegaal, R. 1999. The GAZE groupware system: Mediating joint attention in multiparty communication and collaboration. In Proceedings of the ACM CHI Human Factors in Computing Systems Conference, Addison-Wesley/ACM Press, 294--301.
[30]
Waxer, P. 1977. Nonverbal cues for anxiety: An examination of emotional leakage. J. Abnorm. Psych. 86, 306--314.
[31]
Wollaston, W. H. 1824. On the apparent direction of eyes in a portrait. Phil Trans. Royal Soc. London, B114, 247--260.

Cited By

View all
  • (2024)The Effect of Eye Contact in Multi-Party Conversations with Virtual Humans and Mitigating the Mona Lisa EffectElectronics10.3390/electronics1302043013:2(430)Online publication date: 19-Jan-2024
  • (2022)Looking Into Mona Lisa’s Smiling Eyes: Allusion to an IllusionFrontiers in Human Neuroscience10.3389/fnhum.2022.87828816Online publication date: 1-Jul-2022
  • (2021)Accuracy of Interpersonal Distance and Line of Sight Between a Virtual Robot Head and Humansバーチャルロボットヘッドに対する人間の対人距離と視線推定に関する研究Journal of Japan Society for Fuzzy Theory and Intelligent Informatics10.3156/jsoft.33.4_75733:4(757-767)Online publication date: 15-Nov-2021
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Interactive Intelligent Systems
ACM Transactions on Interactive Intelligent Systems  Volume 1, Issue 2
January 2012
157 pages
ISSN:2160-6455
EISSN:2160-6463
DOI:10.1145/2070719
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 January 2012
Accepted: 01 October 2011
Revised: 01 August 2011
Received: 01 January 2011
Published in TIIS Volume 1, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. 3D projected avatars
  2. Gaze perception
  3. Mona Lisa gaze effect
  4. embodied conversational agents
  5. multiparty dialogue
  6. robot head
  7. situated interaction

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)37
  • Downloads (Last 6 weeks)7
Reflects downloads up to 16 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2024)The Effect of Eye Contact in Multi-Party Conversations with Virtual Humans and Mitigating the Mona Lisa EffectElectronics10.3390/electronics1302043013:2(430)Online publication date: 19-Jan-2024
  • (2022)Looking Into Mona Lisa’s Smiling Eyes: Allusion to an IllusionFrontiers in Human Neuroscience10.3389/fnhum.2022.87828816Online publication date: 1-Jul-2022
  • (2021)Accuracy of Interpersonal Distance and Line of Sight Between a Virtual Robot Head and Humansバーチャルロボットヘッドに対する人間の対人距離と視線推定に関する研究Journal of Japan Society for Fuzzy Theory and Intelligent Informatics10.3156/jsoft.33.4_75733:4(757-767)Online publication date: 15-Nov-2021
  • (2021)VR social copresence with light field displaysACM Transactions on Graphics10.1145/3478513.348048140:6(1-13)Online publication date: 10-Dec-2021
  • (2021)Improving VIP viewer gaze estimation and engagement using adaptive dynamic anamorphosisInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2020.102563147(102563)Online publication date: Mar-2021
  • (2020)Analysis of Behavioral Patterns for Social Virtual Reality Based Active Learning2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)10.1109/TALE48869.2020.9368359(914-917)Online publication date: 8-Dec-2020
  • (2020)Gaze by Semi-Virtual Robotic Heads: Effects of Eye and Head Motion2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS45743.2020.9341209(11065-11071)Online publication date: 24-Oct-2020
  • (2020)Turn-taking in Conversational Systems and Human-Robot Interaction: A ReviewComputer Speech & Language10.1016/j.csl.2020.101178(101178)Online publication date: Dec-2020
  • (2019)The Mona Lisa Illusion—Scientists See Her Looking at Them Though She Isn’ti-Perception10.1177/204166951882170210:1Online publication date: 7-Jan-2019
  • (2019)Multimodal conversational interaction with robotsThe Handbook of Multimodal-Multisensor Interfaces10.1145/3233795.3233799(77-104)Online publication date: 1-Jul-2019
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media