Abstract
Gaze is reported to have important functions in communication, such as expressing emotional states, exercising social control, highlighting the informational structure of speech, and coordinating floor-apportionment. For these reasons, studying these communicative functions is expected to contribute to HCI systems by identifying communication characteristics and the role of each participant. This study analyzes how the communicative functions of utterances affect the listener’s gazing activities from the viewpoint of grounding, based on a triadic corpus with newly labeled grounding tags. The results showed that the duration of a listener’s gaze is longer in second language (L2) conversations, in goal-oriented conversations, and during utterances presenting new information. These results suggest that linguistic proficiency, conversation topic, and grounding factors all affect a listener’s gazing activities, providing us with some information that could assist in the design of HCI, HRI, and CSCW systems that better reflect the interaction contexts and linguistic proficiency of users.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Mehrabian, A., Ferris, S.R.: Inference of attitudes from nonverbal communication in two channels. J. Consul. Psychol. 31(3), 248–252 (1967)
Mehrabian, A., Wiener, M.: Decoding of inconsistent communications. J. Pers. Soc. Psychol. 6(1), 109–114 (1967)
Clark, H.H., Brennan, S.E.: Grounding in communication. In: Resnik, L.B., Levine, J.M., Teasley, S.D. (eds.) Perspectives on Socially Shared Cognition, pp. 503–512. APA Books (1991)
Clark, H.H.: Using Language. Cambridge University Press, Cambridge (1996)
Clark, H.H., Krych, M.A.: Speaking while monitoring addressees for understanding. J. Mem. Lang. 50, 62–81 (2004)
Argyle, M., Lallijee, M., Cook, M.: The effects of visibility on interaction in a dyad. Hum. Relat. 21, 3–17 (1968)
Duncan, S.: Some signals and rules for taking speaking turns in conversations. J. Pers. Soc. Psychol. 23, 283–292 (1972)
Holler, J., Kendrick, K.H.: Unaddressed participants’ gaze in multi-person interaction: optimizing recipiency. Front. Psychol. 6, 14, Article no. 98 (2015). https://doi.org/10.3389/fpsyg.2015.00098
Kendon, A.: Some functions of gaze-direction in social interaction. Acta Physiol. 26, 22–63 (1967)
Kalma, A.: Gazing in triads: a powerful signal in floor apportionment. Br. J. Soc. Psychol. 31(1), 21–39 (1992)
Lerner, G.H.: Selecting next speaker: the context-sensitive operation of a context-free organization. Lang. Soc. 32(02), 177–201 (2003)
Beattie, G.W.: Floor apportionment and gaze in conversational dyads. Br. J. Soc. Clin. Psychol. 17, 7–15 (1978)
Rutter, D.R., Stephenson, G.M., Ayling, K., White, P.A.: The timing of looks in dyadic conversation. Br. J. Soc. Clin. Psychol. 17, 17–21 (1978)
Kleinke, C.L.: Gaze and eye contact: a research review. Psychol. Bull. 100, 78–100 (1986)
Ho, S., Foulsham, T., Kingstone, A.: Speaking and listening with the eyes: gaze signaling during dyadic interactions. PLoS ONE 10(8), e0136905 (2015). https://doi.org/10.1371/journal.pone.0136905
Jokinen, K., Furukawa, H., Nishida, M., Yamamoto, S.: Gaze and turn-taking behavior in casual conversational interactions. ACM Trans. Interact. Intell. Syst. (TiiS) 3(2), 1–30 (2013)
Ishii, R., Otsuka, K., Kumano, S., Yamato, J.: Prediction of who will be the next speaker and when using gaze behavior in multiparty meetings. ACM Trans. Interact. Intell. Syst. 6(1), 31, Article no. 4 (2016). https://doi.org/10.1145/2757284
Vertegaal, R., Slagter, R., Verr, G., Nijholt, A.: Eye gaze patterns in conversations: there is more to conversational agents than meets the eyes. In: CHI 2001 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 301–308. ACM Press, Seattle (2001)
Ijuin, K., Umata, I., Kato, T., Yamamoto, S.: Difference in eye gaze for floor apportionment in native- and second-language conversations. J. Nonverbal Behav. 42, 113–128 (2018)
Bavelas, J.B., Coates, L., Johnson, T.: Listener responses as a collaborative process: the role of gaze. J. Commun. 52(3), 566–580 (2002). https://doi.org/10.1111/j.1460-2466.2002.tb02562.x
Hirvenkari, L., Ruusuvuori, J., Saarinen, V.M., Kivioja, M., Peräkylä, A., Hari, R.: Influence of turn-taking in a two-person conversation on the gaze of a viewer. PloS One 8(8), e71569 (2013). https://doi.org/10.1371/journal.pone.0071569
Cassell, J., et al.: Animated conversation: rule-based generation of facial expression, gesture & spoken intonation for multiple conversational agents. In: Proceedings of the 21st Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 1994), pp. 413–420. ACM, New York (1994). https://doi.org/10.1145/192161.192272
Garau, M., Slater, M., Bee, S., Sasse, M.A.: The impact of eye gaze on communication using humanoid avatars. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2001), pp. 309–316. ACM, New York (2001). https://doi.org/10.1145/365024.365121
Heylen, D., van Es, I., Nijholt, A., van Dijk, B.: Controlling the gaze of conversational agents. In: van Kuppevelt, J.C.J., Dybkjær, L., Bernsen, N.O. (eds.) Advances in Natural Multimodal Dialogue Systems. TLTB, pp. 245–262. Springer, Netherlands (2005). https://doi.org/10.1007/1-4020-3933-6_11
Rehm, M., André, E.: Where do they look? Gaze behaviors of multiple users interacting with an embodied conversational agent. In: Panayiotopoulos, T., Gratch, J., Aylett, R., Ballin, D., Olivier, P., Rist, T. (eds.) IVA 2005. LNCS (LNAI), vol. 3661, pp. 241–252. Springer, Heidelberg (2005). https://doi.org/10.1007/11550617_21
Sidner, C.L., Kidd, C.D., Lee, C., Lesh, N.: Where to look: a study of human-robot engagement. In: IUI 2004: Proceedings of the 9th International Conference on Intelligent User Interfaces, pp. 78–84 (2004). https://doi.org/10.1145/964456.964458
Bennewitz, M., Faber, F., Joho, D., Schreiber, M., Behnke, S.: Integrating vision and speech for conversations with multiple persons. In: 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2523–2528 (2005). https://doi.org/10.1109/IROS.2005.1545158
Kuno, Y., Sadazuka, K., Michie Kawashima, M., Yamazaki, K., Yamazaki, A., Kuzuoka, H.: Museum guide robot based on sociological interaction analysis. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2007), pp. 1191–1194. ACM, New York (2007). https://doi.org/10.1145/1240624.1240804
Foster, M.E., Gaschler, A., Giuliani, M., Isard, A., Pateraki, M., Petrick, R.P.A.: Two people walk into a bar: dynamic multi-party social interaction with a robot agent. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI 2012), pp. 3–10. ACM, New York (2012). https://doi.org/10.1145/2388676.2388680
Lala, D., Inoue, K., Kawahara, T.: Smooth turn-taking by a robot using an online continuous model to generate turn-taking cues. In: 2019 International Conference on Multimodal Interaction (ICMI 2019), Suzhou, China, 14–18 October 2019, 9 p. ACM, New York (2019). https://doi.org/10.1145/3340555.3353727
Jaber, R., McMillan, D., Belenguer, J.S., Brown, B.: Patterns of gaze in speech agent interaction. In: 1st International Conference on Conversational User Interfaces (CUI 2019), Dublin, Ireland, 22–23 August 2019, 10 p. ACM, New York (2019). https://doi.org/10.1145/3342775.3342791
Veinott, E., Olson, J., Olson, G., Fu, X.: Video helps remote work: speakers who need to negotiate common ground benefit from seeing each other. In: Proceedings of the SIGCHI Conference on Computer Human Interaction (CHI 1999), pp. 302–309. ACM, New York (1999)
Schegloff, E.A., Jefferson, G., Sacks, H.: The preference for self-correction in the organization of repair in conversation. Language 53(1977), 361–382 (1977)
Traum, D.R.: A computational theory of grounding in natural language conversation. Ph.D. dissertation, University of Rochester (1994)
Hosoda, Y.: Repair and relevance of differential language expertise. Appl. Linguist. 27(2006), 25–50 (2006)
Yamamoto, S., Taguchi, K., Umata, I., Kabashima, K., Nishida, M.: Differences in interactional attitudes in native and second language conversations: quantitative analyses of multimodal three-party corpus. In: Proceedings of the 35th Annual Meeting of the Cognitive Science Society, pp. 3823–3828 (2013)
Umata, I., Yamamoto, S., Ijuin, K., Nishida, M.: Effects of language proficiency on eye-gaze in second language conversations: toward supporting second language collaboration. In: Proceedings of the International Conference on Multimodal Interaction (ICMI 2013), pp. 413–419 (2013)
Yamamoto, S., Taguchi, K., Ijuin, K., Umata, I., Nishida, M.: Multimodal corpus of multiparty conversations in L1 and L2 languages and findings obtained from it. Lang. Resour. Eval. 49, 857–882 (2015). https://doi.org/10.1007/s10579-015-9299-2
Rossano, F.: Gaze in conversation. In: Stivers, T., Sidnell, J. (ed.) The Handbook of Conversation Analysis, pp. 308–329. Wiley-Blackwell, Malden (2013). https://doi.org/10.1002/9781118325001.ch15
Kendrick, K.H., Holler, J.: Gaze direction signals response preference in conversation (2017)
Rossano, F., Brown, P., Levinson, S.C.: Gaze, Questioning, and Culture, pp. 187–249. Cambridge University Press, Cambridge (2009). https://doi.org/10.1017/CBO9780511635670.008
Stivers, T., Rossano, F.: Mobilizing response. Res. Lang. Soc. Interact. 43, 3–31 (2010)
Umata, I., Ijuin, K., Kato, T., Yamamoto, S.: Floor apportionment function of speaker’s gaze in grounding acts. In: Proceedings of ICMI 2019: International Conference on Multimodal Interaction (ICMI 2019 Adjunct), Suzhou, China, 14–18 October 2019, 7 p. ACM, New York (2019). https://doi.org/10.1145/3351529.3360660
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Umata, I., Ijuin, K., Kato, T., Yamamoto, S. (2020). Effects of Linguistic Proficiency and Conversation Topic on Listener’s Gaze in Triadic Conversation. In: Meiselwitz, G. (eds) Social Computing and Social Media. Design, Ethics, User Behavior, and Social Network Analysis. HCII 2020. Lecture Notes in Computer Science(), vol 12194. Springer, Cham. https://doi.org/10.1007/978-3-030-49570-1_47
Download citation
DOI: https://doi.org/10.1007/978-3-030-49570-1_47
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-49569-5
Online ISBN: 978-3-030-49570-1
eBook Packages: Computer ScienceComputer Science (R0)