Skip to main content
SpringerLink
Log in

Multimodal Collaboration in Expository Discourse: Verbal and Nonverbal Moves Alignment

  • Conference paper
  • First Online:
Speech and Computer (SPECOM 2023)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 14338))

Included in the following conference series:

  • 423 Accesses

Abstract

The paper explores multimodal collaboration in expository discourse considering verbal and nonverbal moves used by the participants in turn-taking. It reports the results of an experiment which tested speech, gesture and gaze alignment as affected by intentional and spontaneous communication. The results based on the 46-min long pilot subcorpus (out of 640-min long collected database) reveal that collaborative expository discourse is vastly multimodal where nonverbal moves in contact-establishing gesture and face-oriented gaze modulate each type of verbal move. Despite the fact that verbal moves do not manifest specific alignment with nonverbal moves, there are several tendencies in their distribution, for instance, verbal request move and elaboration moves are more frequently accompanied by nonverbal moves than other verbal moves. The study also shows that verbal and nonverbal moves alignment is not modulated by either intentional or spontaneous collaboration since there are no distinctions observed in the nonverbal move use with either common or new topic elaboration move, which proves that more intricate patterns of moves are exploited to provide intentionality and spontaneity in expository discourse.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 79.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 99.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Below, we present the examples of their use omitting the punctuation marks in verbal moves, leaving them only when they are necessary to understand the contents of the moves.

  2. 2.

    In this meaning the word fight is not used in the English language.

References

  1. Ono, T., Thompson, S.A.: Interaction and syntax in the structure of conversational discourse: collaboration, overlap, and syntactic dissociation. In: Hovy, E.H., Scott, D.R. (eds.) Computational and Conversational Discourse, pp. 67–96. Springer Berlin Heidelberg, Berlin, Heidelberg (1996). https://doi.org/10.1007/978-3-662-03293-0_3

    Chapter  Google Scholar 

  2. Clancy, B., McCarthy, M.: Co-constructed turn-taking. In: Aijmer, K., Rühlemann, C. (eds.) Corpus pragmatics: A handbook, pp. 430–453 (2014)

    Google Scholar 

  3. Kibrik, A.A., Korotaev, N.A., Podlesskaya, V.I.: Chapter 1. Russian spoken discourse: Local structure and prosody. In: Izre’el, S., Mello, H., Panunzi, A., Raso, T. (eds.) In Search of Basic Units of Spoken Language: A Corpus-Driven Approach, pp. 35–76. John Benjamins Publishing Company, Amsterdam (2020). https://doi.org/10.1075/scl.94.01kib

    Chapter  Google Scholar 

  4. Lerner, G.H.: Collaborative turn sequences. In: Lerner, G.H. (ed.) Conversation analysis: Studies from the first generation, pp. 225–256. John Benjamins, Philadelphia (2004)

    Chapter  Google Scholar 

  5. Sacks, H.: Lectures on Conversation. Basil Blackwell, Oxford (1992)

    Google Scholar 

  6. Grenoble, L.: Syntax and co-constructed turns in Russian dialogue. Issues Linguist. 1, 25–36 (2008). (in Russian)

    Google Scholar 

  7. Richardson, D., Dale, R., Tomlinson, J.: Conversation, gaze coordination, and beliefs about visual context. Cogn. Sci. 33, 1468–1482 (2009)

    Article  Google Scholar 

  8. Brône, G., Feyaerts, K. & Oben, B.: Multimodal turn-taking in dialogue: on the interplay of eye gaze, speech and gesture. Proceedings of AFLiCo5: Empirical approaches to multi-modality and to language variation, 21–22 (2013)

    Google Scholar 

  9. Bangerter, A.: Using pointing and describing to achieve joint focus of attention in dialogue. Psychol. Sci. 15, 415–419 (2004)

    Article  Google Scholar 

  10. Mol, L., Krahmer, E., Maes, A., Swerts, M.: Adaptation in gesture: converging hands or converging minds? J. Mem. Lang. 66, 249–264 (2012)

    Article  Google Scholar 

  11. Maman, L., et al.: GAME-ON: a multimodal dataset for cohesion and group analysis. IEEE Access, 8, 124185–124203 (2020)

    Google Scholar 

  12. Korotaev, N.A. Collaborative constructions in Russian conversations: a multichannel perspective. Computational Linguistics and Intellectual Technologies. Papers from the Annual International Conference “Dialogue” (2023) 22, 250–258 (2023)

    Google Scholar 

  13. Mondada, L.: L’organisation séquentielle des ressources linguistiques dans l’élaboration collective des descriptions. Langage et Société 89(1), 9–36 (1999)

    Article  Google Scholar 

  14. Bogdanova-Beglarian, N., Blinova, O., Sherstinova, T., Gorbunova, D., Zaides, K., Popova, T.: Pragmatic Markers in Dialogue and Monologue: Difficulties of Identification and Typical Formation Models. In: Karpov, A., Potapova, R. (eds.) SPECOM 2020. LNCS (LNAI), vol. 12335, pp. 68–78. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60276-5_7

    Chapter  Google Scholar 

  15. Longacre, R.: The grammar of discourse. Plenum, New York (1983)

    Google Scholar 

  16. Hughes, R.: Teaching and Researching: Speaking, 2nd edn. Routledge, London (2010)

    Google Scholar 

  17. Thornbury, S., Slade, D.: Conversation: From Description to Pedagogy. Cambridge University Press, Cambridge (2006)

    Book  Google Scholar 

  18. Levinson, S.C.: Turn-taking in human communication – origins and implications for language processing. Trends Cogn. Sci. 20(1), 6–14 (2016)

    Article  Google Scholar 

  19. Bohus, D., Horvitz, E.: Facilitating multiparty dialog with gaze, gesture, and speech. In: Proceedings of the International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction (ICMI-MLMI), Beijing, China, 8–12 November 2010, 1–8. New York, NY: ACM (2010)

    Google Scholar 

  20. Hessels, R.S., Benjamins, J.S., van Doorn, A.J., Koenderink, J.J., Hooge, I.T.C.: Perception of the potential for interaction in social scenes. i-Perception 12(5), 204166952110402 (2021). https://doi.org/10.1177/20416695211040237

    Article  Google Scholar 

  21. Goodwin, C.: Action and embodiment within situated human interaction. J. Pragmat. 32(10), 1489–1522 (2000)

    Article  Google Scholar 

  22. Kendon, A.: Gestures as illocutionary and discourse structure markers in Southern Italian conversation. J. Pragmat. 23(3), 247–279 (1995)

    Article  Google Scholar 

  23. Ford, C., Fox, B.A., Thompson, S.A.: Practices in the construction of turns: The’TCU’ revisited. Pragmatics 6(3), 427–454 (1996)

    Google Scholar 

  24. Curioni, A, Knoblich, G.K., Sebanz, N., Sacheli, L.M. The engaging nature of interactive gestures. PLoS ONE 15(4), e0232128 (2020)

    Google Scholar 

  25. Holler, J.: Speakers’ use of interactive gestures as markers of common ground. In: S. Kopp and I. Wachsmuth (eds.) GW 2009, Lecture Notes in Computer Science 5934, pp. 11–22. (2010)

    Google Scholar 

  26. Degutyte, Z., Astell, A.: The role of eye gaze in regulating turn taking in conversations: a systematized review of methods and findings. Front. Psychol. 12, 616471 (2021)

    Article  Google Scholar 

  27. Kendon, A.: Some functions of gaze-direction in social interaction. Acta Physiol (Oxf.) 26, 22–63 (1967)

    Google Scholar 

  28. Cummins, F.: Gaze and blinking in dyadic conversation: a study in coordinated behaviour among individuals. Lang. Cogn. Process. 27, 1525–1549 (2012)

    Article  Google Scholar 

  29. Ho, S., Foulsham, T., Kingstone, A.: Speaking and listening with the eyes: gaze signaling during dyadic interactions. PLoS ONE 10, e0136905 (2015)

    Article  Google Scholar 

  30. Rutter, D.R., Stephenson, G.M., Ayling, K., White, P.A.: The timing of looks in dyadic conversation. British J. Social Clin. Psychol. 17, 17–21 (1978)

    Article  Google Scholar 

  31. Brône, G., Oben, B., Jehoul, A., Vranjes, J., Feyaerts, K.: Eye gaze and viewpoint in multimodal interaction management. Cogn. Linguist. 28(3), 449–483 (2017)

    Article  Google Scholar 

  32. Kendrick, K.H., Holler, J.: Gaze direction signals response preference in conversation. Res. Lang. Soc. Interact. 50, 12–32 (2017)

    Article  Google Scholar 

  33. Argyle, M., Cook, M.: Gaze and Mutual Gaze. Cambridge University Press, London (1976)

    Google Scholar 

Download references

Acknowledgements

The research presented in Sections 1 and 2 is part of the project “Multimodal research of the speaker’s communicative behavior in different discourse types” (075–03-2020–013) carried out at Moscow State Linguistic University. The research presented in Sections 3, 4, and 5 is part of the project “Kinesic and vocal aspects of communication: parameters of variance” (FMNE-2022–0015) carried out at the Institute of Linguistics RAS.

Author information

Authors and Affiliations

  1. Institute of Linguistics of Russian Academy of Sciences, Moscow State Linguistic University, Ostozhenka 38, 19013, Moscow, Russia

    Olga Iriskhanova, Maria Kiose, Anna Leonteva, Olga Agafonova & Andrey Petrov

Authors
  1. Olga Iriskhanova
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Maria Kiose
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Anna Leonteva
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Olga Agafonova
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andrey Petrov
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Maria Kiose .

Editor information

Editors and Affiliations

  1. St. Petersburg Federal Research Center of the Russian Academy of Sciences, St. Petersburg, Russia

    Alexey Karpov

  2. Koneru Lakshmaiah Education Foundation, Vaddeswaram, India

    K. Samudravijaya

  3. Indian Institute of Information Technology Dharwad, Dharwad, India

    K. T. Deepak

  4. Indian Institute of Technology Dharwad, Dharwad, India

    Rajesh M. Hegde

  5. KIIT Group of Colleges, Gurugram, India

    Shyam S. Agrawal

  6. Indian Institute of Technology Dharwad, Dharwad, India

    S. R. Mahadeva Prasanna

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Iriskhanova, O., Kiose, M., Leonteva, A., Agafonova, O., Petrov, A. (2023). Multimodal Collaboration in Expository Discourse: Verbal and Nonverbal Moves Alignment. In: Karpov, A., Samudravijaya, K., Deepak, K.T., Hegde, R.M., Agrawal, S.S., Prasanna, S.R.M. (eds) Speech and Computer. SPECOM 2023. Lecture Notes in Computer Science(), vol 14338. Springer, Cham. https://doi.org/10.1007/978-3-031-48309-7_29

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-48309-7_29

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-48308-0

  • Online ISBN: 978-3-031-48309-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation