Skip to main content
SpringerLink
Log in

Introduction to the 2nd Edition of “Semantic, Artificial and Computational Interaction Studies”

  • Conference paper
  • First Online:
HCI International 2022 - Late Breaking Papers. Multimodality in Advanced Interaction Environments (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13519))

Included in the following conference series:

  • 770 Accesses

Abstract

“Behavioromics” is a term that has been invented to cover the study of multimodal interaction from various disciplines and points of view. These disciplines and points of view, however, lack a platform for exchange. The workshop session on “Semantic, artificial and computational interaction studies” provides such a platform. We motivate behavioromics, sketch its historical background, and summarize this year’s contributions.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    For an overview of the history of gesture studies and its precursors see the first chapters of [41].

  2. 2.

    An attempt of an overview of the range of communicative gestures is given in [49].

  3. 3.

    In a footnote, Laparle remarks that one can think of interaction spaces in terms of conceptual or mental spaces [22]. However, there is another apparent connection, namely thinking of interaction spaces as parts of dialogue gameboards [25]. This construal would immediately connect interaction spaces to dialogue semantic frameworks and management action could be modeled as effects of conversational moves, as has been shown for some interactive pointing gestures [26].

  4. 4.

    In considering “laughter as language” [27], Mazzocconi can be seen as joining in McNeill’s famous question “So You Think Gestures are Nonverbal?” [52].

References

  1. Alahverdzhieva, K., Flickinger, D., Lascarides, A.: Multimodal grammar implementation. In: Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 582–586. NAACL-HLT 2012 (2012)

    Google Scholar 

  2. Alahverdzhieva, K., Lascarides, A., Flickinger, D.: Aligning speech and co-speech gesture in a constraint-based grammar. J. Lang. Model. 5(3), 421–464 (2017)

    Google Scholar 

  3. Argyle, M.: Bodily Communication, 2nd edn. Routledge, London, New York (1988)

    Google Scholar 

  4. Banarescu, L., et al.: Abstract meaning representation for sembanking. In: Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pp. 178–186 (2013)

    Google Scholar 

  5. Barasch, M.: Giotto and the Language of Gesture. Cambridge University Press, Cambridge (1987)

    Google Scholar 

  6. Baroni, M.: Grounding distributional semantics in the visual world. Lang. Linguist. Compass 10(1), 3–13 (2016)

    Article  Google Scholar 

  7. Bergmann, K., Kopp, S.: Verbal or visual? How information is distributed across speech and gesture in spatial dialog. In: Schlangen, D., Fernández, R. (eds.) Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pp. 90–97. Brandial’06, Universitätsverlag Potsdam, Potsdam (2006)

    Google Scholar 

  8. Birdwhistell, R.L.: Kinesics and Context. Conduct and Communication Series. University of Pennsylvania Press, Philadelphia (1970)

    Google Scholar 

  9. Birdwhistell, R.L.: Background to kinesics. ETC Rev. Gen. Semant. 40(3), 352–361 (1983)

    Google Scholar 

  10. Bolt, R.A.: “Put-that-there’’: voice and gesture at the graphics interface. SIGGRAPH Comput. Graph. 14, 262–270 (1980). https://doi.org/10.1145/965105.807503

    Article  Google Scholar 

  11. Burnett, H.: Signalling games, sociolinguistic variation and the construction of style. Linguist. Philos. 42(5), 419–450 (2019). https://doi.org/10.1007/s10988-018-9254-y

    Article  Google Scholar 

  12. Cafaro, A., et al.: The Noxi database: multimodal recordings of mediated novice-expert interactions. In: Proceedings of the 19th ACM International Conference on Multimodal Interaction (ICMI 201207), pp. 350–359. Association for Computing Machinery, New York (2017). https://doi.org/10.1145/3136755.3136780

  13. Cassell, J.: Embodied conversational interface agents. Communun. ACM 43(4), 70–78 (2000). https://doi.org/10.1145/332051.332075

    Article  Google Scholar 

  14. Clark, H.H.: Using Language. Cambridge University Press, Cambridge (1996)

    Book  Google Scholar 

  15. Clark, H.H.: Pointing and placing. In: Kita, S. (ed.) Pointing: Where Language, Culture, and Cognition Meet, chap. 10, pp. 243–269. Lawrence Erlbaum Associates Inc, Mahwah (2003)

    Google Scholar 

  16. Cooper, R.: Charting a way through the trees. Theor. Linguist. 43(1–2), 121–128 (2017). https://doi.org/10.1515/tl-2017-0009

    Article  Google Scholar 

  17. de Saussure, F.: Course de linguistique générale. In: Bally, C., Sechehaye, A. (eds.) Bibliothèque Scientifique Payot, Payot, Paris (1916)

    Google Scholar 

  18. Deacon, T.: The Symbolic Species. Penguin Press, London (1997)

    Google Scholar 

  19. Ebert, C., Ebert, C., Hörnig, R.: Demonstratives as dimension shifters. Proc. Sinn Bedeutung 24(1), 161–178 (2020). https://doi.org/10.18148/sub/2020.v24i1.859

  20. Efron, D.: Gesture and Environment. King’s Crown Press, Morningside Heights (1941)

    Google Scholar 

  21. Ekman, P., Friesen, W.V.: The repertoire of nonverbal behavior: categories, origins, usage, and coding. Semiotica 1(1), 49–98 (1969)

    Article  Google Scholar 

  22. Fauconnier, G.: Mental Spaces: Aspects of Meaning Construction in Natural Language. Cambridge University Press, Cambridge (1994)

    Book  Google Scholar 

  23. Frege, G.: Der Gedanke. Beiträge zur Philosophie des deutschen Idealismus 1(2), 58–77 (1918)

    Google Scholar 

  24. Fricke, E.: Grammatik multimodal. Wie Wörter und Gesten zusammenwirken. No. 40 in Linguistik - Impulse und Tendenzen, De Gruyter, Berlin and Boston (2012)

    Google Scholar 

  25. Ginzburg, J.: The Interactive Stance: Meaning for Conversation. Oxford University Press, Oxford (2012)

    Book  Google Scholar 

  26. Ginzburg, J., Lücking, A.: I thought pointing is rude: a dialogue-semantic analysis of pointing at the addressee. In: Grosz, P., Martí, L., Pearson, H., Sudo, Y., Zobel, S. (eds.) Proceedings of Sinn und Bedeutung, vol. 25, pp. 276–291. SuB 25 (2021). https://ojs.ub.uni-konstanz.de/sub/index.php/sub/article/view/937. Special Session: Gestures and Natural Language Semantics

  27. Ginzburg, J., Mazzocconi, C., Tian, Y.: Laughter as language. Glossa 5(1), 104 (2020). https://doi.org/10.5334/gjgl.1152

    Article  Google Scholar 

  28. Ginzburg, J., Poesio, M.: Grammar is a system that characterizes talk in interaction. Front. Psychol. 7, 1938 (2016). https://doi.org/10.3389/fpsyg.2016.01938

    Article  Google Scholar 

  29. Giorgolo, G.: A formal semantics for iconic spatial gestures. In: Aloni, M., Bastiaanse, H., de Jager, T., Schulz, K. (eds.) Logic, Language and Meaning. LNCS (LNAI), vol. 6042, pp. 305–314. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-14287-1_31

  30. Giorgolo, G., Asudeh, A.: Multimodal communication in LFG: gestures and the correspondence architecture. In: Proceedings of LFG11, pp. 257–277 (2011)

    Google Scholar 

  31. Goldin-Meadow, S., Brentari, D.: Gesture, sign, and language: the coming of age of sign language and gesture studies. Behav. Brain Sci. 40, e46 (2017). https://doi.org/10.1017/S0140525X15001247

    Article  Google Scholar 

  32. Goodwin, C.: The intelligibility of gesture within a framework of co-operative action. In: Seyfeddinipur, M., Gullberg, M. (eds.) From Gesture in Conversation to Visible Action as Utterance, pp. 199–216. John Benjamins, Amsterdam and Philadelphia (2014)

    Google Scholar 

  33. Gurion, T., Healey, P.G., Hough, J.: Comparing models of speakers’ and listeners’ head nods. In: Proceedings of the 24th Workshop on the Semantics and Pragmatics of Dialogue. SEMDIAL (2020)

    Google Scholar 

  34. Hauser, M.D., Chomsky, N., Fitch, W.T.: The faculty of language: what is it, who has it, and how did it evolve? Science 298(5598), 1569–1579 (2002). https://doi.org/10.1126/science.298.5598.1569, review

  35. Holler, J., Levinson, S.C.: Multimodal language processing in human communication. Trends Cognit. Sci. 23(8), 639–652 (2019). https://doi.org/10.1016/j.tics.2019.05.006, opinion

  36. Johnston, M.: Unification-based multimodal parsing. In: Proceedings of the 36th Annual Meeting on Association for Computational Linguistics - Volume I, Montreal, Quebec, pp. 624–630 (1998)

    Google Scholar 

  37. Johnston, M., Bangalore, S.: Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11(2), 159–187 (2005)

    Article  Google Scholar 

  38. Jolly, S.: Understanding body language: Birdwhistell’s theory of kinesics. Corp. Commun. Int. J. 5(3), 133–139 (2000). https://doi.org/10.1108/13563280010377518

    Article  Google Scholar 

  39. Kendon, A.: Some relationships between body motion and speech. An analysis of an example. In: Siegman, A.W., Pope, B. (eds.) Studies in Dyadic Communication, chap. 9, pp. 177–210. Pergamon Press, Elmsford (1972)

    Google Scholar 

  40. Kendon, A.: Gesticulation and speech: two aspects of the process of utterance. In: Key, M.R. (ed.) The Relationship of Verbal and Nonverbal Communication, pp. 207–227. No. 25 in Contributions to the Sociology of Language, Mouton, The Hague (1980)

    Google Scholar 

  41. Kendon, A.: Gesture: Visible Action as Utterance. Cambridge University Press, Cambridge (2004)

    Book  Google Scholar 

  42. Kennington, C., Schlangen, D.: Incremental unit networks for multimodal, fine-grained information state representation. In: Proceedings of the 1st Workshop on Multimodal Semantic Representations, pp. 89–94. MMSR, Association for Computational Linguistics, Groningen, Netherlands (Online) (2021). https://aclanthology.org/2021.mmsr-1.8

  43. Kiela, D., Bulat, L., Clark, S.: Grounding semantics in olfactory perception. In: Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 231–236. Association for Computational Linguistics, Beijing (2015). https://doi.org/10.3115/v1/P15-2038

  44. Kiela, D., Clark, S.: Learning neural audio embeddings for grounding semantics in auditory perception. J. Artif. Intell. Res. 60, 1003–1030 (2017)

    Article  MathSciNet  Google Scholar 

  45. Koulidobrova, E., Lillo-Martin, D.: A ‘point’ of inquiry: the case of the (non-) pronominal IX in ASL. In: Grosz, P., Patel-Grosz, P. (eds.) Impact of Pronominal Form on Interpretation. Studies in Generative Grammar. De Gruyter Mouton, Berlin and Boston (2016)

    Google Scholar 

  46. Lascarides, A., Stone, M.: Formal semantics of iconic gesture. In: Schlangen, D., Fernández, R. (eds.) Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pp. 64–71. Brandial’06, Universitätsverlag Potsdam, Potsdam (2006)

    Google Scholar 

  47. Linell, P.: The Written Language Bias in Linguistics: Its Nature. Origins and Transformations. Advances in Communication and Linguistic Theory. Routledge, New York (2005)

    Google Scholar 

  48. Lücking, A.: Modeling co-verbal gesture perception in type theory with records. In: Ganzha, M., Maciaszek, L., Paprzycki, M. (eds.) Proceedings of the 2016 Federated Conference on Computer Science and Information Systems. Annals of Computer Science and Information Systems, vol. 8, pp. 383–392. IEEE (2016). https://doi.org/10.15439/2016F83

  49. Lücking, A., Pfeiffer, T.: Framing multimodal technical communication. With focal points in speech-gesture-integration and gaze recognition. In: Mehler, A., Romary, L. (eds.) Handbook of Technical Communication, chap. 18, pp. 591–644. No. 8 in Handbooks of Applied Linguistics, De Gruyter Mouton, Berlin and Boston (2012)

    Google Scholar 

  50. Lücking, A., Pfeiffer, T., Rieser, H.: Pointing and reference reconsidered. J. Pragmat. 77, 56–79 (2015). https://doi.org/10.1016/j.pragma.2014.12.013

    Article  Google Scholar 

  51. Lücking, A., Rieser, H., Staudacher, M.: Multi-modal integration for gesture and speech. In: Schlangen, D., Fernández, R. (eds.) Proceedings of the 10th Workshop on the Semantics and Pragmatics of Dialogue, pp. 106–113. Brandial’06, Universitätsverlag Potsdam, Potsdam (2006)

    Google Scholar 

  52. McNeill, D.: So you think gestures are nonverbal? Psychol. Rev. 92(3), 350–371 (1985)

    Article  Google Scholar 

  53. McNeill, D.: Hand and Mind - What Gestures Reveal about Thought. Chicago University Press, Chicago (1992)

    Google Scholar 

  54. Mehler, A., Lücking, A.: Pathways of alignment between gesture and speech: assessing information transmission in multimodal ensembles. In: Giorgolo, G., Alahverdzhieva, K. (eds.) Proceedings of the International Workshop on Formal and Computational Approaches to Multimodal Communication Under the Auspices of ESSLLI 2012, Opole, Poland, 6–10 August (2012)

    Google Scholar 

  55. Oviatt, S.L.: Multimodal interfaces. In: Jacko, J.A., Sears, A. (eds.) The Human-Computer Interaction Handbook, chap. 21, pp. 286–304. L. Erlbaum Associates Inc., Hillsdale (2003)

    Google Scholar 

  56. Özkan, E.E., Gurion, T., Hough, J., Healey, P.G., Jamone, L.: Specific hand motion patterns correlate to miscommunications during dyadic conversations. In: 2021 IEEE International Conference on Development and Learning, pp. 1–6. ICDL (2021). https://doi.org/10.1109/ICDL49984.2021.9515613

  57. Palmer, M., Gildea, D., Kingsbury, P.: The proposition bank: an annotated corpus of semantic roles. Comput. Linguist. 31(1), 71–106 (2005). https://doi.org/10.1162/0891201053630264

    Article  Google Scholar 

  58. Poggi, I., Pelachaud, C.: Performative faces. Speech Commun. 26(1), 5–21 (1998). https://doi.org/10.1016/S0167-6393(98)00047-8

    Article  Google Scholar 

  59. Pouw, W., Dixon, J.A.: Gesture networks: introducing dynamic time warping and network analysis for the kinematic study of gesture ensembles. Discourse Processes 57(4), 301–319 (2020). https://doi.org/10.1080/0163853X.2019.1678967

    Article  Google Scholar 

  60. Quintilian, M.F.: Institutio oratoria. http://thelatinlibrary.com/quintilian/quintilian.institutio.shtml (1 Century)

  61. Rasenberg, M., Özyürek, A., Dingemanse, M.: Alignment in multimodal interaction: an integrative framework. Cognit. Sci. 44(11), e12911 (2020). https://doi.org/10.1111/cogs.12911

    Article  Google Scholar 

  62. Rieser, H.: Pointing in dialogue. In: Proceedings of the Eighth Workshop on the Semantics and Pragmatics of Dialogue, pp. 93–100. Catalog ’04, Barcelona (2004)

    Google Scholar 

  63. Rieser, H.: On factoring out a gesture typology from the Bielefeld speech-and-gesture-alignment corpus (SAGA). In: Kopp, S., Wachsmuth, I. (eds.) GW 2009. LNCS (LNAI), vol. 5934, pp. 47–60. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-642-12553-9_5

  64. Rieser, H., Lawler, I.: Multi-modal meaning - an empirically-founded process algebra approach. Semant. Pragmat. 13(8), n/a (2020). https://doi.org/10.3765/sp.13.8

  65. Schegloff, E.A.: On some gestures’ relation to talk. In: Atkinson, J.M., Heritage, J. (eds.) Structures of Social Action. Studies in Conversational Analysis, chap. 12, pp. 266–296. Studies in Emotion and Social Interaction. Cambridge University Press, Cambridge (1984)

    Google Scholar 

  66. Schlenker, P.: Gesture projection and cosuppositions. Linguist. Philos. 41(3), 295–365 (2018). https://doi.org/10.1007/s10988-017-9225-8

    Article  Google Scholar 

  67. Vilhjálmsson, H., et al.: The behavior markup language: recent developments and challenges. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 99–111. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74997-4_10

  68. Wang, I., et al.: EGGNOG: a continuous, multi-modal data set of naturally occurring gestures with ground truth labels. In: The 12th IEEE International Conference on Automatic Face Gesture Recognition, pp. 414–421. FG 2017 (2017). https://doi.org/10.1109/FG.2017.145

  69. Wohltjen, S., Wheatley, T.: Eye contact marks the rise and fall of shared attention in conversation. Proc. Natl. Acad. Sci. 118(37), e2106645118 (2021). https://doi.org/10.1073/pnas.2106645118

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Linguistics, Goethe University Frankfurt, Frankfurt, Germany

    Cornelia Ebert

  2. Text Technology Lab, Goethe University Frankfurt, Frankfurt, Germany

    Andy Lücking & Alexander Mehler

  3. Université Paris Cité, Paris, France

    Andy Lücking

Authors
  1. Cornelia Ebert
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andy Lücking
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Alexander Mehler
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Andy Lücking .

Editor information

Editors and Affiliations

  1. The Open University of Japan, Chiba, Japan

    Masaaki Kurosu

  2. Tokyo University of Science, Tokyo, Saitama, Japan

    Sakae Yamamoto

  3. Tokyo City University, Tokyo, Japan

    Hirohiko Mori

  4. Soar Technology Inc., Orlando, FL, USA

    Dylan D. Schmorrow

  5. Katmai Government Services, Orlando, FL, USA

    Cali M. Fidopiastis

  6. Smart Future Initiative, Frankfurt am Main, Germany

    Norbert A. Streitz

  7. Kyushu University, Fukuoka, Japan

    Shin'ichi Konomi

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ebert, C., Lücking, A., Mehler, A. (2022). Introduction to the 2nd Edition of “Semantic, Artificial and Computational Interaction Studies”. In: Kurosu, M., et al. HCI International 2022 - Late Breaking Papers. Multimodality in Advanced Interaction Environments. HCII 2022. Lecture Notes in Computer Science, vol 13519. Springer, Cham. https://doi.org/10.1007/978-3-031-17618-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17618-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17617-3

  • Online ISBN: 978-3-031-17618-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation