Skip to main content
Springer Nature Link
Log in

Multi-track Annotation of Child Language and Gestures

  • Chapter
Multimodal Corpora (MMCorp 2008)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5509))

Included in the following conference series:

  • 1307 Accesses

Abstract

This paper presents the method and tools applied to the annotation of a corpus of children’s oral and multimodal discourse. The multimodal reality of speech has been long established and is now studied extensively. Linguists and psycholinguists who focus on language acquisition also begin to study child language with a multimodal perspective. In both cases, the annotation of multimodal corpora remains a crucial issue as the preparation of the analysis tools has to be in line with the objectives and goals of the research. In this paper we present a coding manual aimed at the annotation of linguistic and gesture production of narratives performed by children and adults of different languages, with emphasis to the relationship between speech and gesture and how it develops. We also present a third coder (3-coder) validation method for gesture annotation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. Beattie, G., Shovelton, H.: When size really matters: How a single semantic feature is represented in the speech and gesture modalities. Gesture 6(1), 63–84 (2006)

    Article  Google Scholar 

  2. Berman, R.A., Slobin, D.I.: Relating events in narrative: A cross-linguistic developmental study. Lawrence Erlbaum Associates, Hillsdale (1994)

    Google Scholar 

  3. Brookes, H.: O clever ‘He’s streetwise’. When gestures become quotable: the case of the clever gesture. Gesture 1, 167–184 (2001)

    Article  Google Scholar 

  4. Butcher, C., Goldin-Meadow, S.: Gesture and the transition from one- to two-word speech: When hand and mouth come together. In: McNeill, D. (ed.) Language and gesture, pp. 235–257. Cambridge University Press, Cambridge (2000)

    Chapter  Google Scholar 

  5. Brugman, H., Russel, A.: Annotating Multi-media / Multi-modal resources with ELAN. In: 4th International Conference on Language Resources and Language Evolution (LREC 2004), Lisbon, May 26-28 (2004)

    Google Scholar 

  6. Calbris, G.: Multicanalité de la communication et multifonctionnalité du geste. In: Perrot, J. (ed.) Polyphonie pour Yvan Fonagy, Paris, L’Harmattan (1997)

    Google Scholar 

  7. Calbris, G.: L’expression gestuelle de la pensée d’un homme politique. Paris, Editions du CNRS (2003)

    Google Scholar 

  8. Capirci, O., Iverson, J.M., Pizzuto, E., Volterra, V.: Gesture and words during the transition to two-word speech. Journal of Child Language 23, 645–673 (1996)

    Article  Google Scholar 

  9. Colletta, J.-M.: Le développement de la parole chez l’enfant âgé de 6 à 11 ans. Corps, langage et cognition. Hayen, Mardaga (2004)

    Google Scholar 

  10. Colletta, J.-M.: Comparative analysis of children’s narratives at different ages: A multimodal approach (in press)

    Google Scholar 

  11. Colletta, J.-M., Pellenq, C.: Les coverbaux de l’explication chez l’enfant âgé de 3 à 11 ans. In: Actes du 2e Congrès de l’ISGS: Interacting bodies, corps en interaction, Lyon, June 15-18 (2005), CDRom Proceedings (2007)

    Google Scholar 

  12. Colletta, J.-M., Pellenq, C.: The development of multimodal explanations in French children. In: Nippold, M., Scott, C. (eds.) Expository Discourse Development and Disorders: International Perspectives (in press)

    Google Scholar 

  13. Colletta, J.-M., Pellenq, C., Guidetti, M.: Age-related changes in co-speech gesture and narrative. Evidence form French children and adults (submitted)

    Google Scholar 

  14. Cosnier, J.: Etude de la mimogestualité. In: Pléty, R. (ed.) Ethologie des communications humaines: aide-mémoire méthodologique, pp. 103–115. ARCI et Presses Universitaires de Lyon, Lyon (1993)

    Google Scholar 

  15. Diessel, H.: The acquisition of complex sentences. Cambridge University Press, Cambridge (2004)

    Book  Google Scholar 

  16. Ech Chafai, N., Pelachaud, C., Pelé, D.: Analysis of gesture expressivity modulations from cartoons animations. In: LREC 2006 Workshop on “Multimodal Corpora”, Genova, Italy, May 27 (2006)

    Google Scholar 

  17. Fayol, M.: Des idées au texte. Psychologie cognitive de la production verbale, orale et écrite. Paris, P.U.F. (1997)

    Google Scholar 

  18. Fayol, M.: Comprendre et produire des textes écrits: l’exemple du récit. In: Kail, M., Fayol, M. (eds.) L’acquisition du langage, T.2: Le langage en développement. Au-delà de trois ans, pp. 183–213. Paris, P.U.F. (2000)

    Google Scholar 

  19. Goldin-Meadow, S.: Hearing gesture. How our hands help us think. Harvard University Press, Cambridge (2003)

    Google Scholar 

  20. Goldin-Meadow, S.: Talking and thinking with our hands. Current Directions in Psychological Science 15, 34–39 (2006)

    Article  Google Scholar 

  21. Guidetti, M.: Yes or no? How young French children combine gestures and speech to agree and refuse. Journal of Child Language 32(4), 911–924 (2005)

    Article  Google Scholar 

  22. Hickmann, M.: Children’s discourse: person, space and time across languages. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  23. Jisa, H.: Growing into academic French. In: Berman, R. (ed.) Language Development across Childhood and Adolescence, Trends in Language Acquisition Research, vol. 3, pp. 135–161. John Benjamins, Amsterdam (2004)

    Chapter  Google Scholar 

  24. Jisa, H., Kern, S.: Relative clauses in French children’s narrative texts. Journal of Child Language 25, 623–652 (1998)

    Article  Google Scholar 

  25. Kendon, A.: Some relationships between body motion and speech. In: Siegman, A.W., Pope, B. (eds.) Studies in dyadic communication, pp. 177–210. Pergamon Press, Elmsford (1972)

    Chapter  Google Scholar 

  26. Kendon, A.: Gesticulation and speech, two aspects of the process of utterance. In: Key, M.R. (ed.) The relationship of verbal and nonverbal communication, pp. 207–227. The Hague, Mouton (1980)

    Google Scholar 

  27. Kendon, A.: Gesture. Visible action as utterance. Cambridge University Press, Cambridge (2004)

    Google Scholar 

  28. Kendon, A.: Reflections on the Development of Gestural Competence. In: Conference, ANR Multimodality Project, Université Stendhal, Grenoble (July 2006)

    Google Scholar 

  29. Kipp, M., Neff, M., Albrecht, I.: An Annotation Scheme for Conversational Gestures: How to economically capture timing and form. In: Proceedings of the Workshop on Multimodal Corpora (LREC 2006), pp. 24–27 (2006)

    Google Scholar 

  30. Kipp, M., Neff, M., Kipp, K.H., Albrecht, I.: Towards Natural Gesture Synthesis: Evaluating gesture units in a data-driven approach to gesture synthesis. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 15–28. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  31. Kita, S.: Language and gesture interface: A study of spontaneous gestures and Japanese mimetics. Unpublished doctoral dissertation. University of Chicago (1993)

    Google Scholar 

  32. Kita, S.: Jesuchaa: Kangaeru karada [Gesture: The body that thinks]. Kanebo Shobo, Tokyo (2002)

    Google Scholar 

  33. Kita, S., Özyürek, A.: What does cross-linguistic variation in semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Journal of Memory and Language 48, 16–32 (2003)

    Article  Google Scholar 

  34. Kita, S., Özyürek, A.: How does spoken language shape iconic gestures. In: Duncan, S.D., Cassell, J., Levy, E.T. (eds.) Gesture and the dynamic dimension of language. Essays in honor of David McNeill, pp. 67–74. John Benjamins, Amsterdam (2007)

    Chapter  Google Scholar 

  35. Kopp, S., Krenn, B., Marsella, S., Marshall, A.N., Pelachaud, C., Pirker, H., Thórisson, K.R., Vilhjálmsson, H.: Towards a Common Framework for Multimodal Generation: The Behavior Markup Language. In: Gratch, J., Young, M., Aylett, R.S., Ballin, D., Olivier, P. (eds.) IVA 2006. LNCS (LNAI), vol. 4133, pp. 205–217. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  36. Kunene, R., Colletta, J.-M.: A Cross-linguistic annotation model for children and adults oral language and spontaneous gestures. In: Workshop on Speech and Face to Face communication, Grenoble, France, October 27-29 (2008)

    Google Scholar 

  37. Le Chenadec, G., Maffiolo, V., Chateau, N.: Analysis of the multimodal behavior of users in HCI: the expert viewpoint of close relations. In: 4th Joint Workshop on Multimodal Interaction and Related Machine Learning Algorithms, Brno, Czech Republic, June 28-30 (2007)

    Google Scholar 

  38. Le Chenadec, G., Maffiolo, V., Chateau, N., Colletta, J.M.: Creation of a Corpus of Multimodal Spontaneous Expressions of Emotions in Human-Interaction. In: LREC 2006, Genoa, Italy (2006)

    Google Scholar 

  39. Martin, J.C.: TYCOON: six primitive types of cooperation for observing, evaluating and specifying cooperations. In: AAAI Fall 1999. Symposium on Psychological Models of Communication in Collaborative Systems, November 5-7 (1999); Sea Crest Conference Center on Cape Cod, North Falmouth, Massachusetts, USA (1999)

    Google Scholar 

  40. McNeill, D.: Hand and mind. What gestures reveal about thought. University of Chicago Press, Chicago (1992)

    Google Scholar 

  41. McNeill, D.: Gesture and thought. University of Chicago Press, Chicago (2005)

    Book  Google Scholar 

  42. McNeill, D., Quek, F., McCullough, K.E., Duncan, S., Furuyama, N., Bryll, R., Ma, X.F., Ansari, R.: Catchments, prosody and discourse. Gesture 1(1), 9–33 (2001)

    Article  Google Scholar 

  43. Özcaliskan, S., Goldin-Meadow, S.: How gesture helps children construct language. In: Clark, E.V., Kelly, B.F. (eds.) Constructions in Acquisition, pp. 31–58. CSLI Publications, Palo Alto (2006)

    Google Scholar 

  44. Özyürek, A., Kita, S., Allen, S., Furman, R., Brown, A.: How does linguistic framing influence co-speech gestures? Insights from cross-linguistic differences and similarities. Gesture 5(1/2), 216–241 (2005)

    Google Scholar 

  45. Scherer, K.R.: Les fonctions des signes non verbaux dans la conversation. In: Cosnier, J., Brossard, A. (eds.) La communication non verbale, pp. 71–100. Neuchâtel, Delachaux et Niestlé (1984)

    Google Scholar 

  46. Slobin, D.I.: From « thought and language » to « thinking for speaking ». In: Gumperz, J.J., Levinson, S.C. (eds.) Rethinking linguistic relativity, pp. 70–96. Cambridge University Press, Cambridge (1996)

    Google Scholar 

  47. Talmy, L.: Lexicalisation patterns: semantic structures in lexical forms. In: Language typology and syntactic description. Grammatical categories and the lexicon, vol. 3, pp. 57–149. Cambridge University Press, Cambridge (1985)

    Google Scholar 

  48. Vilhjalmsson, H., Cantelmo, N., Cassell, J., Chafai, E.N., Kipp, M., Kopp, S., Mancini, M., Marsella, S., Marshall, A.N., Pelachaud, C., Ruttkay, Z., Thórisson, K.R., van Welbergen, H., van der Werf, R.J.: The Behavior Markup Language: Recent Developments and Challenges. In: Pelachaud, C., Martin, J.-C., André, E., Chollet, G., Karpouzis, K., Pelé, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 99–111. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  49. Volterra, V., Caselli, M.C., Capirci, O., Pizzuto, E.: Gesture and the emergence and development of language. In: Tomasello, M., Slobin, D. (eds.) Beyond Nature-Nurture. Essays in Honor of Elizabeth Bates, pp. 3–40. Lawrence Erlbaum Associates, Mahwah (2005)

    Google Scholar 

  50. de Weck, G.: La cohésion dans les textes d’enfants. Etude du développement des processus anaphoriques. Neuchâtel, Delachaux et Niestlé (1991)

    Google Scholar 

  51. Weinrich, H.: Grammaire textuelle du français. Paris, Didier/Hatier (1989)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Lidilem – IUFM and Université Stendhal, BP25, 38040, Grenoble Cedex 9, France

    Jean-Marc Colletta, Ramona N. Kunene, Aurélie Venouil, Virginie Kaufmann & Jean-Pascal Simon

Authors
  1. Jean-Marc Colletta
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ramona N. Kunene
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aurélie Venouil
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Virginie Kaufmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jean-Pascal Simon
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Deutsches Forschungszentrum für künstliche Intelligenz (DFKI), Campus D3.2, 66123, Saarbrücken, Germany

    Michael Kipp

  2. Laboratoire d’Informatique pour la Mécanique et les Sciences de l’Ingénieur (LIMSI-CNRS), BP 133, 91403, Orsay Cedex, France

    Jean-Claude Martin

  3. Faculty of Humanities, Centre for Language Technology, University of Copenhagen, Njalsgade 140-142, 2300, Copenhagen, Denmark

    Patrizia Paggio

  4. Computer Science, Human Media Interaction, University of Twente, PO Box 217, 7500, Enschede, AE, The Netherlands

    Dirk Heylen

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Colletta, JM., Kunene, R.N., Venouil, A., Kaufmann, V., Simon, JP. (2009). Multi-track Annotation of Child Language and Gestures. In: Kipp, M., Martin, JC., Paggio, P., Heylen, D. (eds) Multimodal Corpora. MMCorp 2008. Lecture Notes in Computer Science(), vol 5509. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04793-0_4

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-04793-0_4

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-04792-3

  • Online ISBN: 978-3-642-04793-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics