Skip to main content
SpringerLink
Log in

Multimodal Corpora Annotation: Validation Methods to Assess Coding Scheme Reliability

  • Chapter
Multimodal Corpora (MMCorp 2008)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 5509))

Included in the following conference series:

Abstract

Many multimodal corpora have been collected and annotated in the last years. Unfortunately, in many cases most of the multimodal coding schemes have been shown not to be reliable. This poor reliability may be caused either by the nature of multimodal data or by the nature of statistic methods to assess reliability. In this paper we will review the statistical measures currently used to assess agreement on multimodal corpora annotation. We will also propose alternative statistical methods to the well known kappa statistics.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Abrilian, S., Buisine, S., Devillers, L., Martin, J.-C.: EmoTV: Annotation of Real-life Emotions for the Specification of Multimodal Affective Interfaces. In: Proceedings of International HCI (2005)

    Google Scholar 

  2. Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: The MUMIN coding scheme for the annotation of feedback, turn management and sequencing phenomena. Language Resources and Evaluation 41(3-4), 273–287 (2007)

    Article  Google Scholar 

  3. Allwood, J., Cerrato, L., Jokinen, K., Navarretta, C., Paggio, P.: A Coding Scheme for the Annotation of Feedback, Turn Management and Sequencing Phenomena. In: Martin, J.-C., Kühnlein, P., Paggio, P., Stiefelhagen, R., Pianesi, F. (eds.) Multimodal Corpora: From Multimodal Behavior Theories to Usable Models, pp. 38–42 (2006)

    Google Scholar 

  4. Artstein, R., Poesio, M.: Inter-Coder Agreement for Computational Linguistics. Computational Linguistics 34, 555–596 (2008)

    Article  Google Scholar 

  5. Artstein, R., Poesio, M.: Bias decreases in proportion to the number of annotators. In: Proceedings of FG-MoL (2005)

    Google Scholar 

  6. Carletta, J.: Unleashing the killer corpus: experiences in creating the multi-everything AMI Meeting Corpus. Language Resources and Evaluation 41, 181–190 (2007)

    Article  Google Scholar 

  7. Carletta, J.: Assessing agreement on classification tasks: The kappa statistic. Computational Linguistics 22, 249–254 (1996)

    Google Scholar 

  8. Cavicchio, F., Poesio, M.: Annotation of Emotion in Dialogue: The Emotion in Cooperation in Perception. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS (LNAI), vol. 5078, pp. 233–239. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  9. Cerrato, L.: A coding scheme for the annotation of feedback phenomena in conversational speech. In: Martin, J.-C., Os, E.D., Kühnlein, P., Boves, L., Paggio, P., Catizone, R. (eds.) Proceedings of Workshop Multimodal Corpora: Models of Human Behavior for the Specification and Evaluation of Multimodal Input and Output Interfaces, pp. 25–28 (2004)

    Google Scholar 

  10. Colletta, J.-M., Kunene, R., Venouil, A., Tcherkassof, A.: Double Level Analysis of the Multimodal Expressions of Emotions in Human-machine Interaction. In: Martin, J.-C., Patrizia, P., Kipp, M., Heylen, D. (eds.) Multimodal Corpora: From Models of Natural Interaction to Systems and Applications, pp. 5–11 (2008)

    Google Scholar 

  11. Colletta, J.-M., Kunene, R.N., Venouil, A., Kaufmann, V., Simon, J.-P.: Multitrack Annotation of Child Language and Gestures. In: Kipp, M., Martin, J.-C., Paggio, P., Heylen, D. (eds.) Multimodal Corpora: From Models of Natural Interaction to Systems and Applications, LNAI 5509. LNCS (LNAI), vol. 5509, pp. 54–72. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  12. Douglas-Cowie, E., Devillers, L., Martin, J.-C., Cowie, R., Savvidou, S., Abrilian, S., Cox, C.: Multimodal Databases of Everyday Emotion: Facing up to Complexity. In: Interspeech 2005, Lisbon, Portugal, September 4-8, pp. 813–816 (2005)

    Google Scholar 

  13. Dybkjær, L., Bernsen, N.O.: Recommendations for Natural Interactivity and Multimodal Annotation Schemes. In: Martin, J.-C., Os, E.D., Kühnlein, P., Boves, L., Paggio, P., Catizone, R. (eds.) Proceedings of Workshop Multimodal Corpora: Models of Human Behavior For The Specification And Evaluation of Multimodal Input And Output Interfaces, pp. 5–8 (2004)

    Google Scholar 

  14. Ekman, P., Friesen, W.V.: A new pan-cultural facial expression of emotion. Motivation and emotion 10, 159–168 (1986)

    Article  Google Scholar 

  15. Fleiss, J.L.: Measuring nominal scale agreement among many raters. Psychological Bulletin 76, 378–382 (1971)

    Article  Google Scholar 

  16. Goeleven, E., De Raedt, R., Leyman, L., Verschuere, B.: The Karolinska Directed Emotional Faces: A validation study. Cognition and Emotion 22, 1094–1118 (2008)

    Article  Google Scholar 

  17. Guerini, M., Stock, O., Zancanaro, M.: A Taxonomy of Strategies for Multimodal Persuasive Message Generation. Applied Artificial Intelligence 21(2), 99–136 (2007)

    Article  Google Scholar 

  18. Gut, U., Looks, K., Thies, A., Gibbon, D.: CoGesT–Conversational Gesture Transcription System. Version 1.0. Technical report. Bielefeld University (2003)

    Google Scholar 

  19. Hall, J.A., Levin, S.: Affect and verbal-nonverbal discrepancy in schizophrenic and non-schizophrenic family communication. British Journal of Psychiatry 137, 78–92 (1980)

    Article  Google Scholar 

  20. Hietanen, J.K., Leppänen, J.M., Lehtonen, U.: Perception of Emotions in the Hand Movement Quality of Finnish Sign Language. Journal of Nonverbal Behavior 28, 53–64 (2004)

    Article  Google Scholar 

  21. Kipp, M., Neff, M., Albrecht, I.: An Annotation Scheme for Conversational Gestures: How to economically capture timing and form. In: Martin, J.-C., Kühnlein, P., Paggio, P., Stiefelhagen, R., Pianesi, F. (eds.) Multimodal Corpora: From Multimodal Behavior Theories to Usable Models, pp. 24–28 (2006)

    Google Scholar 

  22. Kowtko, J.C., Isard, S.D., Doherty, G.M.: Conversational games within dialogue. Technical Report HCRC/RP-31, Human Communication Research Centre, University of Edinburgh (1992)

    Google Scholar 

  23. Krippendorff, K.: Reliability in content analysis: Some common misconceptions and recommendations. Human Communication Research 30, 411–433 (2004)

    Google Scholar 

  24. Krippendorff, K.: Content Analysis: An introduction to its Methodology. Sage Publications, Thousand Oaks (1980)

    MATH  Google Scholar 

  25. Litman, D., Hirschberg, J.: Disambiguating cue phrases in text and speech. In: Proceedings of the Thirteenth International Conference on Computational Linguistics, pp. 51–256 (1990)

    Google Scholar 

  26. Magno Caldognetto, E., Poggi, I., Cosi, P., Cavicchio, F., Merola, G.: Multimodal Score: an Anvil Based Annotation Scheme for Multimodal Audio-Video Analysis. In: Martin, J.-C., Os, E.D., Kühnlein, P., Boves, L., Paggio, P., Catizone, R. (eds.) Proceedings of Workshop "Multimodal Corpora: Models of Human Behavior For The Specification And Evaluation of Multimodal Input And Output Interfaces", pp. 29–33 (2004)

    Google Scholar 

  27. Martel, C., Osborn, C., Friedman, J., Howard, P.: The FORM Gesture Annotation System. In: Maybury, M., Martin, J.-C. (eds.) Proceedings of Multimodal Resources and Multimodal Systems Evaluation Workshop, pp. 10–15 (2002)

    Google Scholar 

  28. Martell, C., Kroll, J.: Using FORM Gesture Data to Predict Phase Labels. In: Martin, J.-C., Kühnlein, P., Paggio, P., Stiefelhagen, R., Pianesi, F. (eds.) Multimodal Corpora: From Multimodal Behavior Theories to Usable Models, pp. 29–32 (2006)

    Google Scholar 

  29. McNeill, D.: Hand and mind: What the hands reveal about thought. University of Chicago Press, Chicago (1992)

    Google Scholar 

  30. Passonneau, R.J., Litman, D.: Intention-based segmentation: human reliability and correlation with linguistic cues. In: Proceedings of the 31st Annual Meeting of the ACL, pp. 148–155 (1993)

    Google Scholar 

  31. Pianesi, F., Leonardi, C., Zancanaro, M.: Multimodal Annotated Corpora of Consensus Decision Making Meetings. In: Martin, J.-C., Kühnlein, P., Paggio, P., Stiefelhagen, R., Pianesi, F. (eds.) Multimodal Corpora: From Multimodal Behavior Theories to Usable Models, pp. 6–9 (2006)

    Google Scholar 

  32. Poggi, I., Vincze, L.: The Persuasive Impact of Gesture and Gaze. In: Kipp, M., Martin, J.-C., Paggio, P., Heylen, D. (eds.) Multimodal Corpora: From Models of Natural Interaction to Systems and Applications, LNAI 5509. LNCS (LNAI), vol. 5509, pp. 73–92. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  33. Poggi, I.: Mind, hands, face and body. A goal and belief view of multimodal communication. Weidler Buchverlag, Berlin (2007)

    Google Scholar 

  34. Poggi, I., Cavicchio, F., Magno Caldognetto, E.: Irony in a judicial debate: analyzing the subtleties of irony while testing the subtleties of an annotation scheme. Language Resources and Evaluation 41(3-4), 215–232 (2007)

    Article  Google Scholar 

  35. Reidsma, D., Carletta, J.: Reliability Measurement without Limits. Computational Linguistics 34, 319–326 (2008)

    Article  Google Scholar 

  36. Reidsma, D., Heylen, D., op den Akker, R.: On the Contextual Analysis of Agreement Scores. In: Kipp, M., Martin, J.-C., Paggio, P., Heylen, D. (eds.) Multimodal Corpora: From Models of Natural Interaction to Systems and Applications. LNCS (LNAI), vol. 5509, pp. 122–137. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  37. Scott, W.A.: Reliability of content analysis: The case of nominal scale coding. Public Opinion Quarterly 19, 321–325 (1955)

    Article  Google Scholar 

  38. Siegel, S., Castellan Jr., N.J.: Nonparametric Statistics for the Behavioral Sciences, 2nd edn. McGraw-Hill, New York (1988)

    Google Scholar 

  39. Sosnovsky, S., Brusilovsky, P., Lee, D.H., Zadorozhny, V., Zhou, X.: Re-assessing the Value of Adaptive Navigation Support in E-Learning Context. In: Nejdl, W., Kay, J., Pu, P., Herder, E. (eds.) AH 2008. LNCS, vol. 5149, pp. 193–203. Springer, Heidelberg (2008)

    Chapter  Google Scholar 

  40. Wagner, H.L.: On measuring performance in category judgment studies on nonverbal behavior. Journal of Non-verbal Behavior 17, 3–28 (1993)

    Article  Google Scholar 

  41. Wagner, H.L., Smith, J.: Facial Expressions in the Presence of Friends or Strangers. Journal of Nonverbal Behavior 15, 201–214 (1991)

    Article  Google Scholar 

  42. Woodworth, R.S.: Experimental Psychology, 1st edn. Henry Holt, New York (1938)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. CIMeC Università di Trento, Corso Bettini 31, 38068, Rovereto (Tn), Italy

    Federica Cavicchio & Massimo Poesio

Authors
  1. Federica Cavicchio
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Massimo Poesio
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Deutsches Forschungszentrum für künstliche Intelligenz (DFKI), Campus D3.2, 66123, Saarbrücken, Germany

    Michael Kipp

  2. Laboratoire d’Informatique pour la Mécanique et les Sciences de l’Ingénieur (LIMSI-CNRS), BP 133, 91403, Orsay Cedex, France

    Jean-Claude Martin

  3. Faculty of Humanities, Centre for Language Technology, University of Copenhagen, Njalsgade 140-142, 2300, Copenhagen, Denmark

    Patrizia Paggio

  4. Computer Science, Human Media Interaction, University of Twente, PO Box 217, 7500, Enschede, AE, The Netherlands

    Dirk Heylen

Rights and permissions

Reprints and permissions

Copyright information

© 2009 Springer-Verlag Berlin Heidelberg

About this chapter

Cite this chapter

Cavicchio, F., Poesio, M. (2009). Multimodal Corpora Annotation: Validation Methods to Assess Coding Scheme Reliability. In: Kipp, M., Martin, JC., Paggio, P., Heylen, D. (eds) Multimodal Corpora. MMCorp 2008. Lecture Notes in Computer Science(), vol 5509. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-04793-0_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-642-04793-0_7

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-642-04792-3

  • Online ISBN: 978-3-642-04793-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation