Skip to main content
SpringerLink
Log in

An Italian Multimodal Corpus: The Building Process

  • Conference paper
On the Move to Meaningful Internet Systems: OTM 2014 Workshops (OTM 2014)

Part of the book series: Lecture Notes in Computer Science ((LNISA,volume 8842))

  • 1956 Accesses

Abstract

During the design of multimodal interaction environments, the use of a corpus of multimodal sentences is very important in order to achieve various tasks of multimodal interaction. In last decade, several researchers addressed the creation of multimodal corpora for English, French, and various other languages. However, from the analysis of these multimodal corpora, there clearly is a lack of multimodal corpora for Italian. This paper describes the building process of an Italian multimodal corpus. Starting from the manual analysis of multimedia dialogues, this process extracts different multimodal data, i.e. speech and gestures, which are used to generate grammar rules and to train the multimodal interpreter in order to set the framework for the multimodal corpus building. Following that, the set framework is used to annotate semi-automatically multimodal information, such as syntactic roles and semantics, on new dialogues to be included in the corpus.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Ringeval, F., Sonderegger, A., Sauer, J., Lalanne, D.: Introducing the RECOLA Multimodal Corpus of Remote Collaborative and Affective Interactions. In: 2nd International Workshop EmoSPACE (2013)

    Google Scholar 

  2. Inoue, M., Hanada, R., Furuyama, N., Irino, T., Ichinomiya, T., Massaki, H.: Multimodal Corpus for Psychotherapeutic Situations. In: International Workshop Series on Multimodal Corpora, Tools and Resources, pp. 18–21 (2012)

    Google Scholar 

  3. Melvin, R.S., May, W., Narayanan, S., Georgiou, P.G., Ganjavi, S.: Creation of a Doctor-Patient Dialogue Corpus Using Standardized Patients. In: LREC (2004)

    Google Scholar 

  4. Fleury, A., Vacher, M., Portet, F., Chahuara, P., Noury, N.: A multimodal corpus recorded in a health smart home. In: Proceedings of the LREC 2010, pp. 99–105 (2010)

    Google Scholar 

  5. Vacher, M., Lecouteux, B., Chahuara, P., Portet, F., Meillon, B., Bonnefond, N.: The Sweet-Home speech and multimodal corpus for home automation interaction. In: LREC 2014, pp. 1–8 (2014)

    Google Scholar 

  6. Costantini, E., Burger, S., Pianesi, F.: NESPOLE!’s Multilingual and Multimodal Corpus. In: LREC (2002)

    Google Scholar 

  7. D’Ulizia, A., Ferri, F., Grifoni, P.: Toward the Development of an Integrative Framework for Multimodal Dialogue Processing. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2008 Workshops. LNCS, vol. 5333, pp. 509–518. Springer, Heidelberg (2008)

    Google Scholar 

  8. Caschera, M.C., Ferri, F., Grifoni, P.: Multimodal interaction systems: information and time features. International Journal of Web and Grid Services 3(1), 82–99 (2007)

    Article  Google Scholar 

  9. Caschera, M.C., Ferri, F., Grifoni, P.: An Approach for Managing Ambiguities in Multimodal Interaction. In: Meersman, R., Tari, Z. (eds.) OTM-WS 2007, Part I. LNCS, vol. 4805, pp. 387–397. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  10. Avola, D., Caschera, M.C., Ferri, F., Grifoni, P.: Ambiguities in Sketch-Based Interfaces. In: HICSS 2007, p. 290. IEEE Computer Society (2007)

    Google Scholar 

  11. Caschera, M.C., Ferri, F., Grifoni, P.: Ambiguity detection in multimodal systems. In: Advanced Visual Interfaces, AVI 2008, pp. 331–334. ACM Press (2008)

    Google Scholar 

  12. Avola, D., Caschera, M.C., Grifoni, P.: Solving ambiguities for Sketch-Based interaction in mobile environments. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2006 Workshops. LNCS, vol. 4277, pp. 904–915. Springer, Heidelberg (2006)

    Chapter  Google Scholar 

  13. D’Ulizia, A., Ferri, F.: Formalization of multimodal languages in pervasive computing paradigm. In: Damiani, E., Yetongnon, K., Chbeir, R., Dipanda, A. (eds.) SITIS 2006. LNCS, vol. 4879, pp. 126–136. Springer, Heidelberg (2009)

    Chapter  Google Scholar 

  14. Ferri, F., D’Ulizia, A., Grifoni, P.: Multimodal Language Specification for Human Adaptive Mechatronics. JNIT 3(1), 47–57 (2012)

    Article  Google Scholar 

  15. D’Ulizia, A.: Exploring Multimodal Input Fusion Strategies. In: Handbook of Research on Multimodal Human Computer Interaction and Pervasive Services: Evolutionary Techniques for Improving Accessibility, pp. 34–57. IGI Publishing (2009)

    Google Scholar 

  16. D’Ulizia, A., Ferri, F., Grifoni, P.: A Hybrid Grammar-Based Approach to Multimodal Languages Specification. In: Meersman, R., Tari, Z. (eds.) OTM-WS 2007, Part I. LNCS, vol. 4805, pp. 367–376. Springer, Heidelberg (2007)

    Chapter  Google Scholar 

  17. D’Ulizia, A., Ferri, F., Grifoni, P.: A Survey of Grammatical Inference Methods for Natural Language Learning. Artificial Intelligence Review 36(1), 1–27 (2011)

    Article  Google Scholar 

  18. Manchón, P., Pérez, G., Amores, G.: Multimodal Fusion: A New Hybrid Strategy for Dialogue Systems. In: Proceedings of ICMI 2006, pp. 357–363. ACM (2006)

    Google Scholar 

  19. D’Ulizia, A., Ferri, F., Grifoni, P.: Generating Multimodal Grammars for Multimodal Dialogue Processing. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 40(6), 1130–1145 (2010)

    Article  Google Scholar 

  20. Shimazu, H., Takashima, Y.: Multimodal Definite Clause Grammar. Systems and Computers in Japan 26(3), 93–102 (1995)

    Article  Google Scholar 

  21. Johnston, M., Bangalore, S.: Finite-state multimodal integration and understanding. Nat. Lang. Eng. 11(2), 159–187 (2005)

    Article  Google Scholar 

  22. Reitter, D., Panttaja, E.M., Cummins, F.: UI on the fly: Generating a multimodal user interface. In: Proceedings of Human Language Technology Conference (2004)

    Google Scholar 

  23. Pereira, F., Warren, D.H.D.: Definite Clause Grammars for Language Analysis - A survey of the Formalism and a Comparison with Augmented Transition Networks. Artificial Intelligence 13(3) (1980)

    Google Scholar 

  24. Baldridge, J., Kruijff, G.J.M.: Multimodal combinatory categorial grammar. In: Proceedings of the 10th Conference of the European Chapter of the ACL, pp. 211–218 (2003)

    Google Scholar 

  25. D’Andrea, A., D’Ulizia, A., Ferri, F., Grifoni, P.: A Multimodal Pervasive Framework for Ambient Assisted Living. In: Proceedings of the PETRA. ACM Digital Library (2009)

    Google Scholar 

  26. Caschera, M.C.: Interpretation methods and ambiguity management in multimodal systems. In: Handbook of Research on Multimodal Human Computer Interaction and Pervasive Services: Evolutionary Techniques for Improving Accessibility, pp. 87–102. IGI Publishing (2009)

    Google Scholar 

  27. Caschera, M.C., Ferri, F., Grifoni, P.: InteSe: An Integrated Model for Resolving Ambiguities in Multimodal Sentences. IEEE Transactions on Systems, Man, and Cybernetics: Systems 43(4), 911–931 (2013)

    Article  Google Scholar 

  28. Avola, D., Caschera, M.C., Ferri, F., Grifoni, P.: Classifying and resolving ambiguities in sketch-based interaction. Int. J. Virt. Technol. Multimedia 1(2), 104–139 (2010)

    Article  Google Scholar 

  29. Caschera, M.C., Ferri, F., Grifoni, P.: Personal sphere information, histories and social interaction between people on the Internet. In: Meersman, R., Tari, Z., Herrero, P. (eds.) OTM 2008 Workshops. LNCS, vol. 5333, pp. 480–488. Springer, Heidelberg (2008)

    Google Scholar 

  30. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–285 (1989)

    Article  Google Scholar 

  31. Makhoul, J., Starner, T., Schwartz, R., Chou, G.: On-line cursive handwriting recognition using hidden Markov models and statistical grammars. In: Proc. Workshop Hum. Lang. Technol., pp. 432–436 (1994)

    Google Scholar 

  32. Jelinek, F.: Robust part-of-speech tagging using a hidden Markov model. Comput. Speech Lang. 6(3), 225–242 (1992)

    Article  Google Scholar 

  33. Li, N., Busso, C.: Evaluating the robustness of an appearance-based gaze estimation method for multimodal interfaces. In: ICMI 2013, pp. 91–98 (2013)

    Google Scholar 

  34. Caschera, M.C., Ferri, F., Grifoni, P.: From Modal to Multimodal Ambiguities: a Classification Approach. JNIT 4(5), 87–109 (2013)

    Article  Google Scholar 

  35. Allwood, J.: Multimodal Corpora. In: Lüdeling, A., Kytö, M. (eds.) Corpus Linguistics. An International Handbook, pp. 207–225. Mouton de Gruyter, Berlin (2008)

    Google Scholar 

  36. Caschera, M.C., D’Ulizia, A., Ferri, F., Grifoni, P.: Multiculturality and Multimodal Languages. In: Multiple Sensorial Media Advances and Applications: New Developments in MulSeMedia, pp. 99–114. IGI Global Publishing (2012)

    Google Scholar 

  37. http://badip.uni-graz.at/it/lista-di-corpora

  38. Caschera, M.C., D’Ulizia, A., Ferri, F., Grifoni, P.: Methods for dynamic building of multimodal corpora. In: LTC 2013, pp. 499–503 (2013)

    Google Scholar 

  39. https://www.youtube.com/user/L2pack

  40. http://www.anvil-software.org/

  41. Bosco, C., Montemagni, S., Simi, M.: Converting Italian Treebanks: Towards an Italian Stanford Dependency Treebank. In: ACL Workshop (2013)

    Google Scholar 

  42. D’Ulizia, A., Ferri, F., Grifoni, P.: A Learning Algorithm for Multimodal Grammar Inference. IEEE Transactions on Systems, Man, and Cybernetics - Part B: Cybernetics 41(6), 1495–1510 (2011)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Research on Population and Social Policies (IRPPS), National Research Council (CNR), 00185, Rome, Italy

    Maria Chiara Caschera, Arianna D’Ulizia, Fernando Ferri & Patrizia Grifoni

Authors
  1. Maria Chiara Caschera
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arianna D’Ulizia
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fernando Ferri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Patrizia Grifoni
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. TU Graz, RechbauerstraĂźe 12, 8010, Graz, Austria

    Robert Meersman

  2. CRAN, University of Lorraine, Campus Sciences, BP 70239, 54506, Vandoevre-les-Nancy, France

    Hervé Panetto

  3. Department of Software Engineering, Atilim Univeristy, Kızılcaşar Mh Mh, 06830, Ankara, Turkey

    Alok Mishra

  4. Facultad de Informática, Universidad de Murcia, Campus de Espinardo, 30100, Murcia, Spain

    Rafael Valencia-GarcĂ­a

  5. Dep. of Informatics Engineering, University of Porto, Rua Dr. Roberto Frias, 4200-465, Porto, Portugal

    AntĂłnio Lucas Soares

  6. SIGMA / LIG, Joseph Fourier University, 220 Rue de la Chimie, BP 53, 38041, Grenoble Cedex 9, France

    Ioana Ciuciu

  7. Institute for Research on Population and Social, National Research Council, Policies, Via Palestro 32, 00185, Rome, Italy

    Fernando Ferri

  8. Dep. Business Informatics, Communications Engineering, Johannes Kepler University, Freistaedterstrasse 315, 4040, Linz, Austria

    Georg Weichhart

  9. STINA Business Solutions GmbH, Schottenfeldgasse 63/2, 1070, Wien, Austria

    Thomas Moser

  10. SAP Research, 805 avenue du Dr Donat, 06259, Sophia-Antipolis, Mougins, France

    Michele Bezzi

  11. Department of Computing, Hong Kong Polytechnic University, PQ806, Mong Man Wai Building, Hong Kong,, China

    Henry Chan

Rights and permissions

Reprints and permissions

Copyright information

© 2014 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Caschera, M.C., D’Ulizia, A., Ferri, F., Grifoni, P. (2014). An Italian Multimodal Corpus: The Building Process. In: Meersman, R., et al. On the Move to Meaningful Internet Systems: OTM 2014 Workshops. OTM 2014. Lecture Notes in Computer Science, vol 8842. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-45550-0_57

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-45550-0_57

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-45549-4

  • Online ISBN: 978-3-662-45550-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation