Skip to main content
Springer Nature Link
Log in

Developing multimodal interfaces: A theoretical framework and guided propagation networks

  • Conference paper
  • First Online:
Multimodal Human-Computer Communication (CMC 1995)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 1374))

Included in the following conference series:

  • 307 Accesses

Abstract

In this paper we propose an approach for the design of related theoretical and software tools for developing multimodal interfaces. A theoretical framework is described based on the notion of types of cooperation between modalities It forms the basis of a specification language that we used for developing multimodal interfaces to three test applications. This specification language is interpreted by a multimodal module made of Guided Propagation Networks.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  • André, E. and Rist, T. (1995) Generating coherent presentations employing textual and visual material. Artificial Intelligence Review 9 (2–3), 147–165.

    Article  Google Scholar 

  • Baekgaard, A. (1995) Constraining of input media in a spoken dialog system. In Proc. 4th European Conference on Speech Communication and Technology (EUROSPEECH'95), 1181–1184.

    Google Scholar 

  • Bellalem, N. and Romary, L. (1995) Reference interpretation in a multimodal environment combining speech and gesture. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh.

    Google Scholar 

  • Béroule, D. (1985) Un modelé de mémoire adaptative, dynamique et associative pour le traitement automatique de la parole. Thesis, University of Paris XI, Orsay.

    Google Scholar 

  • Béroule, D. (1988) The never-ending learning. In R. Eckmiller and C. v. d. Malsburg (eds.), Neural Computers. NATO ASI Series F, vol 41. Berlin: Springer, 219–230.

    Google Scholar 

  • Béroule, D. (1990) Guided propagation: current state of theory and application. In F. Fogelman Souliè and J. Hérault (eds.) Neurocomputing, NATO ASI Series, Vol. F 68, 241–260. Berlin: Springer.

    Chapter  Google Scholar 

  • Béroule, D., Von Hoe, R. and Ruellan, H. (1994) A Guided Propagation Model of Reading. Annual Progress Report 28, Instituut voor Perceptie Onderzoek IPO, Eindhoven, 21–29.

    Google Scholar 

  • Blanchet, P. (1992) Une architecture connexionniste pour l'apprentissage par l'expérience et la représentation des connaissances. Thesis, University of Paris XI, Orsay.

    Google Scholar 

  • Bolt, R.A. (1980) 'Put — That — There': Voice and Gesture at The Graphics Interface. Computer Graphics 14 (3), 262–270.

    Article  MathSciNet  Google Scholar 

  • Bos, E. (1993) Easier said or done? Studies in multimodal human-computer interaction. NICI technical report 93-02, University of Nijmegen.

    Google Scholar 

  • Bourdot, P., Krus, M., Gherbi, R. (1995) Management of non-standard devices for multimodal user interfaces under UNIX/X11. This volume.

    Google Scholar 

  • Bressolle, M.C, Pavard, B., Leroux, M. (1997) The role of multimodal communication in cooperation and intention recognition: the case of air traffic control. This volume.

    Google Scholar 

  • Briffault, X. (1996) Une interface multimodale pour l'aide a la navigation. Working paper, LIMSI, Orsay. http://www.limsi.fr/Individu/xavier/index.html

    Google Scholar 

  • Bunt, H., Beun, R. J., and Borghuis, T. (eds.) Proceedings of the International Conference on Cooperative Multimodal Communication CMC/95. Eindhoven, May 24–26.

    Google Scholar 

  • Carbonnel, J.R. (1970) Mixed-Initiative Man-Computer Dialogues. Bolt, Beranek and Newman (BBN) Report N 1971, Cambridge, MA.

    Google Scholar 

  • Catinis, L., Caelen, J. (1995) Analyse du comportement multimodal de l'usager humain dans une tache de dessin. Actes des 7. Journées sur l'Ingéniérie de l'Interaction Homme-Machine (IHM'95), 123-129.

    Google Scholar 

  • Cheyer, A. and Julia, L. (1995) Multimodal maps: an agentbased approach. This volume.

    Google Scholar 

  • Coutaz, J., Salber, D., Carraux, E. and Portolan, N. (1996) NEIMO, a multiworkstation usability lab for observing and analyzing multimodal interaction. To appear in CHI'96 Conference Proceedings Companion. Video.

    Google Scholar 

  • Coutaz, J. and Nigay, L. (1994) Les propriétés CARE dans les interfaces multimodales. Actes des 6èmes Journées sur l'Ingéniérie de l'Interaction Homme-Machine (IHM'94), Lille, p. 7–14.

    Google Scholar 

  • Escande, P., Béroule, D. and Blanchet, P. (1991) Speech recognition experiments with Guided Propagation. Proc. of IJCNN'91.

    Google Scholar 

  • Daniel, M.P., Carite, L. and Denis, M. (1994) Modes of linearization in the description of spatial configurations. In Portugali, J. (ed.), The construction of cognitive maps. Dordrecht: Kluwer, 297–318.

    Google Scholar 

  • Dowell, J., Shmueli, Y., and Salter, I. (1995) Applying a cognitive model of the user to the design of a multimodal speech interface. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh.

    Google Scholar 

  • Faure, C. and Julia, L. (1994) An agent-based architecture for a multimodal interface. Working notes of the AAAI symposium on Intelligent Multi-Media Multi-Modal Systems. March 21–23, Stanford.

    Google Scholar 

  • Foote, J.T., Brown, M.G., Jones, G.J.F., Sparck Jones, K., and Young, S.J. (1995) Video mail retrieval by voice: towards intelligent retrieval and browsing of multimedia documents. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh.

    Google Scholar 

  • Frohlich, D.M. (1991) The design space of interfaces. In L. Kjelldahl (ed.) Multimedia: principles, systems and applications. Berlin: Springer.

    Google Scholar 

  • GonÇalves, M.R. (1996) Working notes on itinerary descriptions. LIMSI, Orsay. http://www.limsi.fr/Individu/goncalve/index.html

    Google Scholar 

  • Hare, M., Doubleday, A., Bennett, I., and Ryan, M. (1995) Intelligent presentation of information retrieved from heterogeneous multimedia databases. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh

    Google Scholar 

  • Han, Y. and Zukerman, I. (1997) A cooperative approach for multimodal presentation planning. This volume.

    Google Scholar 

  • Huls, C. and Bos, E. (1997) Studies into full integration of language and action. This volume.

    Google Scholar 

  • Hurault-Plantet and Briffault (1996) Atelier de génie linguistique et visualisation graphique. http://www.limsi.fr/Individu/gs/GroupeLC/Outils.html

    Google Scholar 

  • Hutchins, E.L., Holland, J.D. and Norman, D.A. (1986) Direct manipulation interfaces. In Norman, D.A. and Draper, S.W. (eds.), User centred system design: new perspectives on human computer design. Hillsdale, NJ: Lawrence Erlbaum.

    Google Scholar 

  • Inder, R., Oberlander, J., and Tobin, R. (1995) Intelligent support for navigation in hypermedia: discourse structure and the Web. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interface: Research and Applications, University of Edinburgh.

    Google Scholar 

  • Jackendoff, R. (1987) On beyond zebra: the relation between linguistic and visual information. Cognition 26 (2), 89–114.

    Article  Google Scholar 

  • Lee, J. (ed.) (1995) Pre-Proceedings First International Workshop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications. University of Edinburgh.

    Google Scholar 

  • Mackinlay, J., Card, S.K. & Robertson, G.G. (1990) A Semantic Analysis of the Design Space of Input Devices. Human-Computer Interaction. vol. 5, no 2–3, pp. 145–190.

    Article  Google Scholar 

  • Martin, J.C. (1995) Coopérations entre modalités et liage par synchronie dans les interfaces multimodales. Ph.D. Thesis, TELECOM Paris. http://www.limsi.fr/Individu.martin

    Google Scholar 

  • Martin, J.C. (1996) Types et buts de coopération entre modalités dans les interfaces multimodales. Techniques et Science Informatiques 15, 10/1996, 1367–1397.

    Google Scholar 

  • Martin, J.C. (1997) Towards intelligent cooperation between modalities. The example of a system enabling multimodal interaction with a map. Proc. IJCAI'97 International Workshop on Intelligent Multimodal Systems, 63–69. http://www.limsi.fr:80/Individu/martin/ijcai/article.html

    Google Scholar 

  • Martin, J.C. and Béroule, D. (1993) Types et buts de coopérations entre modalités. In Proc. 5th Conf. on Human-Computer Interaction IHM'93, 17–22.

    Google Scholar 

  • Martin, J.C. and Béroule, D. (1995) Temporal codes within a typology of cooperation between modalities. Artificial Intelligence Review 9, 1–8.

    Google Scholar 

  • Maybury, M. (1991) Introduction. Intelligent multimedia interfaces. Cambridge, MA: AAAI Press.

    Google Scholar 

  • Nigay, L. and Coutaz, J. (1993) A design space for multimodal systems: concurrent processing and data fusion. Proc. of Interchi'93, 172–178.

    Google Scholar 

  • Nigay, L. and Coutaz, J. (1995) Multifeature systems: from HCI properties to software design. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh.

    Google Scholar 

  • O'Nuallain, S. and Smith, A.G. (1994) An investigation into the common semantics of language and vision. Artificial Intelligence Review 8 (2–3), 113–122.

    Article  Google Scholar 

  • Olivier, P. and Tsujii, J.I. (1994) Quantitative perceptual representation of prepositional semantics. Artificial Intelligence Review 8 (2–3).

    Google Scholar 

  • Roques, M. (1994) Dynamic Grammatical Representations in Guided Propagation Networks. In R. C. Carrasco and J. Oncina (eds.) Grammatical Inference and Applications, Lecture Notes in Artificial Intelligence 862, 189–202. Berlin: Springer.

    Chapter  Google Scholar 

  • Salisbury, M.W., Hendrickson, J.H., Lammers, T.L., Fu, C., and Moody, S.A. (1990) Talk and draw: bundling speech and graphics. IEEE Computer 23 (8), 59–65.

    Article  Google Scholar 

  • Santana, S. and Pineda, L.A. (1995) Producing coordinated natural language and graphical explanations in the context of a geometric problem-solving task. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh.

    Google Scholar 

  • Shastri, L. and Ajjanagadde, V. (1993) Prom simple associations to systematic reasoning: a connectionist representation of rules, variables and dynamic bindings using temporal synchrony. Behavioural and Brain Sciences, 16, 417–494.

    Article  Google Scholar 

  • Sims, R. and Hedberg, J. (1995) Dimensions of learner control: a reappraisal of interactive multimedia instruction. In: Lee, J. (1995) Pre-Proceedings First Int. Worskhop on Intelligence and Multimodality in Multimedia Interfaces: Research and Applications, University of Edinburgh.

    Google Scholar 

  • Siroux, J., Guyomard, M., Multon, F., and Remondeau, C. (1997) Modeling and processing of the oral and tactile activities in the Georal tactile system. This volume.

    Google Scholar 

  • Sowa, J. (1983) Conceptual Structures: Information Processing in Mind and Machine. Reading, MA: Addison-Wesley.

    Google Scholar 

  • Stern, R.M. (1995) Robust speech recognition. Section 14 in electronic book: Survey of the State of the Art in Human Language Technology. http://www.cse.ogi.edu/CSLU/HLTsurvey/ch1node6.html/

    Google Scholar 

  • Vaananen, K. (1995) Four pillars for improving the quality of multimedia applications. In Proc. First Int. Workshop on Evaluation Methods and Quality Criteria for Multimedia Applications, San Francisco.

    Google Scholar 

  • Vo, M. T. and Waibel, A. (1993) Multimodal Human-Computer Interaction. In Proc. International Symposium on Spoken Dialogue: New Directions in Human and Man-Machine Communication, Tokyo, 95–101.

    Google Scholar 

  • Veldman, R. (1995) Experiments on robust parsing in a multimodal Guided Propagation Network. LIMSI (ERASMUS) Report 95-11, Orsay

    Google Scholar 

  • Wahlster, W., André, E., Finkler, W., Profitlich, H.J., and Rist, T. (1991) Plan-based integration of natural language and graphics generation. AI Journal 63, 387–427.

    Google Scholar 

  • Wang, E., Shahnvaz, H., Hedman, L., Papadopoulos, K., and Watkinson, N. (1993) A usability evaluation of text and speech redundant help messages on a reader interface. In G. Salvendy & M. Smith (eds.), Human-Computer Interaction: Software and Hardware Interfaces, 724–729.

    Google Scholar 

  • Westerlund, P., Béroule, D. and Roques, M. (1994) Experiments of robust parsing using a Guided Propagation Network. In Proc. International Conference on New Methods in Language Processing (NEMLAP'94) Manchester.

    Google Scholar 

  • Webber, B. (1997) Instructing Animated Agents: Viewing Language in Behavioural Terms. This volume.

    Google Scholar 

  • Yankelovich, N., Levow, G., Marx, M. (1995) Designing Speech Acts: Issues in Speech User Interfaces. Proc. of CHI '95, Conference on Human Factors in Computing Systems.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. LIMSI-CNRS, B. P. 133, 9143, Orsay Cedex, France

    J. C. Martin, R. Veldman & D. Béroule

Authors
  1. J. C. Martin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. R. Veldman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. D. Béroule
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Harry Bunt Robbert-Jan Beun Tijn Borghuis

Rights and permissions

Reprints and permissions

Copyright information

© 1998 Springer-Verlag

About this paper

Cite this paper

Martin, J.C., Veldman, R., Béroule, D. (1998). Developing multimodal interfaces: A theoretical framework and guided propagation networks. In: Bunt, H., Beun, RJ., Borghuis, T. (eds) Multimodal Human-Computer Communication. CMC 1995. Lecture Notes in Computer Science, vol 1374. Springer, Berlin, Heidelberg. https://doi.org/10.1007/BFb0052318

Download citation

  • DOI: https://doi.org/10.1007/BFb0052318

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-64380-7

  • Online ISBN: 978-3-540-69764-0

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics