Skip to main content
SpringerLink
Log in

Multimodal interaction with virtual worlds XMMVR: eXtensible language for MultiModal interaction with virtual reality worlds

  • Original Paper
  • Published:
Journal on Multimodal User Interfaces Aims and scope Submit manuscript

Abstract

Based on a philosophy of integrating components from multimodal interaction applications with 3D graphical environments, reusing already defined markup language for describing graphics, graphical and spoken interactions based on the interactive movie metaphor, a markup language for modeling scenes, behavior and interaction is sought. With the definition of this language, we hope to have a common framework for developing applications that allow multimodal interaction at 3D stages. Thus we have defined the basis of an architecture that allows us to integrate the components of such multimodal interaction applications in 3D virtual environments.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28

Similar content being viewed by others

References

  1. Martin J-C (1998) Tycoon: theoretical and software tools for multimodal interfaces.In: Lee J (ed) Intelligence and Multimodality in Multimedia Interfaces. AAAI Press, California

  2. Barnett J, Dahl D, Kliche I, Tumuluri R, Yudkowsky M, Bodell M, Porter B, Raggett D, Raman T, Wahbe A (2008) Multimodal architecture and interfaces. http://www.w3.org/TR/mmi-arch/

  3. Dumas B, Lalanne D, Oviatt S (2009) Human machine interaction, In: ch. Multimodal interfaces: a survey of principles, models and frameworks. Springer, Berlin, Heidelberg pp 3–26

  4. Amditis A, Bimpas M, Blach R (2007) Towards networked and structured vr european research area: Intuition network of excellence and future research challenges. MMI Interakt Hum 1:43– 54

    Google Scholar 

  5. Salisbury MW, Hendrickson JH, Lammers TL, Fu C, Moody SA (1990) Talk and draw: bundling speech and graphics. Computer 23:59–65

    Article  Google Scholar 

  6. Mcglashan S, Axling T (1996) Talking to agents in virtual worlds. In: Proc of 3rd UK VR-SIG Conference

  7. Billinghurst M (1998) Put that where? Voice and gesture at the graphics interface. SIGGRAPH Comput Graph 32(4):60–63

    Article  Google Scholar 

  8. Kaplan RM (1975) On process models for sentence comprehension. Explorations in congnition, San Francisco

  9. Oviatt S, Cohen P (2000) Multimodal interfaces that process what comes naturally. Commun ACM 43:45–53

    Google Scholar 

  10. Gustafson J, Bell L, Beskow J, Boye J, Carlson R, Edlund J, Granström B, House D, Wirn M (2000) AdApt: a multimodal conversational dialogue system in an apartment domain. In: Proceedings of ICSLP 2000, pp 134–137

  11. Nugues P (2000) Verbal interactions in virtual worlds. CoRR, vol. cs.CL/0006027

  12. van Ballegooij A, Eliéns A (2001) Navigation by query in virtual worlds. In: Web3D 01: Proceedings of the sixth international conference on 3D Web technology, (New York, NY, USA), pp 77–83, ACM

  13. Corradini A, Mehta M, ole Bernsen N, Charfuelan M (2005) Animating an interactive conversational character for an educational game system. In: Proceedings of the 10th Int. Conf. on Intelligent User Interfaces, pp 183–190

  14. Kaiser E, Olwal A, McGee D, Benko H, Corradini A, Li X, Cohen P, Feiner S (2003) Mutual disambiguation of 3D multimodal interaction in augmented and virtual reality. In: Proceedings of the 5th international conference on Multimodal interfaces, ICMI ’03, (New York, NY, USA), pp 12–19, ACM

  15. Bouchet J, Nigay L (2004) Icare: a component-based approach for the design and development of multimodal interfaces. In: Dykstra-Erickson E, Tscheligi M (eds) CHI Extended Abstracts, pp 1325–1328, ACM

  16. Bouchet J ( 2004) Icare software components for rapidly developing multimodal interfaces. pp 251–258, ACM Press, Pennsylvania

  17. Serrano M, Nigay L, Lawson J-Y L, Ramsay A, Murray-Smith R, Denef S ( 2004) The openinterface framework: a tool for multimodal interaction. In: CHI ’08 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’08, (New York, NY, USA), pp 3501–3506, ACM

  18. Kipp M, Heloir A, Schröder M, Gebhard P (2010) Realizing multimodal behavior: closing the gap between behavior planning and embodied agent presentation. In: Proceedings of the 10th international conference on Intelligent virtual agents, IVA’10, pp 57–63, Springer, Berlin, Heidelberg

  19. Cerekovic A, Pejsa T, Pandzic IS (2009) Realactor: Character animation and multimodal behavior realization system. In: Proceedings of the 9th International Conference on Intelligent Virtual Agents, IVA ’09, pp 486–487, Springer, Berlin, Heidelberg

  20. van Welbergen H, Reidsma D, Ruttkay ZM, Zwiers J (2010) Elckerlyc-a bml realizer for continuous, multimodal interaction with a virtual human. J Multimodal User Interfaces 3:271–284

    Article  Google Scholar 

  21. Feuerstack S (2013) MINT Framework. http://www.multi-access.de/. Accessed 22 Feb 2014

  22. Mauro Dos Santos Anjo EP, Feuerstack S (2012) A real-time system to recognize static hand gestures of brazilian sign language (libras) alphabet using kinect. In: Proceedings of IHC 2012, the 6th Latin American Conference on Human-Computer Interaction, (Cuiabá/Mato Grosso, Brazil), Nov 5–9 (2012)

  23. Feuerstack S, Pizzolato E (2011) Building multimodal interfaces out of executable, model-based interactors and mappings. In: Proceedings of the HCI International 2011; 14th International Conference on Human-Computer Interaction; Human-Computer Interaction, Part I (J. Jacko, ed.), no. LNCS 6761, (Hilton Orlando Bonnet Creek, Orlando, Florida, USA), pp 221–228, Springer, Heidelberg (2011), Jul 9–14 (2011)

  24. Feuerstack S, Oliveira A, Araujo R (2011) Model-based design of interactions that can bridge realities the augmented drag-and-drop. In: Proceedings of the 13th Symposium on Virtual and Augmented Reality (SVR 2011), (Uberlândia, Minas Gerais, Brazil), 23–26 May 2011. ISSN 2177–676

  25. VoiceXML Forum, VoiceXML Forum: Voice eXtensible Markup Language (2010). http://www.voicexml.org/. Accessed 22 Feb 2014

  26. W3C, HTML5 (2013). http://www.w3.org/html/. Accessed 22 Feb 2014

  27. Microsoft, Speech Application Language Tags (SALT) (2003). http://msdn.microsoft.com/en-us/library/ms994629.aspx. Accessed 22 Feb 2014

  28. Axelsson J et al. (2010) XHTML+Voice Profile. http://www.w3.org/TR/xhtml+voice/. Accessed 22 Feb 2014

  29. Hartman J, Wernecke J (1996) The VRML 2.0 handbook: building moving worlds on the web. Addison-Wesley, California

  30. Web3D Consortium, X3D (2010). http://www.web3d.org/x3d. Accessed 22 Feb 2014

  31. Dachselt R, Rukzio E (2003) BEHAVIOR3D: An XML-Based Framework for 3D Graphics Behavior. In: Web3D 03: Proceeding of the eighth international conference on 3D Web technology, ACM, p 101 (Press)

  32. W3C, XMLSchema (2014). http://www.w3.org/XML/Schema#Tools

  33. Szabó K (1995) Metaphors and the user interface. http://www.katalinszabo.com/metaphor.htm. Accessed 22 Feb 2014

  34. Erickson TD (1995) Human-computer interaction. In: Working with interface metaphors. Morgan Kaufmann Publishers Inc, San Francisco, CA, USA pp 147–151

  35. CERN, Twenty years of a free, open web (2013). http://info.cern.ch/. Accessed 22 Feb 2014

  36. Dachselt R (2000) Action spaces: a metaphorical concept to support navigation and interaction in 3d interfaces. In: Workshop Usability Centred Design and Evaluation of Virtual 3D Environments

  37. Mcglashan S, Axling T (1996) A speech interface to virtual environments. In: International Workshop SPEECH AND COMPUTER

  38. Bolt RA (1980) Put-that-there: voice and gesture at the graphics interface. In: SIGGRAPH 80: Proceedings of the 7th annual conference on Computer graphics and interactive techniques, ACM, New York, NY, USA, pp 262–270

  39. Megginson D (2010) SAX. http://www.saxproject.org/. Accessed 22 Feb 2014

  40. Verbio, Verbio (2012). http://www.verbio.com/. Accessed 22 Feb 2014

  41. Cortona, CORTONA 3D (2010). http://www.cortona3d.com/. Accessed 22 Feb 2014

  42. Phelps AM (2010) Introduction to the External Authoring Interface, EAI. http://andysgi.rit.edu/andyworld10/gallery/archives/vrml/media/eaiclass.doc. Accessed 22 Feb 2014

  43. Microsoft, Microsoft Java Virtual Machine (2012). http://support.microsoft.com/gp/lifean12. Accessed 22 Feb 2014

  44. Adobe, Adobe Shockwave (2010). http://www.adobe.com/products/shockwaveplayer/. Accessed 22 Feb 2014

  45. Gabbard JL, Hix D, Swan JE (1999) User-centered design and evaluation of virtual environments. IEEE Comput Graph Appl 19:51–59

    Article  Google Scholar 

  46. Lund AM (2001) Measuring usability with the use questionnaire

  47. GestureML, GestureML (2013). http://www.gestureml.org/. Accessed 22 Feb 2014

  48. Microsoft, Kinect (2013). http://www.microsoft.com/en-us/kinectforwindows/. Accessed 22 Feb 2014

  49. Arduino (2012). http://www.arduino.cc/es/. Accessed 22 Feb 2014

  50. MYO (2013) Myo. https://www.thalmic.com/myo/. Accessed 22 Feb 2014

  51. Motion L (2013) Leap Motion. https://www.leapmotion.com/. Accessed 22 Feb 2014

  52. WebGL (2013) WebGL. http://www.khronos.org/webgl/. Accessed 22 Feb 2014

  53. X3DOM (2013) X3DOM. http://www.x3dom.org/. Accessed 22 Feb 2014

  54. JVoiceXML (2013) JVoiceXML. http://jvoicexml.sourceforge.net/. Accessed 22 Feb 2014

  55. ARML (2013) ARML. http://openarml.org/. Accessed 22 Feb 2014

  56. Bevan N (2008) Classifying and selecting UX and usability measures. In: Proceedings of Meaningful Measures: Valid Useful User Experience Measurement (VUUM), 5th COST294-MAUSE Open Workshop

  57. de Lera E, Garreta-Domingo M (2007) Ten emotion heuristics: Guidelines for assessing the user’s affective dimension easily and cost-effectively. In: Proceedings of the 21st British HCI Group Annual Conference on People and Computers: HCI...But Not As We Know It, vol 2, BCS-HCI ’07, British Computer Society, Swinton, UK, UK, pp 163–166

  58. Desmet PMA, Overbeeke CJ, Tax SJET (2001) Designing products with added emotional value: development and application of an approach for research through design. Design J 4(1):32–47

    Article  Google Scholar 

Download references

Acknowledgments

This article and the doctoral thesis that is based on, have been developed as part of the research line of multimodal interaction systems developed by ECA-SIMM research group from the Computer Science Department (ATC, CCIA, LSI) at University of Valladolid (Spain) in relation to projects ACME (VA077A08) and ARACNOS (VA053A05) funded by Junta de Castilla y León.

Author information

Authors and Affiliations

  1. Universidad de Valladolid, Campus Miguel Delibes s/n, 47011, Valladolid, Spain

    Hector Olmedo, David Escudero & Valentín Cardeñoso

Authors
  1. Hector Olmedo
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David Escudero
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Valentín Cardeñoso
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hector Olmedo.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Olmedo, H., Escudero, D. & Cardeñoso, V. Multimodal interaction with virtual worlds XMMVR: eXtensible language for MultiModal interaction with virtual reality worlds. J Multimodal User Interfaces 9, 153–172 (2015). https://doi.org/10.1007/s12193-015-0176-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12193-015-0176-5

Keywords

Search

Navigation