Abstract
Speech interfaces are becoming more and more popular as a means to interact with virtual environments but the development and integration of these interfaces is usually still ad hoc, especially the speech grammar creation of the speech interface is a process commonly performed by hand. In this paper, we introduce an approach to automatically generate a speech grammar which is generated using semantic information. The semantic information is represented through ontologies and gathered from the conceptual modelling phase of the virtual environment application. The utterances of the user will be resolved using queries onto these ontologies such that the meaning of the utterance can be resolved. For validation purposes we augmented a city park designer with our approach. Informal tests validate our approach, because they reveal that users mainly use words represented in the semantic data, and therefore also words which are incorporated in the automatically generated speech grammar.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Cohen, P.R.: The role of natural language in a multimodal interface. In: Proceedings of the Fifth ACM Symposium on User Interface Software and Technology, Monteray, CA, USA, pp. 143–149 (1992)
Cohen, P.R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., Chen, L., Clow, J.: Quickset: multimodal interaction for simulation set-up and control. In: Proceedings of the Fifth Conference on Applied Natural Language Processing, pp. 20–24. Morgan Kaufmann, San Francisco (1997). http://dx.doi.org/10.3115/974557.974562
Coninx, K., De Troyer, O., Raymaekers, C., Kleinermann, F.: VR-DeMo: a tool-supported approach facilitating flexible development of virtual environments using conceptual modelling. In: Virtual Concept 2006 (VC 06), Cancun, Mexico (2006)
Conti, G., Ucelli, G., De Amicis, R.: “Verba volant scripta manent” a false axiom within virtual environments. A semi-automatic tool for retrieval of semantics understanding for speech-enabled vr applications. Comput. Graph. 30(4), 619–628 (2006)
Corradini, A., Cohen, P.: On the relationships among speech, gestures, and object manipulation in virtual environments: Initial evidence. In: Proceedings of the International CLASS Workshop on Natural, Intelligent and Effective Interaction in Multimodal Dialogue Systems (2002)
Cuppens, E., Coninx, K.: Cogenive: Code generation for interactive virtual environments. In: The Future of User Interface Design Tools, Workshop of ACM Conference on Human Factors in Computing Systems (CHI 2005), Portland, United States (2005)
Gorniak, P., Roy, D.: Probabilistic grounding of situated speech using plan recognition and reference resolution. In: ICMI ’05: Proceedings of the 7th International Conference on Multimodal Interfaces, pp. 138–143 (2005)
Goubran, R.A., Wood, C.: Building an application framework for speech and pen input integration in multimodal learning interfaces. In: ICASSP ’96: Proceedings of the Acoustics, Speech, and Signal Processing Conference Proceedings, IEEE International Conference, pp. 3545–3548. IEEE Computer Society, Washington (1996). http://dx.doi.org/10.1109/ICASSP.1996.550794
Irawati, S., Calderón, D., Ko, H.: Semantic 3d object manipulation using object ontology in multimodal interaction framework. In: Proceedings of the 2005 International Conference on Augmented Tele-Existence, pp. 35–39 (2005)
Irawati, S., Calderón, D., Ko, H.: Spatial ontology for semantic integration in 3d multimodal interaction framework. In: ACM International Conference on Virtual Reality Continuum and Its Applications VRCIA, pp. 129–135 (2006)
Kaiser, E., Olwal, A., McGee, D., Benko, H., Corradini, A., Li, X., Cohen, P., Feiner, S.: Mutual disambiguation of 3d multimodal interaction in augmented and virtual reality. In: ICMI ’03: Proceedings of the 5th International Conference on Multimodal Interfaces, pp. 12–19 (2003)
Martínez, J.I.: An intelligent guide for virtual environments with fuzzy queries and flexible management of stories. Ph.D. thesis, Universidad de Murcia (2004)
Cernak, M., Sannier, A.: Command speech interface to virtual reality applications. Technical Report, Virtual Reality Applications Center at Iowa State University of Science and Technology (2002)
McGlashan, S.: Speech interfaces to virtual reality. In: Proceedings of 2nd International Workshop on Military Applications of Synthetic Environments and Virtual Reality (1995)
Muller, J., Krapichler, C., Nguyen, L.S., Hans Englmeier, K., Lang, M.: Speech interaction in virtual reality. In: Roceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 3757–3760 (1998)
Otto, K.A.: The semantics of multi-user virtual environments. In: Workshop towards Semantic Virtual Environments (SVE 2005), pp. 35–39 (2005)
OWL Web Ontology Language: OWL. http://www.w3.org/TR/owl-features/. January 2008
Pfeiffer, T., Latoschik, M.E.: Resolving object references in multimodal dialogues for immersive virtual environments. In: Proceedings of the IEEE VR2004, Chicago, USA, pp. 35–42 (2004)
Resource Description Framework (RDF): RDF. http://www.w3.org/RDF/. January 2008
Sharma, R., Zeller, M., Pavlovic, V.I., Huang, T.S., Lo, Z., Chu, S., Zhao, Y., Phillips, J.C., Schulten, K.: Speech/gesture interface to a visual-computing environment. IEEE Comput. Graph. Appl. 20(2), 29–37 (2000)
SPARQL Query Language for RDF: SPARQL. http://www.w3.org/TR/rdf-sparql-query/. January 2008
W2P: http://research.edm.uhasselt.be/w2p/. January 2008
WordNet: http://wordnet.princeton.edu/. January 2008
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Vanacken, L., Raymaekers, C. & Coninx, K. Automatic speech grammar generation during conceptual modelling of virtual environments. TVC 24, 955–961 (2008). https://doi.org/10.1007/s00371-008-0276-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-008-0276-2