Abstract
We will show how to build innovative multimodal dialog user interfaces that integrate multiple heterogeneous web services as data sources on the basis of the Ontology-based Dialog Platform (ODP). More specifically, we will describe how to exploit ODP’s well-defined extension points and how generic ODP processing modules can be adopted, in order to support a rapid dialog system engineering process. By means of the latest ODP-based educational information system CIRIUS and the ODP workbench, a set of Eclipse-based editors and tools, we demonstrate step-by-step along the generic multimodal dialog processing chain what has to be done for developing a new multimodal dialog user interface for a specific application domain.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
There is also a W3C working group examining this area; see http://www.w3.org/2011/mbui/
- 2.
Partial dialog acts have to be resolved by the input fusion component.
- 3.
References
P. Baggia, D.C. Burnett, J. Carter, D.A. Dahl, G. McCobb, D. Raggett, EMMA: Extensible MultiModal Annotation Markup Language – W3C Recommendation (Feb 2009), http://www.w3.org/TR/emma/
S. Bergweiler, Interactive service composition and query, in Towards the Internet of Services: The THESEUS Research Program, ed. by W. Wahlster, H.J. Grallert, S. Wess, H. Friedrich, T. Widenka (Springer, Berlin/Heidelberg/New York, 2014)
S. Bergweiler, M. Deru, D. Porta, Integrating a multitouch Kiosk system with mobile devices and multimodal interaction, in Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (ITS ’10), Saarbrücken (ACM, New York, 2010), pp. 245–246, http://doi.acm.org/10.1145/1936652.1936698
H. Bunt, J. Alexandersson, J. Carletta, J.W. Choe, A.C. Fang, K. Hasida, K. Lee, V. Petukhova, A. Popescu-Belis, L. Romary, C. Soria, D. Traum, Towards an ISO standard for dialogue act annotation, in Proceedings of the Seventh Conference on International Language Resources and Evaluation (LREC ’10), Valletta, ed. by N. Calzolari, K. Choukri, B. Maegaard, J. Mariani, J. Odijk, S. Piperidis, M. Rosner, D. Tapias (European Language Resources Association (ELRA), 2010), http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.178.9209
D.C. Burnett, M.R. Walker, A. Hunt, Speech synthesis markup language (SSML) version 1.0 – W3C recommendation (Sept 2004), http://www.w3.org/TR/speech-synthesis/
G. Calvary, J. Coutaz, D. Thevenin, Q. Limbourg, L. Bouillon, J. Vanderdonckt, A unifying reference framework for multi-target user interfaces. Interact. Comput. 15(3), 289–308 (2003), http://dblp.uni-trier.de/db/journals/iwc/iwc15.html#CalvaryCTLBV03
R.L. Carpenter, The Logic of Typed Feature Structures: With Applications to Unification Grammars, Logic Programs and Constraint Resolution. Volume 32 of Cambridge Tracts in Theoretical Computer Science (Cambridge University Press, Cambridge, UK, 1992)
G. Di Fabbrizio, T. Okken, J.G. Wilpon, A speech mashup framework for multimodal mobile services, in Proceedings of the 11th International Conference on Multimodal Interfaces and the 6th Workshop on Machine Learning for Multimodal Interfaces (ICMI-MLMI ’09), Cambridge, ed. by J.L. Crowley, Y. Ivanov, C.R. Wren, D. Gatica-Perez, M. Johnston, R. Stiefelhagen (ACM, New York, 2009), pp. 71–78, http://dblp.uni-trier.de/db/conf/icmi/icmi2009.html#FabbrizioOW09
D. Ertl, Semi-automatic multimodal user interface generation, in Proceedings of the 1st ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS ’09), Pittsburgh, ed. by T.C.N. Graham, G. Calvary, P.D. Gray (ACM, New York, 2009), pp. 321–324, http://dblp.uni-trier.de/db/conf/eics/eics2009.html#Ertl09
R.T. Fielding, Architectural styles and the design of network-based software architectures. Doctoral dissertation, University of California, 2000, http://www.ics.uci.edu/~fielding/pubs/dissertation/top.htm
S. Gandhe, N. Whitman, D. Traum, R. Artstein, An integrated authoring tool for tactical questioning dialogue systems, in Proceedings of the 6th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems (Association for the Advancement of Artificial Intelligence (AAAI), Pasadena, California, 2009), http://www.ida.liu.se/~arnjo/Ijcai09ws/
A. Gruenstein, I. McGraw, I. Badr, The WAMI toolkit for developing, deploying, and evaluating web-accessible multimodal interfaces, in Proceedings of the 10th International Conference on Multimodal Interfaces (ICMI ’08), Chania, ed. by V. Digalakis, A. Potamianos, M. Turk, R. Pieraccini, Y. Ivanov (ACM, New York, 2008), pp. 141–148, http://dblp.uni-trier.de/db/conf/icmi/icmi2008.html#GruensteinMB08
M. Heinrich, M. Winkler, H. Steidelmüller, M. Zabelt, A. Behring, R. Neumerkel, A. Strunk, MDA applied: a task-model driven tool chain for multimodal applications, in Task Models and Diagrams for User Interface Design, ed. by M. Winckler, H. Johnson, P. Palanque. Volume 4849 of Lecture Notes in Computer Science (Springer, Berlin/Heidelberg/New York, 2007), pp. 15–27, http://portal.acm.org/citation.cfm?id=1782434.1782439
G. Herzog, A. Ndiaye, Building multimodal dialogue applications: system integration in smartkom, in SmartKom: Foundations of Multimodal Dialogue Systems, ed. by W. Wahlster. Cognitive Technologies (Springer, Berlin/Heidelberg/New York, 2006), pp. 439–452, http://dblp.uni-trier.de/db/series/cogtech/54023732.html#HerzogN06
A. Hunt, S. McGlashan, Speech recognition grammar specification version 1.0 – W3C recommendation (Mar 2004), http://www.w3.org/TR/speech-grammar/
H. Kett, M. Winkler, K. Kadner, Integrated service engineering (ISE), in Towards the Internet of Services: The THESEUS Research Program, ed. by W. Wahlster, H.J. Grallert, S. Wess, H. Friedrich, T. Widenka (Springer, Berlin/Heidelberg/New York, 2014)
Q. Limbourg, J. Vanderdonckt, B. Michotte, L. Bouillon, V. López-Jaquero, USIXML: a language supporting multi-path development of user interfaces, in Engineering Human Computer Interaction and Interactive Systems, ed. by R. Bastide, P.A. Palanque, J. Roth. Volume 3425 of Lecture Notes in Computer Science (Springer, Berlin/Heidelberg/New York, 2005), pp. 200–220
M. Löckelt, M. Deru, C.H. Schulz, S. Bergweiler, T. Becker, N. Reithinger, A unified approach for semantic-based multimodal interaction, in Towards the Internet of Services: The THESEUS Research Program, ed. by W. Wahlster, H.J. Grallert, S. Wess, H. Friedrich, T. Widenka (Springer, Berlin/Heidelberg/New York, 2014)
G. Mori, F. Paterno, C. Santoro, Design and development of multidevice user interfaces through multiple logical descriptions. IEEE Trans. Softw. Eng. 30, 507–520 (2004), http://portal.acm.org/citation.cfm?id=1018383
R. Neßelrath, D. Porta, Rapid development of multimodal dialogue applications with semantic models, in Proceedings of the 7th IJCAI Workshop on Knowledge and Reasoning in Practical Dialogue Systems (KRPD ’11), Barcelona, July 2011
F. Paterno, C. Santoro, J. Mantyjarvi, G. Mori, S. Sansone, Authoring pervasive multimodal user interfaces. Int. J. Web Eng. Technol. 4, 235–261 (2008), http://portal.acm.org/citation.cfm?id=1366965.1366970
F. Paterno, C. Santoro, L.D. Spano, MARIA: a universal, declarative, multiple abstraction-level language for service-oriented applications in ubiquitous environments, in ACM Transactions on Computer-Human Interaction (TOCHI), vol. 16 (ACM, New York, 2009), pp. 1–30
D. Porta, Towards model-driven development of mobile multimodal user interfaces for services, in Informatik 2010: Service Science – Neue Perspektiven für die Informatik, Beiträge der 40. Jahrestagung der Gesellschaft für Informatik, ed. by K.P. Fáhnrich, B. Franczyk. Volume 175 of Lecture Notes in Informatics (2010), pp. 497–502, http://dblp.uni-trier.de/db/conf/gi/gi2010-1.html#Porta10
D. Porta, D. Sonntag, R. Nesselrath, A multimodal mobile B2B dialogue interface on the iPhone, in Proceedings of the 4th Workshop on Speech in Mobile and Pervasive Environments (SiMPLE ’09) in Conjunction with Mobile (HCI ’09). (ACM, Bonn, Germany, 2009), http://www.dfki.de/web/forschung/publikationen?pubid=4177
J. Schehl, A. Pfalzgraf, N. Pfleger, J. Steigner, The BabbleTunes system: talk to your iPod! in Proceedings of the 10th International Conference on Multimodal Interfaces (ICMI ’08), Chania (ACM, New York, 2008), pp. 77–80, http://doi.acm.org/10.1145/1452392.1452408
D. Sonntag, M. Deru, S. Bergweiler, Design and implementation of combined mobile and touchscreen-based multimodal web 3.0 interfaces, in Proceedings of the International Conference on Artificial Intelligence (ICAI ’09) (Pasadena, California, 2009)
D. Sonntag, R. Engel, G. Herzog, A. Pfalzgraf, N. Pfleger, M. Romanelli, N. Reithinger, SmartWeb handheld – multimodal interaction with ontological knowledge bases and semantic web services, in Artifical Intelligence for Human Computing, ed. by T.S. Huang, A. Nijholt, M. Pantic, A. Pentland. Volume 4451 of Lecture Notes in Computer Science (Springer, Berlin/Heidelberg/New York, 2007), pp. 272–295
D. Sonntag, M. Möller, A multimodal dialogue mashup for medical image semantics, in Proceedings of the 15th International Conference on Intelligent User Interfaces (IUI ’10), Hong Kong (ACM, New York, 2010), pp. 381–384, http://doi.acm.org/10.1145/1719970.1720036
D. Sonntag, N. Reithinger, G. Herzog, T. Becker, A discourse and dialogue infrastructure for industrial dissemination, in Spoken Dialogue Systems for Ambient Environments: Second International Workshop (IWSDS 2010), Gotemba, 1–2 Oct 2010, ed. by G.G. Lee, J. Mariani, W. Minker, S. Nakamura. Volume 6392 of Lecture Notes in Artificial Intelligence (Springer, Berlin/Heidelberg/New York, 2010a), pp. 132–143, http://www.springerlink.com/content/5149m52mt5378316/
D. Sonntag, C. Weihrauch, O. Jacobs, D. Porta, THESEUS CTC-WP4 usability guidelines for use case applications, Technical report, DFKI GmbH, BMWi (Apr 2010b), http://www.dfki.de/web/forschung/publikationen?pubid=4788
A. Stanciulescu, Q. Limbourg, J. Vanderdonckt, B. Michotte, F. Montero, A transformational approach for multimodal web user interfaces based on UsiXML, in Proceedings of the 7th International Conference on Multimodal interfaces (ICMI ’05), Trento (ACM, New York, 2005), pp. 259–266, http://dblp.uni-trier.de/db/conf/icmi/icmi2005.html#StanciulescuLVMM05
W. Wahlster, Towards symmetric multimodality: fusion and fission of speech, gesture, and facial expression, in KI 2003: Advances in Artificial Intelligence, ed. by A. Günter, R. Kruse, B. Neumann. Volume 2821 of Lecture Notes in Computer Science (Springer, Berlin/Heidelberg/New York, 2003), pp. 1–18
M.H. Weik, Computer Science and Communications Dictionary (Kluwer, Boston, 2001), http://www.springer.com/computer/swe/book/978-0-7923-8425-0
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this chapter
Cite this chapter
Porta, D., Deru, M., Bergweiler, S., Herzog, G., Poller, P. (2014). Building Multimodal Dialog User Interfaces in the Context of the Internet of Services. In: Wahlster, W., Grallert, HJ., Wess, S., Friedrich, H., Widenka, T. (eds) Towards the Internet of Services: The THESEUS Research Program. Cognitive Technologies. Springer, Cham. https://doi.org/10.1007/978-3-319-06755-1_12
Download citation
DOI: https://doi.org/10.1007/978-3-319-06755-1_12
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-06754-4
Online ISBN: 978-3-319-06755-1
eBook Packages: Computer ScienceComputer Science (R0)