Exploiting Semantic Search and Object-Oriented Programming to Ease Multimodal Interface Development
Authors: 
Thibaut Septon, 
Santiago Villarreal-Narvaez, 
Xavier Devroey, Bruno DumasAuthors Info & Claims
Thibaut Septon, Santiago Villarreal-Narvaez, Xavier Devroey, Bruno DumasAuthors Info & ClaimsPages 74 - 80
Abstract
Multimodal interaction has been adopted across various platforms and devices, with supporting tools enhancing the developer experience in developing (). While traditionally, these tools faced challenges balancing expressiveness and usability, recent progress in tends to mitigate this rule. However, adding multimodal interaction still remains challenging, especially when integrating the voice modality, and s remain to be better integrated into today’s applications. To address these challenges, we introduce a Unity tool-based system named. allows developers to use their knowledge in Object-Oriented Programming to handle the expert knowledge required to create a while allowing end users great flexibility in the natural language they can use to interact. Our contributions are: 1) the presentation of ’s architecture and its inherent concepts along with its open-source implementation, and 2) a successful evaluation of its usability for describing s through the questionnaire with twelve participants.
References
[1]
Aaron Bangor, Philip Kortum, and James Miller. 2009. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies 4, 3 (2009), 114–123.
[2]
Richard A. Bolt. 1980. “Put-that-there”: Voice and gesture at the graphics interface. SIGGRAPH Comput. Graph. 14, 3, 262–270. https://doi.org/10.1145/965105.807503
[3]
Jullien Bouchet, Laurence Nigay, and Thierry Ganille. 2004. ICARE software components for rapidly developing multimodal interfaces. In Proceedings of the 6th international conference on Multimodal interfaces. 251–258.
[4]
Marie-Luce Bourguet. 2002. A toolkit for creating and testing multimodal interface designs. companion proceedings of UIST 2 (2002), 29–30.
[5]
John Brooke. 1996. Sus: a “quick and dirty’usability. Usability evaluation in industry 189, 3 (1996), 189–194.
[6]
Paul Chandler and John Sweller. 1992. The Split-Attention Effect as a Factor in the Design of Instruction. British Journal of Educational Psychology 62, 2 (1992), 233–246. https://doi.org/10.1111/j.2044-8279.1992.tb01017.x
[7]
G. Convertino, J. Chen, Y. Ryu, C. North, and B. Yost. 2003. Exploring Context Switching and Cognition in Dual-View Coordinated Visualizations. In International Conference on Coordinated and Multiple Views in Exploratory Visualization. IEEE, 55. https://doi.org/10.1109/CMV.2003.1215003
[8]
Joëlle Coutaz, Laurence Nigay, Daniel Salber, Ann Blandford, Jon May, and Richard M Young. 1995. Four easy pieces for assessing the usability of multimodal interaction: the CARE properties. Human—Computer Interaction: Interact’95 (1995), 115–120.
[9]
Fredy Cuenca, Jan Van den Bergh, Kris Luyten, and Karin Coninx. 2015. Hasselt uims: a tool for describing multimodal interactions with composite events. In Proceedings of the 7th ACM SIGCHI Symposium on Engineering Interactive Computing Systems. 226–229.
[10]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018).
[11]
Bruno Dumas, Denis Lalanne, and Rolf Ingold. 2008. Prototyping multimodal interfaces with the SMUIML modeling language. In CHI 2008 Workshop on User Interface Description Languages for Next Generation User Interfaces, CHI.
[12]
Bruno Dumas, Denis Lalanne, and Rolf Ingold. 2010. Description languages for multimodal interaction: a set of guidelines and its illustration with SMUIML. Journal on multimodal user interfaces 3 (2010), 237–247.
[13]
Bruno Dumas, Denis Lalanne, and Sharon Oviatt. 2009. Multimodal Interfaces: A Survey of Principles, Models and Frameworks. Springer Berlin Heidelberg, Berlin, Heidelberg, 3–26. https://doi.org/10.1007/978-3-642-00437-7_1
[14]
Bruno Dumas, Beat Signer, and Denis Lalanne. 2011. A graphical uidl editor for multimodal interaction design based on smuiml. (2011).
[15]
Jerome Fink, Pierre Poitier, Maxime André, Loup Meurice, Benoît Frénay, Anthony Cleve, Bruno Dumas, and Laurence Meurant. 2023. Sign Language to Text Dictionary with Lightweight Transformer Models. In Proceedings of the 32nd International Joint Conference on Artificial Intelligence (IJCAI 2023): AI for Social Good track.
[16]
Charles L Forgy. 1989. Rete: A fast algorithm for the many pattern/many object pattern match problem. In Readings in Artificial Intelligence and Databases. Elsevier, 547–559.
[17]
Lode Hoste, Bruno Dumas, and Beat Signer. 2011. Mudra: a unified multimodal interaction framework. In Proceedings of the 13th international conference on multimodal interfaces. 97–104.
[18]
Karina Li, Daniel Wan Rosli, Shuning Zhang, Yuhan Zhang, Monica S Lam, James A Landay, 2023. ReactGenie: An Object-Oriented State Abstraction for Complex Multimodal Interactions Using Large Language Models. arXiv preprint arXiv:2306.09649 (2023).
[19]
Vivian Genaro Motti. 2020. Wearable Interaction. Springer International Publishing. https://doi.org/10.1007/978-3-030-27111-4
[20]
Sharon Oviatt. 1999. Ten myths of multimodal interaction. Commun. ACM 42, 11 (1999), 74–81.
[21]
Sharon Oviatt. 2022. Multimodal Interaction, Interfaces, and Analytics. Springer International Publishing, Cham, 1–29. https://doi.org/10.1007/978-3-319-27648-9_22-1
[22]
Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for Computational Linguistics, Hong Kong, China, 3982–3992. https://doi.org/10.18653/v1/D19-1410
[23]
Ritam Jyoti Sarmah, Yunpeng Ding, Di Wang, Cheuk Yin Phipson Lee, Toby Jia-Jun Li, and Xiang’Anthony’ Chen. 2020. Geno: A Developer Tool for Authoring Multimodal Interaction on Existing Web Applications. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 1169–1181.
[24]
Jeff Sauro. 2011. A practical guide to the system usability scale: Background, benchmarks & best practices. Measuring Usability LLC.
[25]
Marcos Serrano, Laurence Nigay, Jean-Yves L Lawson, Andrew Ramsay, Roderick Murray-Smith, and Sebastian Denef. 2008. The openinterface framework: A tool for multimodal interaction. In CHI’08 Extended abstracts on human factors in computing systems. 3501–3506.
[26]
Jacqueline N Stetson and Thomas S Tullis. 2004. A comparison of questionnaires for assessing website usability. UPA Presentation (2004).
[27]
Matthew Turk. 2014. Multimodal interaction: A review. Pattern recognition letters 36 (2014), 189–195.
[28]
Minh Tue Vo and Cindy Wood. 1996. Building an application framework for speech and pen input integration in multimodal learning interfaces. In 1996 IEEE International Conference on Acoustics, Speech, and Signal Processing Conference Proceedings, Vol. 6. IEEE, 3545–3548.
Index Terms
- Exploiting Semantic Search and Object-Oriented Programming to Ease Multimodal Interface Development
Recommendations
A multimodal dialogue interface for mobile local search
IUI '13 Companion: Proceedings of the companion publication of the 2013 international conference on Intelligent user interfaces companionSpeak4itSM uses a multimodal interface to perform mobile search for local businesses. Users combine simultaneous speech and touch to input queries or commands, for example, by saying, "gas stations", while tracing a route on a touchscreen. This ...
A Wizard of Oz study for an AR multimodal interface
ICMI '08: Proceedings of the 10th international conference on Multimodal interfacesIn this paper we describe a Wizard of Oz (WOz) user study of an Augmented Reality (AR) interface that uses multimodal input (MMI) with natural hand interaction and speech commands. Our goal is to use a WOz study to help guide the creation of a ...
Multimodal slideshow: demonstration of the openinterface interaction development environment
ICMI '08: Proceedings of the 10th international conference on Multimodal interfacesIn this paper, we illustrate the OpenInterface Interaction Development Environment (OIDE) that addresses the design and development of multimodal interfaces. Multimodal interaction software development presents a particular challenge because of the ever ...
Comments
Information & Contributors
Information
Published In
June 2024
129 pages
ISBN:9798400706516
DOI:10.1145/3660515
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 24 June 2024
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Conference
EICS '24
Sponsor:
EICS '24: The 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems
June 24 - 28, 2024
Cagliari, Italy
Acceptance Rates
Overall Acceptance Rate 73 of 299 submissions, 24%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 35Total Downloads
- Downloads (Last 12 months)35
- Downloads (Last 6 weeks)5
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format