Álan L.V. Guedes
Roberto G. de Albuquerque Azevedo
Simone D.J. Barbosa
ABSTRACT
Recent advances in technologies for speech, touch and gesture recognition have given rise to a new class of user interfaces that does not only explore multiple modalities but also allows for multiple interacting users. Even so, current declarative multimedia languages e.g. HTML, SMIL, and NCL?support only limited forms of user input (mainly keyboard and mouse) for a single user. In this paper, we aim at studying how the NCL multimedia language could take advantage of those new recognition technologies. To do so, we revisit the model behind NCL, named NCM (Nested Context Model), and extend it with first-class concepts supporting multiuser and multimodal features. To evaluate our approach, we instantiate the proposal and discuss some usage scenarios, developed as NCL applications with our extended features.
- Azevedo, R.G.D.A. and Soares, L.F.G. 2012. Embedding 3D Objects into NCL Multimedia Presentations. Proceedings of the 17th International Conference on 3D Web Technology (New York, NY, USA, 2012), 143--151. Google Scholar
Digital Library
- Batista, C.E.C.F. et al. 2010. Estendendo o uso das classes de dispositivos Ginga-NCL. WebMedia '10: Proceedings of the 16th Brazilian Symposium on Multimedia and the Web (2010).Google Scholar
- Batista, C.E.C.F. 2013. GINGA-MD?: Uma Plataforma para Suporte à Execução de Aplicações Hipermídia. Pontifícia Universidade Católica do Rio de Janeiro.Google Scholar
- Bolt, R.A. 1998. "Put-that-there": voice and gesture at the graphics interface. Readings in Intelligent User Interfaces. M.T. Maybury and W. Wahlster, eds. Morgan Kaufmann Publishers Inc. 19--28. Google ScholarDigital Library
- Bulterman, D.C.A. and Rutledge, L.W. 2008. SMIL 3.0: Flexible Multimedia for Web, Mobile Devices and Daisy Talking Books. Springer Publishing Company, Incorporated. Google ScholarDigital Library
- Carvalho, L. and Macedo, H. 2010. Estendendo a NCL para Promover Interatividade Vocal em Aplicações Ginga na TVDi Brasileira. WebMedia '10: Proceedings of the 16th Brazilian Symposium on Multimedia and the Web (2010).Google Scholar
- Carvalho, L.A.M.C. et al. 2008. Architectures for Interactive Vocal Environment to Brazilian Digital TV Middleware. Proceedings of the 2008 Euro American Conference on Telematics and Information Systems (New York, NY, USA, 2008), 22:1--22:8. Google ScholarDigital Library
- Elmqvist, N. 2011. Distributed User Interfaces: State of the Art. Distributed User Interfaces. J.A. Gallud et al., eds. Springer London. 1--12.Google Scholar
- Ghinea, G. et al. 2014. Mulsemedia: State of the Art, Perspectives, and Challenges. ACM Transactions on Multimedia Computing, Communications, and Applications. 11, 1s (Oct. 2014), 17:1--17:23. Google ScholarDigital Library
- Google (n.d.). Supporting Multiple Game Controllers | Android Developers. https://developer.android.com/intl/pt-br/training/gamecontrollers/multiple-controllers.html. Accessed on 2016-05-02.Google Scholar
- Guedes, A.L.V. et al. 2015. Specification of Multimodal Interactions in NCL. Proceedings of the 21st Brazilian Symposium on Multimedia and the Web (2015), 181--187. Google ScholarDigital Library
- Guerrero Garcia, J. et al. 2010. Designing workflow user interfaces with UsiXML. 1st Int. Workshop on User Interface eXtensible Markup Language UsiXML'2010 (2010).Google Scholar
- Haber, C. 2001. Modeling Multiuser Interactions. Proceedings at the First European Computer Supported Collaborative Learning Conference, Maastricht, Germany (2001), 22--24.Google Scholar
- Hachaj, T. and Ogiela, M.R. 2012. Semantic Description and Recognition of Human Body Poses and Movement Sequences with Gesture Description Language. Computer Applications for Biotechnology, Multimedia, and Ubiquitous City. T. Kim et al., eds. Springer Berlin Heidelberg. 1--8.Google Scholar
- ISO/IEC 2013. ISO/IEC 23005--3:2013 Information Technology - Media Context and Control - Part 3: Sensory Information. www.iso.org/iso/home/store/catalogue_ics/catalogue_detail_ics.htm?csnumber=60391. Accessed on 2016-08-08.Google Scholar
- Laurence Nigay and Coutaz, J. 1997. Multifeature Systems: The CARE Properties and Their Impact on Software Design. Multimedia Interfaces: Research and Applications, chapter 9 (1997).Google Scholar
- Lazar, J. et al. 2010. Research Methods in Human-Computer Interaction. Wiley Publishing. Google ScholarDigital Library
- Limbourg, Q. et al. 2004. USIXML: A User Interface Description Language Supporting Multiple Levels of Independence. ICWE Workshops (2004), 325--338.Google Scholar
- Microsoft (n.d.). Getting Started With XInput. https://msdn.microsoft.com/enus/library/windows/desktop/ee417001#multiple_controllers. Accessed on 2016-08-02.Google Scholar
- Müller-Tomfelde, C. and Fjeld, M. 2010. Introduction: A Short History of Tabletop Research, Technologies, and Products. Tabletops - Horizontal Interactive Displays. C. Müller-Tomfelde, ed. Springer London. 1--24.Google ScholarCross Ref
- OpenMobileAlliance 2001. WAG UAProf. http://www.openmobilealliance.org/Technical/wapindex.aspx. Accessed on 2016-08-02.Google Scholar
- Oviatt, S. 2007. Multimodal Interfaces. The Human-Computer Interaction Handbook. CRC Press. 413--432.Google Scholar
- Sant'Anna, F. et al. 2008. NCLua: Objetos Imperativos Lua Na Linguagem Declarativa NCL. Proceedings of the 14th Brazilian Symposium on Multimedia and the Web (New York, NY, USA, 2008), 83--90. Google ScholarDigital Library
- Soares, L.F.G. et al. 2010. Ginga-NCL: Declarative Middleware for Multimedia IPTV Services. IEEE Communications Magazine. 48, June (Jun. 2010), 74--81. Google ScholarDigital Library
- Soares, L.F.G. et al. 2009. Multiple Exhibition Devices in DTV Systems. Proceedings of the 17th ACM International Conference on Multimedia (New York, NY, USA, 2009), 281--290. Google ScholarDigital Library
- Soares, L.F.G. 2009. Nested Context Model 3.0: Part 1 -- NCM Core. ftp://obaluae.inf.puc-rio.br/pub/docs/techreports/05_18_soares.pdf. Accessed on 2016-08-02.Google Scholar
- Stefik, M. et al. 1987. WYSIWIS Revised: Early Experiences with Multiuser Interfaces.ACM Trans. Inf. Syst. 5, 2 (Apr. 1987), 147--167. Google ScholarDigital Library
- Turk, M. 2014. Multimodal interaction: A review. Pattern Recognition Letters. 36, (2014), 189--195. Google ScholarDigital Library
- Vilhjálmsson, H. et al. 2007. The Behavior Markup Language: Recent Developments and Challenges. Intelligent Virtual Agents. C. Pelachaud et al., eds. Springer Berlin Heidelberg. 99--111. Google ScholarDigital Library
- W3C 2014. FOAF Vocabulary Specification. http://xmlns.com/foaf/spec/. Accessed on 2016-08-02.Google Scholar
- W3C 2011. Ink Markup Language (InkML). http://www.w3.org/TR/2011/REC-InkML-20110920/. Accessed on 2016-08-02.Google Scholar
- W3C 2012. Multimodal Architecture and Interfaces. http://www.w3.org/TR/mmi-arch/. Accessed on 2016-08-02.Google Scholar
- W3C 2014. RDF/XML Syntax Specification. https://www.w3.org/TR/REC-rdf-syntax/. Accessed on 2016-08-02.Google Scholar
- W3C 2008. SPARQL Query Language for RDF. https://www.w3.org/TR/rdf-sparql-query/. Accessed on 2016-08-02.Google Scholar
- W3C 2004. Speech Recognition Grammar Specification Version 1.0. http://www.w3.org/TR/speech-grammar/. Accessed on 2016-08-02.Google Scholar
- W3C 2010. Speech Synthesis Markup Language (SSML) Version 1.1. http://www.w3.org/TR/speech-synthesis11/. Accessed on 2016-08-02.Google Scholar
- W3C 2012. State Chart XML (SCXML): State Machine Notation for Control Abstraction. http://www.w3.org/TR/scxml/. Accessed on 2016-08-02.Google Scholar
- W3C 2001. XHTML+Voice Profile 1.0. http://www.w3.org/TR/xhtml+voice/. Accessed on 2016-08-02.Google Scholar
Index Terms
- Extending NCL to Support Multiuser and Multimodal Interactions
Recommendations
Extending Ginga-NCL to Specify Multimodal Interactions With Multiple Users
WebMedia '20: Proceedings of the Brazilian Symposium on Multimedia and the WebCurrently, most declarative languages used to define multimedia documents do not support the specification of an interactive application with multiple users and multimodal interaction. To handle multimodal and multi-user interaction in these languages, ...
Specification of Multimodal Interactions in NCL
WebMedia '15: Proceedings of the 21st Brazilian Symposium on Multimedia and the WebThis paper proposes an approach to integrate multimodal events--both user-generated, e.g., audio recognizer, motion sensors; and user-consumed, e.g., speech synthesizer, haptic synthesizer--into programming languages for the declarative specification of ...
Extending multimedia languages to support multimodal user interactions
Historically, the Multimedia community research has focused on output modalities, through studies on timing and multimedia processing. The Multimodal Interaction community, on the other hand, has focused on user-generated modalities, through studies on ...
Comments