Abstract
This paper presents a model-based approach for designing Polymodal Menus, a new type of multimodal adaptive menu for small screen graphical user interfaces where item selection and adaptivity are responsive to more than one interaction modality: a menu item can be selected graphically, tactilely, vocally, gesturally, or any combination of them. The prediction window containing the most predicted menu items by assignment, equivalence, or redundancy is made equally adaptive. For this purpose, an adaptive menu model maintains the most predictable menu items according to various prediction methods. This model is exploited throughout various steps defined on a new Adaptivity Design Space based on a Perception-Decision-Action cycle com-ing from cognitive psychology. A user experiment compares four conditions of Polymodal Menus (graphical, vocal, gestural, and mixed) in terms of menu selection time, error rate, user subjective satisfaction and user preference, when item prediction has a low or high level of accuracy. Polymodal Menus offer alternative input/output modalities to select menu items in various contexts of use, especially when graphical modality is constrained.
- Motoyuki Akamatsu, I. Scott MacKenzie, and Thierry Hasbrouc. 1995. A Comparison of Tactile, Auditory, and Visual Feedback in a Pointing Task using a Mouse-Type Device. Ergonomics 38, (1995) 816--827.Google ScholarCross Ref
- Pierre Akiki, Arosha K. Bandara, and Yijun Yu. 2014. Adaptive Model-Driven User Interface Development Systems. ACM Computing Surveys 47, 1 (July 2014), Article #9. Google ScholarDigital Library
- Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2016. Visual Menu Techniques. ACM Computing Surveys 49, 4 (Feb. 2017), Article #40. Google ScholarDigital Library
- Mohammed Belatar and François Coldefy. 2010. Sketched Menu: a Tabletop-Menu Technique for GUI Object Creation. In Proceedings of the 2nd ACM Symposium on Engineering Interactive Computing Systems (EICS'2010). ACM, New York, NY, 77--86. Google ScholarDigital Library
- Carlos Belmonte, Juana Gallar, Laura G. Lopez-Briones, and Miguel A. Pozo. 1994. Polymodality in Nociceptive Neurons: Ex-perimental Models of Chemotransduction. In Cellular Mechanisms of Sensory Processing, NATO ASI Series, 79 (Jan. 1994), 87--117.Google Scholar
- Matthias Bezold and Wolfgang Minker. 2011. Adaptive Multimodal Interactive Systems. Springer, Berlin Google ScholarDigital Library
- Sara Bouzit, Denis Chêne, and Gaëlle Calvary. 2014. From Appearing to Disappearing Ephemeral Adaptation for Small Screens. In Proceedings of the 26th Australian Computer-Human Interaction Conference (OzCHI'2014). ACM, New York, NY, 41--48. Google ScholarDigital Library
- Sara Bouzit, Gaëlle Calvary, Denis Chêne, and Jean Vanderdonckt. A Design Space for Engineering Graphical Adaptive Menus. In Proceedings of the 8th ACM Symposium on Engineering Interactive Computing Systems (EICS'2016). ACM, New York, NY, 239--244. Google ScholarDigital Library
- Robert Bridle and Eric McCreath. 2005. Predictive Menu Selection on a Mobile Phone. In Proceedings of Workshop W7 on mining spatio-temporal data (ECML/PKDD'2005). Retrieved May 21st, 2017 from http://users.cecs.anu.edu.au/~Eric.McCreath/papers/ mining05.pdfGoogle Scholar
- Eduardo Castillejo, Aitor Almeida, and Diego López-de-Ipiña. 2014. Modelling Users, Context and Devices for Adaptive User Interface Systems. International Journal of Pervasive Computing and Communications 10, 1 (2014), 69--91.Google ScholarCross Ref
- Andy Cockburn, Carl Gutwin, and Saul Greenberg. 2007. A Predictive Model of Menu Performance. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2007). ACM, New York, NY, 627--636. Google ScholarDigital Library
- Andy Cockburn and Stephen Brewster. 2005. Multimodal Feedback for the Acquisition of Small Targets. Ergonomics 48, 9 (Feb. 2007), 1129--1150.Google ScholarCross Ref
- Joëlle Coutaz, Laurence Nigay, Daniel Salber, Ann Blandford, Jon May and Richard M. Young. 1995. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: the CARE Properties. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT'95). Chapman & Hall, London, 115--120.Google Scholar
- Leah Findlater and Joanna McGrenere. 2008. Impact of Screen Size on Performance, Awareness, and User Satisfaction with Adaptive Graphical User Interfaces. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2008). ACM, New York, NY, 1247--1256. Google ScholarDigital Library
- Leah Findlater, Karyn Moffatt, Joanna McGrenere, and Jessica Dawson. 2009. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2009). ACM, New York, NY, 1655--1664. Google ScholarDigital Library
- Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the Design Space for Adaptive Graphical User Interfaces. In Proceedings of the ACM Working Conference on Advanced Visual Interfaces (AVI'2006). ACM, New York, NY, 201--208. Google ScholarDigital Library
- Krzysztof Z. Gajos, Katherine Everitt, Desney S. Tan, Mary Czerwinski, and Daniel S. Weld. 2008. Predictability and Accuracy in Adaptive User Interfaces. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2008). ACM, New York, NY, 1271--1274. Google ScholarDigital Library
- Mikhail V Goubko, and Alexander Varnavsky. 2016. Users' preference as a criterion for hierarchical menu optimization. In Proc. of ACM Symposium on Engineering Interactive Computing Systems (EICS'2016). ACM Press, New York, NY, 305--310. Google ScholarDigital Library
- Francesca Gullà, Silvia Ceccacci, Michele Germani, and Lorenzo Cavalieri. 2015. Design Adaptable and Adaptive User Interfaces: A Method to Manage the Information. In Proc. of Italian Forum 2014 on Ambient Assisted Living. Biosystems & Biorobotics Series, vol. 11. Springer, Berlin, 47--58.Google Scholar
- Bronwin Jason, Andre Calitz, and Jean Greyling. 2010. The evaluation of an adaptive user interface model. In Proc. of the 2010 Annual Research Conf. of the South African Institute of Computer Scientists and Information Technologists (SAICSIT'2010). ACM Press, New York, NY, 132--143. Google ScholarDigital Library
- Talia Lavie and Joachim Meyer. 2010. Benefits and Costs of Adaptive User Interfaces. Int. J. of Human-Comp. Studies 68, 8, 508--524. Google ScholarDigital Library
- Dong-Seok Lee and Wan Chul Yoon. 2004. Quantitative Results Assessing Design Issues of Selection-Supportive Menus. Int. J. of Industrial Ergonomics 33, 1 (Jan. 2004), 41--52.Google Scholar
- Victor López-Jaquero, Francisco Montero, José P. Molina, Antonio Fernández-Caballero, and Pascual González. Model-Based Design of Adaptive User Interfaces through Connectors. In Proceedings of 10th International Workshop on Design, Specification, and Verification of Interactive Systems (DSVIS'2003). Lecture Notes in Computer Science, vol. 2844. Springer, Berlin, 245--257.Google ScholarCross Ref
- Xiaochuan Ma, Paul P. Maglio, and Hui Su. 2003. Multimodal Menu Interface for Mobile Web Browsing. In Proceedings of 9th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT'2003). IOS Press, Amsterdam, 272--278. Retrieved May 23rd, 2017 from http://www.idemployee.id.tue.nl/g.w.m.rauterberg/conferences/INTERACT2003/INTERACT2003-p272.PDFGoogle Scholar
- Ian Oakley, Stephen Brewster, and Philip Gray. 2001. Solving Multi-Target Haptic Problems in Menu Interaction. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2001), Extended Abstracts. ACM, New York, NY. Google ScholarDigital Library
- Martin Pielot, Anastasia Kazakova, Tobias. Hesselmann, Wilko. Heuten, and Susanne Boll. 2012. PocketMenu: Non-Visual Menus for Touch Screen Devices. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI'2012). ACM Press, New York, NY, 327--330. Google ScholarDigital Library
- Andrew Sears and Ben Shneiderman. 1994. Split Menus: Effectively Using Selection Frequency to Organize Menus. ACM Trans. on Comp.-Human Interaction 1, 1 (March 1994), 27--51. Google ScholarDigital Library
- Choonsung Shin, Jin-Hyuk Hong, and Anind K. Dey. 2012. Understanding and Prediction of Mobile Application Usage for Smart Phones. In Proceedings of International Conference on Ubiquitous Computing (UbiComp'2012). ACM, New York, NY, 173--182. Google ScholarDigital Library
- Erum Tanvir, Jonathan Cullen, Pourang Irani, and Andy Cockburn. 2008. AAMU: Adaptive Activation Area Menus for Improving Selection in Cascading Pull-Down Menus. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2008). ACM, New York, NY, 1381--1384. Google ScholarDigital Library
- Libb Thims. 2005. On the Nature of the Human Chemical Bond. Journal of Human Thermodynamics 1, 5, 36--61. Retrieved April 3rd, 2017 from http://www.humanthermodynamics.com/JHT/Human-Chemical-Bond.htmlGoogle Scholar
- Theophanis Tsandilas and M. C. Schraefel. 2005. An Empirical Assessment of Adaptation Techniques. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2005), Extended Abstracts. ACM, New York, NY, 2009--2012. Google ScholarDigital Library
- Yaohua Yu and Zhengjie Liu. 2010. Improving the Performance and Usability for Visual Menu Interface on Mobile Computers. In Proceedings of the ACM Working Confeernce on Advanced Visual Interfaces (AVI'2010). ACM, New York, NY, 369--372. Google ScholarDigital Library
- Shengdong Zhao, Pierre Dragicevic, Mark Chignell, Ravin Balakrishnan, and Patrick Baudisch. 2007. Earpod: Eyes-Free Menu Selection Using Touch Input and Reactive Audio Feedback. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2007). ACM, New York, NY, 1395--1404. Google ScholarDigital Library
Index Terms
Polymodal Menus: A Model-based Approach for Designing Multimodal Adaptive Menus for Small Screens
Recommendations
Exploring a Design Space of Graphical Adaptive Menus: Normal vs. Small Screens
Special Issue on IUI 2018Graphical Adaptive Menus are Graphical User Interface menus whose predicted items of immediate use can be automatically rendered in a prediction window. Rendering this prediction window is a key question for adaptivity to enable the end-user to ...
A design space for engineering graphical adaptive menus
EICS '16: Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing SystemsThis paper presents a design space for exploring design options of adaptive graphical menus based on Bertin's eight visual variables: position, size, shape, value, color, orientation, texture, and motion. In order to transform a traditional (static) ...
Cloud Menus: a Circular Adaptive Menu for Small Screens
IUI '18: 23rd International Conference on Intelligent User InterfacesThis paper presents Cloud Menus, a split adaptive menu for small screens where the predicted menu items are arranged in a circular tag cloud with a location consistent with their corresponding position in the static menu and a font size depending on ...
Comments