skip to main content
research-article

Polymodal Menus: A Model-based Approach for Designing Multimodal Adaptive Menus for Small Screens

Published:30 June 2017Publication History
Skip Abstract Section

Abstract

This paper presents a model-based approach for designing Polymodal Menus, a new type of multimodal adaptive menu for small screen graphical user interfaces where item selection and adaptivity are responsive to more than one interaction modality: a menu item can be selected graphically, tactilely, vocally, gesturally, or any combination of them. The prediction window containing the most predicted menu items by assignment, equivalence, or redundancy is made equally adaptive. For this purpose, an adaptive menu model maintains the most predictable menu items according to various prediction methods. This model is exploited throughout various steps defined on a new Adaptivity Design Space based on a Perception-Decision-Action cycle com-ing from cognitive psychology. A user experiment compares four conditions of Polymodal Menus (graphical, vocal, gestural, and mixed) in terms of menu selection time, error rate, user subjective satisfaction and user preference, when item prediction has a low or high level of accuracy. Polymodal Menus offer alternative input/output modalities to select menu items in various contexts of use, especially when graphical modality is constrained.

References

  1. Motoyuki Akamatsu, I. Scott MacKenzie, and Thierry Hasbrouc. 1995. A Comparison of Tactile, Auditory, and Visual Feedback in a Pointing Task using a Mouse-Type Device. Ergonomics 38, (1995) 816--827.Google ScholarGoogle ScholarCross RefCross Ref
  2. Pierre Akiki, Arosha K. Bandara, and Yijun Yu. 2014. Adaptive Model-Driven User Interface Development Systems. ACM Computing Surveys 47, 1 (July 2014), Article #9. Google ScholarGoogle ScholarDigital LibraryDigital Library
  3. Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2016. Visual Menu Techniques. ACM Computing Surveys 49, 4 (Feb. 2017), Article #40. Google ScholarGoogle ScholarDigital LibraryDigital Library
  4. Mohammed Belatar and François Coldefy. 2010. Sketched Menu: a Tabletop-Menu Technique for GUI Object Creation. In Proceedings of the 2nd ACM Symposium on Engineering Interactive Computing Systems (EICS'2010). ACM, New York, NY, 77--86. Google ScholarGoogle ScholarDigital LibraryDigital Library
  5. Carlos Belmonte, Juana Gallar, Laura G. Lopez-Briones, and Miguel A. Pozo. 1994. Polymodality in Nociceptive Neurons: Ex-perimental Models of Chemotransduction. In Cellular Mechanisms of Sensory Processing, NATO ASI Series, 79 (Jan. 1994), 87--117.Google ScholarGoogle Scholar
  6. Matthias Bezold and Wolfgang Minker. 2011. Adaptive Multimodal Interactive Systems. Springer, Berlin Google ScholarGoogle ScholarDigital LibraryDigital Library
  7. Sara Bouzit, Denis Chêne, and Gaëlle Calvary. 2014. From Appearing to Disappearing Ephemeral Adaptation for Small Screens. In Proceedings of the 26th Australian Computer-Human Interaction Conference (OzCHI'2014). ACM, New York, NY, 41--48. Google ScholarGoogle ScholarDigital LibraryDigital Library
  8. Sara Bouzit, Gaëlle Calvary, Denis Chêne, and Jean Vanderdonckt. A Design Space for Engineering Graphical Adaptive Menus. In Proceedings of the 8th ACM Symposium on Engineering Interactive Computing Systems (EICS'2016). ACM, New York, NY, 239--244. Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. Robert Bridle and Eric McCreath. 2005. Predictive Menu Selection on a Mobile Phone. In Proceedings of Workshop W7 on mining spatio-temporal data (ECML/PKDD'2005). Retrieved May 21st, 2017 from http://users.cecs.anu.edu.au/~Eric.McCreath/papers/ mining05.pdfGoogle ScholarGoogle Scholar
  10. Eduardo Castillejo, Aitor Almeida, and Diego López-de-Ipiña. 2014. Modelling Users, Context and Devices for Adaptive User Interface Systems. International Journal of Pervasive Computing and Communications 10, 1 (2014), 69--91.Google ScholarGoogle ScholarCross RefCross Ref
  11. Andy Cockburn, Carl Gutwin, and Saul Greenberg. 2007. A Predictive Model of Menu Performance. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2007). ACM, New York, NY, 627--636. Google ScholarGoogle ScholarDigital LibraryDigital Library
  12. Andy Cockburn and Stephen Brewster. 2005. Multimodal Feedback for the Acquisition of Small Targets. Ergonomics 48, 9 (Feb. 2007), 1129--1150.Google ScholarGoogle ScholarCross RefCross Ref
  13. Joëlle Coutaz, Laurence Nigay, Daniel Salber, Ann Blandford, Jon May and Richard M. Young. 1995. Four Easy Pieces for Assessing the Usability of Multimodal Interaction: the CARE Properties. In Proceedings of IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT'95). Chapman & Hall, London, 115--120.Google ScholarGoogle Scholar
  14. Leah Findlater and Joanna McGrenere. 2008. Impact of Screen Size on Performance, Awareness, and User Satisfaction with Adaptive Graphical User Interfaces. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2008). ACM, New York, NY, 1247--1256. Google ScholarGoogle ScholarDigital LibraryDigital Library
  15. Leah Findlater, Karyn Moffatt, Joanna McGrenere, and Jessica Dawson. 2009. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2009). ACM, New York, NY, 1655--1664. Google ScholarGoogle ScholarDigital LibraryDigital Library
  16. Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S. Weld. 2006. Exploring the Design Space for Adaptive Graphical User Interfaces. In Proceedings of the ACM Working Conference on Advanced Visual Interfaces (AVI'2006). ACM, New York, NY, 201--208. Google ScholarGoogle ScholarDigital LibraryDigital Library
  17. Krzysztof Z. Gajos, Katherine Everitt, Desney S. Tan, Mary Czerwinski, and Daniel S. Weld. 2008. Predictability and Accuracy in Adaptive User Interfaces. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2008). ACM, New York, NY, 1271--1274. Google ScholarGoogle ScholarDigital LibraryDigital Library
  18. Mikhail V Goubko, and Alexander Varnavsky. 2016. Users' preference as a criterion for hierarchical menu optimization. In Proc. of ACM Symposium on Engineering Interactive Computing Systems (EICS'2016). ACM Press, New York, NY, 305--310. Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. Francesca Gullà, Silvia Ceccacci, Michele Germani, and Lorenzo Cavalieri. 2015. Design Adaptable and Adaptive User Interfaces: A Method to Manage the Information. In Proc. of Italian Forum 2014 on Ambient Assisted Living. Biosystems & Biorobotics Series, vol. 11. Springer, Berlin, 47--58.Google ScholarGoogle Scholar
  20. Bronwin Jason, Andre Calitz, and Jean Greyling. 2010. The evaluation of an adaptive user interface model. In Proc. of the 2010 Annual Research Conf. of the South African Institute of Computer Scientists and Information Technologists (SAICSIT'2010). ACM Press, New York, NY, 132--143. Google ScholarGoogle ScholarDigital LibraryDigital Library
  21. Talia Lavie and Joachim Meyer. 2010. Benefits and Costs of Adaptive User Interfaces. Int. J. of Human-Comp. Studies 68, 8, 508--524. Google ScholarGoogle ScholarDigital LibraryDigital Library
  22. Dong-Seok Lee and Wan Chul Yoon. 2004. Quantitative Results Assessing Design Issues of Selection-Supportive Menus. Int. J. of Industrial Ergonomics 33, 1 (Jan. 2004), 41--52.Google ScholarGoogle Scholar
  23. Victor López-Jaquero, Francisco Montero, José P. Molina, Antonio Fernández-Caballero, and Pascual González. Model-Based Design of Adaptive User Interfaces through Connectors. In Proceedings of 10th International Workshop on Design, Specification, and Verification of Interactive Systems (DSVIS'2003). Lecture Notes in Computer Science, vol. 2844. Springer, Berlin, 245--257.Google ScholarGoogle ScholarCross RefCross Ref
  24. Xiaochuan Ma, Paul P. Maglio, and Hui Su. 2003. Multimodal Menu Interface for Mobile Web Browsing. In Proceedings of 9th IFIP TC 13 International Conference on Human-Computer Interaction (INTERACT'2003). IOS Press, Amsterdam, 272--278. Retrieved May 23rd, 2017 from http://www.idemployee.id.tue.nl/g.w.m.rauterberg/conferences/INTERACT2003/INTERACT2003-p272.PDFGoogle ScholarGoogle Scholar
  25. Ian Oakley, Stephen Brewster, and Philip Gray. 2001. Solving Multi-Target Haptic Problems in Menu Interaction. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2001), Extended Abstracts. ACM, New York, NY. Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. Martin Pielot, Anastasia Kazakova, Tobias. Hesselmann, Wilko. Heuten, and Susanne Boll. 2012. PocketMenu: Non-Visual Menus for Touch Screen Devices. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI'2012). ACM Press, New York, NY, 327--330. Google ScholarGoogle ScholarDigital LibraryDigital Library
  27. Andrew Sears and Ben Shneiderman. 1994. Split Menus: Effectively Using Selection Frequency to Organize Menus. ACM Trans. on Comp.-Human Interaction 1, 1 (March 1994), 27--51. Google ScholarGoogle ScholarDigital LibraryDigital Library
  28. Choonsung Shin, Jin-Hyuk Hong, and Anind K. Dey. 2012. Understanding and Prediction of Mobile Application Usage for Smart Phones. In Proceedings of International Conference on Ubiquitous Computing (UbiComp'2012). ACM, New York, NY, 173--182. Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. Erum Tanvir, Jonathan Cullen, Pourang Irani, and Andy Cockburn. 2008. AAMU: Adaptive Activation Area Menus for Improving Selection in Cascading Pull-Down Menus. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2008). ACM, New York, NY, 1381--1384. Google ScholarGoogle ScholarDigital LibraryDigital Library
  30. Libb Thims. 2005. On the Nature of the Human Chemical Bond. Journal of Human Thermodynamics 1, 5, 36--61. Retrieved April 3rd, 2017 from http://www.humanthermodynamics.com/JHT/Human-Chemical-Bond.htmlGoogle ScholarGoogle Scholar
  31. Theophanis Tsandilas and M. C. Schraefel. 2005. An Empirical Assessment of Adaptation Techniques. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2005), Extended Abstracts. ACM, New York, NY, 2009--2012. Google ScholarGoogle ScholarDigital LibraryDigital Library
  32. Yaohua Yu and Zhengjie Liu. 2010. Improving the Performance and Usability for Visual Menu Interface on Mobile Computers. In Proceedings of the ACM Working Confeernce on Advanced Visual Interfaces (AVI'2010). ACM, New York, NY, 369--372. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Shengdong Zhao, Pierre Dragicevic, Mark Chignell, Ravin Balakrishnan, and Patrick Baudisch. 2007. Earpod: Eyes-Free Menu Selection Using Touch Input and Reactive Audio Feedback. In Proceedings of ACM Conference on Human Factors in Computing Systems (CHI'2007). ACM, New York, NY, 1395--1404. Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. Polymodal Menus: A Model-based Approach for Designing Multimodal Adaptive Menus for Small Screens

        Recommendations

        Comments

        Login options

        Check if you have access through your login credentials or your institution to get full access on this article.

        Sign in

        Full Access

        • Published in

          cover image Proceedings of the ACM on Human-Computer Interaction
          Proceedings of the ACM on Human-Computer Interaction  Volume 1, Issue EICS
          June 2017
          249 pages
          EISSN:2573-0142
          DOI:10.1145/3120954
          Issue’s Table of Contents

          Copyright © 2017 ACM

          Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

          Publisher

          Association for Computing Machinery

          New York, NY, United States

          Publication History

          • Published: 30 June 2017
          Published in pacmhci Volume 1, Issue EICS

          Permissions

          Request permissions about this article.

          Request Permissions

          Check for updates

          Qualifiers

          • research-article

        PDF Format

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader