Jean Vanderdonckt
Mathieu Zen
Radu-Daniel Vatavu
Skip Abstract Section
Abstract
We introduce AB4Web, a web-based engine that implements a balanced randomized version of the multivariate A/B testing, specifically designed for practitioners to readily compare end-users' preferences for user interface alternatives, such as menu layouts, widgets, controls, forms, or visual input commands. AB4Web automatically generates a balanced set of randomized pairs from a pool of user interface design alternatives, presents them to participants, collects their preferences, and reports results from the perspective of four quantitative measures: the number of presentations, the preference percentage, the latent score of preference, and the matrix of preferences. In this paper, we exemplify the AB4Web tester with a user study for which N=108 participants expressed their preferences regarding the visual design of 49 distinct graphical adaptive menus, with a total number of 5,400 preference votes. We compare the results obtained from our quantitative measures with four alternative methods: Condorcet, de Borda count starting at one and zero, and the Dowdall scoring system. We plan to release AB4Web as a public tool for practitioners to create their own A/B testing experiments.
- David Ahlström, Andy Cockburn, Carl Gutwin, and Pourang Irani. 2010. Why It's Quick to Be Square: Modelling New and Existing Hierarchical Menu Designs. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '10). ACM, New York, NY, USA, 1371--1380. Google Scholar
Digital Library
- Erik Andersen, Yun-En Liu, Rich Snider, Roy Szeto, and Zoran Popovic. 2011. Placing a Value on Aesthetics in Online Casual Games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1275--1278. Google ScholarDigital Library
- Michael P. Arcuri, Thomas Scott Coon, Jeffrey J. Johnson, Alexis Warren, Jacob Manning, and Martijn Eldert van Tilburg. 2000. Adaptive Menus, Patent US6121968A, Microsoft. (Sept. 2000). Filed June 17th, 1998, Issued Sep. 19th., 2000.Google Scholar
- Aslina Baharum and Azizah Jaafar. 2013. Users' Expectation of Web Objects Location: Case Study of ASEAN Countries. In Third International Visual Informatics Conference on Advances in Visual Informatics - Volume 8237 (IVIC 2013). Springer-Verlag, Berlin, Heidelberg, 383--395. Google ScholarDigital Library
- Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2008. Flower Menus: A New Type of Marking Menu with Large Menu Breadth, Within Groups and Efficient Expert Mode Memorization. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI '08). ACM, New York, NY, USA, 15--22.Google ScholarDigital Library
- Gilles Bailly, Eric Lecolinet, and Laurence Nigay. 2016. Visual Menu Techniques. Comput. Surveys 49, 4, Article 60 (Dec. 2016), 41 pages. Google ScholarDigital Library
- Patrick Baudisch, Desney Tan, Maxime Collomb, Dan Robbins, Ken Hinckley, Ken Hinckley, Maneesh Agrawala, Shengdong Zhao, and Gonzalo Ramos. 2006. Phosphor: Explaining Transitions in the User Interface Using Afterglow Effects. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (UIST '06). ACM, New York, NY, USA, 169--178. Google ScholarDigital Library
- Benjamin B. Bederson. 2000. Fisheye Menus. In Proceedings of the 13th Annual ACM Symposium on User Interface Software and Technology (UIST '00). ACM, New York, NY, USA, 217--225. Google ScholarDigital Library
- Michael Bernard. 2001. User Expectations for the Location of Web Objects. In CHI '01 Extended Abstracts on Human Factors in Computing Systems (CHI EA '01). ACM, New York, NY, USA, 171--172. Google ScholarDigital Library
- Michael L. Bernard. 2003. Examining User Expectations for the Location of Common E-Commerce Web Objects. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 47, 11 (2003), 1356--1360. arXiv:https://doi.org/10.1177/154193120304701108Google ScholarCross Ref
- Sara Bouzit, Gaëlle Calvary, Denis Chêne, and Jean Vanderdonckt. 2016. A Design Space for Engineering Graphical Adaptive Menus. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). ACM, New York, NY, USA, 239--244. Google ScholarDigital Library
- Sara Bouzit, Gaëlle Calvary, Denis Chêne, and Jean Vanderdonckt. 2016. A Design Space for Engineering Graphical Adaptive Menus. In Proceedings of the 8th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '16). ACM, New York, NY, USA, 239--244. Google ScholarDigital Library
- Sara Bouzit, Gaëlle Calvary, Denis Chêne, and Jean Vanderdonckt. 2017. Polymodal Menus: A Model-based Approach for Designing Multimodal Adaptive Menus for Small Screens. Proc. ACM Hum.-Comput. Interact. 1, EICS, Article 15 (June 2017), 19 pages. Google ScholarDigital Library
- Sara Bouzit, Denis Chêne, and Gaëlle Calvary. 2014. From Appearing to Disappearing Ephemeral Adaptation for Small Screens. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design (OzCHI '14). ACM, New York, NY, USA, 41--48.Google ScholarDigital Library
- Sara Bouzit, Denis Chêne, and Gaëlle Calvary. 2015. Evanescent Adaptation on Small Screens. In Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction (OzCHI '15). ACM, New York, NY, USA, 62--68. Google ScholarDigital Library
- Ralph Allan Bradley and Milton E. Terry. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39, 3/4 (1952), 324--345. http://www.jstor.org/stable/2334029Google Scholar
- Robert Bridle and Eric McCreath. 2006. Inducing Shortcuts on a Mobile Phone Interface. In Proceedings of the 11th International Conference on Intelligent User Interfaces (IUI '06). ACM, New York, NY, USA, 327--329. Google ScholarDigital Library
- Andy Cockburn, Carl Gutwin, and Saul Greenberg. 2007. A Predictive Model of Menu Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '07). ACM, New York, NY, USA, 627--636. Google ScholarDigital Library
- Ph. Courcoux and M. Semenou. 1997. Preference data analysis using a paired comparison model. Food Quality and Preference 8, 5 (1997), 353 -- 358. Third Sensometrics Meeting.Google ScholarCross Ref
- Peter Emerson. 2013. The original Borda count and partial voting. Social Choice and Welfare 40, 2 (01 Feb 2013), 353--358.Google Scholar
- Leah Findlater and Krzysztof Z. Gajos. 2009. Design Space and Evaluation Challenges of Adaptive Graphical User Interfaces. AI Magazine 30, 4 (2009), 68--73. http://www.aaai.org/ojs/index.php/aimagazine/article/view/2268Google ScholarDigital Library
- Leah Findlater and Joanna McGrenere. 2004. A Comparison of Static, Adaptive, and Adaptable Menus. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '04). ACM, New York, NY, USA, 89--96. Google ScholarDigital Library
- Leah Findlater, Karyn Moffatt, Joanna McGrenere, and Jessica Dawson. 2009. Ephemeral Adaptation: The Use of Gradual Onset to Improve Menu Selection Performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09). ACM, New York, NY, USA, 1655--1664. Google ScholarDigital Library
- Jon Fraenkel and Bernard Grofman. 2014. The Borda Count and its real-world alternatives: Comparing scoring rules in Nauru and Slovenia. Australian Journal of Political Science 49, 2 (2014), 186--205. arXiv:https://doi.org/10.1080/10361146.2014.900530Google ScholarCross Ref
- Krzysztof Z. Gajos, Mary Czerwinski, Desney S. Tan, and Daniel S.Weld. 2006. Exploring the design space for adaptive graphical user interfaces. In Proceedings of the working conference on Advanced visual interfaces, AVI 2006, Venezia, Italy, May 23--26, 2006, Augusto Celentano (Ed.). ACM Press, 201--208. Google ScholarDigital Library
- Krzysztof Z. Gajos, Katherine Everitt, Desney S. Tan, Mary Czerwinski, and Daniel S. Weld. 2008. Predictability and Accuracy in Adaptive User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '08). ACM, New York, NY, USA, 1271--1274. Google ScholarDigital Library
- Björn Hartmann, Loren Yu, Abel Allison, Yeonsoo Yang, and Scott R. Klemmer. 2008. Design As Exploration: Creating Interface Alternatives Through Parallel Authoring and Runtime Tuning. In Proc. of the 21st Annual ACM Symposium on User Interface Software and Technology (UIST '08). ACM, New York, NY, USA, 91--100. Google ScholarDigital Library
- Neil T. Heffernan and Cristina Lindquist Heffernan. 2014. The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learning and Teaching. International Journal of Artificial Intelligence in Education 24, 4 (01 Dec 2014), 470--497.Google ScholarCross Ref
- Anthony Hoeber, Alan Mandler, and Norman Cox. 1992. Method and apparatus for selecting button functions and retaining selected options on a display, Patent US5243697, Oracle America Inc. (1992). https://patents.google.com/ patent/US5243697 Filed March 15th, 1992, Granted July 9th, 1993.Google Scholar
- Bowen Hui, Grant A. Partridge, and Craig Boutilier. 2009. A probabilistic mental model for estimating disruption. In Proc. of the 14th Int. Conf. on Intelligent User Interfaces (IUI '08). ACM, New York, NY, USA, 287--296. Google ScholarDigital Library
- Ron Kohavi and Roger Longbotham. 2017. Online Controlled Experiments and A/B Testing. Springer US, Boston, MA, 922--929.Google Scholar
- John Lamping, Ramana Rao, and Peter Pirolli. 1995. A Focus+Context Technique Based on Hyperbolic Geometry for Visualizing Large Hierarchies. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 401--408. Google ScholarDigital Library
- Dong-Seok Lee and Wan Chul Yoon. 2004. Quantitative results assessing design issues of selection-supportive menus. International Journal of Industrial Ergonomics 33, 1 (2004), 41 -- 52.Google ScholarCross Ref
- Xiaoyang Mao, Yuji Hatanaka, Atsumi Imamiya, Yuki Kato, and Kentaro Go. 2000. Visualizing Computational Wear with Physical Wear. In Proceedings of the 6th ERCIM Workshop "User Interfaces for All" (UI4All '00), Pier-Luigi Emiliani and Constantine Stephanidis (Eds.). CNR-IROE, Pisa, Italy, 12. http://ui4all.ics.forth.gr/UI4ALL-2000/files/Long_papers/ Mao.pdfGoogle Scholar
- Jeffrey Mitchell and Ben Shneiderman. 1989. Dynamic Versus Static Menus: An Exploratory Comparison. SIGCHI Bulletin 20, 4 (April 1989), 33--37. Google ScholarDigital Library
- Kazuma Murao, Carson Reynolds, and Masatoshi Ishikawa. 2012. Blink Suppression Sensing and Classification. In CHI '12 Extended Abstracts on Human Factors in Computing Systems (CHI EA '12). ACM, New York, NY, USA, 2255--2260. Google ScholarDigital Library
- Jungchul Park, Sung H. Han, Yong S. Park, and Youngseok Cho. 2007. Usability of Adaptable and Adaptive Menus. In Usability and Internationalization. HCI and Culture, Nuray Aykin (Ed.). Springer, Berlin, Heidelberg, 405--411. Google ScholarDigital Library
- Marcus Pivato. 2015. Condorcet meets Bentham. Journal of Mathematical Economics 59 (2015), 58 -- 65.Google ScholarCross Ref
- Antoine Ponsard, Kamyar Ardekani, Kailun Zhang, Frederic Ren, Matei Negulescu, and Joanna McGrenere. 2015. Twist and Pulse: Ephemeral Adaptation to Improve Icon Selection on Smartphones. In Proceedings of the 41st Graphics Interface Conference (GI '15). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 219--222. http://dl.acm.org/citation.cfm?id=2788890.2788929 Google ScholarDigital Library
- Ugo Sangiorgi and Jean Vanderdonckt. 2012. GAMBIT: Addressing Multi-platform Collaborative Sketching with Html5. In Proceedings of the 4th ACM SIGCHI Symposium on Engineering Interactive Computing Systems (EICS '12). ACM, New York, NY, USA, 257--262.Google ScholarDigital Library
- Andrew Sears and Ben Shneiderman. 1994. Split Menus: Effectively Using Selection Frequency to Organize Menus. ACM Trans. Comput.-Hum. Interact. 1, 1 (March 1994), 27--51. Google ScholarDigital Library
- A. Dawn Shaikh, Barbara S. Chaparro, and Anirudha Joshi. 2006. Indian Users' Expectations for the Location of Web Objects on Informational Websites. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 17 (2006), 1922--1926. arXiv:https://doi.org/10.1177/154193120605001744Google ScholarCross Ref
- A. Dawn Shaikh and K. Lenz. 2006. Where's the search? Re-examining user expectations of web objects. Usability News 8, 1 (2006), 1356--1360.Google Scholar
- Jorge Gabriel Siqueira and Melise M. V. de Paula. 2018. IPEAD A/B Test Execution Framework. In Proceedings of the XIV Brazilian Symposium on Information Systems (SBSI'18). ACM, New York, NY, USA, Article 14, 8 pages.Google Scholar
- Theophanis Tsandilas and m. c. schraefel. 2005. An Empirical Assessment of Adaptation Techniques. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (CHI EA '05). ACM, New York, NY, USA, 2009--2012.Google Scholar
- Jean Vanderdonckt, Sara Bouzit, Gaëlle Calvary, and Denis Chêne. 2018. Cloud Menus: A Circular Adaptive Menu for Small Screens. In 23rd International Conference on Intelligent User Interfaces (IUI '18). ACM, New York, NY, USA, 317--328. Google ScholarDigital Library
- Raynor Vliegendhart, Eelco Dolstra, and Johan Pouwelse. 2012. Crowdsourced User Interface Testing for Multimedia Applications. In Proceedings of the ACM Multimedia 2012 Workshop on Crowdsourcing for Multimedia (CrowdMM '12). ACM, New York, NY, USA, 21--22.Google ScholarDigital Library
- Joseph Williams and Neil Heffernan. 2015. A Methodology for Discovering How to Adaptively Personalize to Users Using Experimental Comparisons. Social Science Research Network (2015).Google Scholar
- Joseph Jay Williams, Anna N. Rafferty, Dustin Tingley, Andrew Ang, Walter S. Lasecki, and Juho Kim. 2018. Enhancing Online Problems Through Instructor-Centered Tools for Randomized Experiments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 207, 12 pages. Google ScholarDigital Library
Index Terms
- AB4Web: An On-Line A/B Tester for Comparing User Interface Design Alternatives
Recommendations
Exploring a Design Space of Graphical Adaptive Menus: Normal vs. Small Screens
Special Issue on IUI 2018Graphical Adaptive Menus are Graphical User Interface menus whose predicted items of immediate use can be automatically rendered in a prediction window. Rendering this prediction window is a key question for adaptivity to enable the end-user to ...
Multivariate Testing of Native Mobile Applications
MoMM '14: Proceedings of the 12th International Conference on Advances in Mobile Computing and MultimediaA/B testing has a long history in web development and is used on a daily basis by many companies. Although it is a common test method for web pages, it is hardly used for native mobile applications. The reason seems to be that it is much more difficult ...
Graphically enhanced keyboard accelerators for GUIs
GI '10: Proceedings of Graphics Interface 2010We introduce GEKA, a graphically enhanced keyboard accelerator method that provides the advantages of a traditional command line interface within a GUI environment, thus avoiding the "Fitts-induced bottleneck" of pointer movement that is characteristic ...
Comments