skip to main content
10.1145/3544548.3580640acmconferencesArticle/Chapter ViewAbstractPublication PageschiConference Proceedingsconference-collections
research-article

Modeling Touch-based Menu Selection Performance of Blind Users via Reinforcement Learning

Published: 19 April 2023 Publication History

Abstract

Although menu selection has been extensively studied in HCI, most existing studies have focused on sighted users, leaving blind users’ menu selection under-studied. In this paper, we propose a computational model that can simulate blind users’ menu selection performance and strategies, including the way they use techniques like swiping, gliding, and direct touch. We assume that selection behavior emerges as an adaptation to the user’s memory of item positions based on experience and feedback from the screen reader. A key aspect of our model is a model of long-term memory, predicting how a user recalls and forgets item position based on previous menu selections. We compare simulation results predicted by our model against data obtained in an empirical study with ten blind users. The model correctly simulated the effect of the menu length and menu arrangement on selection time, the action composition, and the menu selection strategy of the users.

Supplementary Material

MP4 File (3544548.3580640-talk-video.mp4)
Pre-recorded Video Presentation

References

[1]
Carl Halladay Abraham, Bert Boadi-Kusi, Enyam Komla Amewuho Morny, and Prince Agyekum. 2022. Smartphone usage among people living with severe visual impairment and blindness. Assistive Technology 34, 5 (2022), 611–618.
[2]
Robert St Amant, Thomas E Horton, and Frank E Ritter. 2007. Model-based evaluation of expert cell phone menu interaction. ACM Transactions on Computer-Human Interaction (TOCHI) 14, 1(2007), 1–es.
[3]
Caroline Appert and Shumin Zhai. 2009. Using strokes as command shortcuts: cognitive benefits and toolkit support. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Boston, MA, USA, 2289–2298.
[4]
Apple. 2022. Vision Accessibility in iPhone. https://www.apple.com/accessibility/iphone/vision/
[5]
Gilles Bailly, Antti Oulasvirta, Duncan P Brumby, and Andrew Howes. 2014. Model of visual search and selection time in linear menus. In Proceedings of the sigchi conference on human factors in computing systems. ACM, Toronto, Canada, 3865–3874.
[6]
Gilles Bailly, Antti Oulasvirta, Timo Kötzing, and Sabrina Hoppe. 2013. Menuoptimizer: Interactive optimization of menu systems. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, St. Andrews, UK, 331–342.
[7]
Nikola Banovic, Tovi Grossman, and George Fitzmaurice. 2013. The effect of time-based cost of error in target-directed pointing tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris, France, 1373–1382.
[8]
Richard E Bellman and Lotfi Asker Zadeh. 1970. Decision-making in a fuzzy environment. Management science 17, 4 (1970), B–141.
[9]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. 2016. Openai gym. arXiv preprint arXiv:1606.01540(2016).
[10]
Duncan P Brumby, Samantha CE Davies, Christian P Janssen, and Justin J Grace. 2011. Fast or safe? How performance objectives determine modality output choices while interacting on the move. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, Vancouver, Canada, 473–482.
[11]
Noshaba Cheema, Laura A Frey-Law, Kourosh Naderi, Jaakko Lehtinen, Philipp Slusallek, and Perttu Hämäläinen. 2020. Predicting mid-air interaction movements and fatigue using deep reinforcement learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu, HI, USA, 1–13.
[12]
Xiuli Chen, Aditya Acharya, Antti Oulasvirta, and Andrew Howes. 2021. An adaptive model of gaze-based selection. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Online Virtual Conference, 1–11.
[13]
Xiuli Chen, Gilles Bailly, Duncan P Brumby, Antti Oulasvirta, and Andrew Howes. 2015. The emergence of interactive behavior: A model of rational menu search. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. ACM, Seoul, Korea, 4217–4226.
[14]
Xiuli Chen, Sandra Dorothee Starke, Chris Baber, and Andrew Howes. 2017. A cognitive model of how people make decisions through interaction with visual displays. In Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, Denver, CO, USA, 1205–1216.
[15]
Andy Cockburn, Carl Gutwin, and Saul Greenberg. 2007. A predictive model of menu performance. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM, San Jose, 627–636.
[16]
Alistair DN Edwards. 1989. Modelling blind users’ interactions with an auditory computer interface. International journal of man-Machine studies 30, 5 (1989), 575–589.
[17]
Manahel El Lahib, Joe Tekli, and Youssef Bou Issa. 2018. Evaluating Fitts’ law on vibrating touch-screen to improve visual data accessibility for blind users. International Journal of Human-Computer Studies 112 (2018), 16–27.
[18]
Stephen R Ellis and Robert J Hitchcock. 1986. The emergence of Zipf’s law: Spontaneous encoding optimization by users of a command language. IEEE transactions on systems, man, and cybernetics 16, 3(1986), 423–427.
[19]
Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement.Journal of experimental psychology 47, 6 (1954), 381.
[20]
Krzysztof Gajos and Daniel S Weld. 2004. SUPPLE: automatically generating user interfaces. In Proceedings of the 9th international conference on Intelligent user interfaces. ACM, Funchal, Portugal, 93–100.
[21]
Andrew Gelman and Jennifer Hill. 2006. Data analysis using regression and multilevel/hierarchical models. Cambridge university press.
[22]
Samuel J Gershman, Eric J Horvitz, and Joshua B Tenenbaum. 2015. Computational rationality: A converging paradigm for intelligence in brains, minds, and machines. Science 349, 6245 (2015), 273–278.
[23]
Google. 2022. Accessibility – Touch and point targets. https://material.io/design/usability/accessibility.html#layout-and-typography
[24]
Google. 2022. Android accessibility overview. https://support.google.com/accessibility/android/answer/6006564?hl=en
[25]
Google. 2022. Talkback: Hear your screen read out loud. https://support.google.com/accessibility/android/topic/3529932
[26]
William Grussenmeyer and Eelke Folmer. 2017. Accessible touchscreen technology for people with visual impairments: a survey. ACM Transactions on Accessible Computing (TACCESS) 9, 2 (2017), 1–31.
[27]
Tiago Guerreiro, Paulo Lagoa, Hugo Nicolau, Daniel Gonalves, and Joaquim A. Jorge. 2008. From Tapping to Touching: Making Touch Screens Accessible to Blind Users. IEEE MultiMedia 15, 4 (2008), 48–50.
[28]
Sean G Gustafson, Bernhard Rabe, and Patrick M Baudisch. 2013. Understanding palm-based imaginary interfaces: the role of visual and tactile cues when browsing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, Paris, France, 889–898.
[29]
Andrew Howes, Richard L Lewis, and Alonso Vera. 2009. Rational adaptation under task and processing constraints: implications for testing theories of cognition and action.Psychological review 116, 4 (2009), 717.
[30]
Mohit Jain, Nirmalendu Diwakar, and Manohar Swaminathan. 2021. Smartphone usage by expert blind users. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Online Virtual Conference, 1–15.
[31]
Edwin T Jaynes. 1957. Information theory and statistical mechanics. Physical review 106, 4 (1957), 620.
[32]
Jussi Jokinen, Aditya Acharya, Mohammad Uzair, Xinhui Jiang, and Antti Oulasvirta. 2021. Touchscreen typing as optimal supervisory control. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. ACM, Online Virtual Conference, 1–14.
[33]
Jussi PP Jokinen and Tuomo Kujala. 2021. Modelling Drivers’ Adaptation to Assistance Systems. In 13th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. ACM, Leeds, UK, 12–19.
[34]
Jussi PP Jokinen, Tuomo Kujala, and Antti Oulasvirta. 2021. Multitasking in driving as optimal adaptation under uncertainty. Human factors 63, 8 (2021), 1324–1341.
[35]
Jussi PP Jokinen, Zhenxin Wang, Sayan Sarcar, Antti Oulasvirta, and Xiangshi Ren. 2020. Adaptive feature guidance: Modelling visual search with graphical layouts. International Journal of Human-Computer Studies 136 (2020), 102376.
[36]
Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. 1998. Planning and acting in partially observable stochastic domains. Artificial intelligence 101, 1-2 (1998), 99–134.
[37]
Shaun K. Kane, Jeffrey P. Bigham, and Jacob O. Wobbrock. 2008. Slide Rule: Making Mobile Touch Screens Accessible to Blind People Using Multi-Touch Interaction Techniques. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Halifax, Nova Scotia, Canada) (Assets ’08). Association for Computing Machinery, New York, NY, USA, 73–80.
[38]
Antti Kangasrääsiö, Kumaripaba Athukorala, Andrew Howes, Jukka Corander, Samuel Kaski, and Antti Oulasvirta. 2017. Inferring cognitive models from data using approximate Bayesian computation. In Proceedings of the 2017 CHI conference on human factors in computing systems. ACM, Denver, CO, USA, 1295–1306.
[39]
Akif Khan and Shah Khusro. 2019. Blind-friendly user interfaces – a pilot study on improving the accessibility of touchscreen interfaces. Multimedia Tools and Applications 78 (07 2019), 17495–17519.
[40]
Akif Khan and Shah Khusro. 2021. An insight into smartphone-based assistive solutions for visually impaired and blind people: issues, challenges and opportunities. Universal Access in the Information Society 20, 2 (2021), 265–298.
[41]
Yu-Jung Ko, Aini Putkonen, Ali Selman Aydin, Shirin Feiz, Yuheng Wang, Vikas Ashok, IV Ramakrishnan, Antti Oulasvirta, and Xiaojun Bi. 2021. Modeling Gliding-based Target Selection for Blind Touchscreen Users. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction. ACM, Online Virtual Conference, 1–14.
[42]
Stelios Lelis and Andrew Howes. 2011. Informing decisions: how people use online rating information to make choices. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, Vancouver, Canada, 2285–2294.
[43]
Richard L Lewis, Andrew Howes, and Satinder Singh. 2014. Computational rationality: Linking mechanism and behavior through bounded utility maximization. Topics in cognitive science 6, 2 (2014), 279–311.
[44]
Frank Chun Yat Li, David Dearman, and Khai N Truong. 2010. Leveraging proprioception to make mobile phones more accessible to users with visual impairments. In Proceedings of the 12th international ACM SIGACCESS conference on Computers and accessibility. ACM, Orlando, FL, USA, 187–194.
[45]
Zhi Li, Maozheng Zhao, Dibyendu Das, Hang Zhao, Yan Ma, Wanyu Liu, Michel Beaudouin-Lafon, Fusheng Wang, I.V. Ramakrishnan, and Xiaojun Bi. 2022. Select or Suggest? Reinforcement Learning-based Method for High-Accuracy Target Selection on Touchscreens. In CHI ’22: ACM CHI Conference on Human Factors in Computing Systems. ACM, New Orleans, LA, USA, 1–15.
[46]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971(2015).
[47]
Wanyu Liu, Gilles Bailly, and Andrew Howes. 2017. Effects of frequency distribution on linear menu performance. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, Denver, CO, USA, 1307–1312.
[48]
I Scott MacKenzie. 1992. Fitts’ law as a research and design tool in human-computer interaction. Human-computer interaction 7, 1 (1992), 91–139.
[49]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602(2013).
[50]
Alistair Morrison, Xiaoyu Xiong, Matthew Higgs, Marek Bell, and Matthew Chalmers. 2018. A Large-Scale Study of iPhone App Launch Behaviour. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, Montréal, Canada, 1–13.
[51]
Saied B. Nesbat. 2003. A System for Fast, Full-Text Entry for Small Electronic Devices. In Proceedings of the 5th International Conference on Multimodal Interfaces (Vancouver, Canada) (ICMI ’03). Association for Computing Machinery, New York, NY, USA, 4–11.
[52]
Antti Oulasvirta, Jussi P.P. Jokinen, and Andrew Howes. 2022. Computational Rationality as a Theory of Interaction. In CHI ’22: ACM CHI Conference on Human Factors in Computing Systems,. ACM, New Orleans, LA, USA, 1–14.
[53]
Tomi Peltola, Jussi Jokinen, and Samuel Kaski. 2019. Probabilistic Formulation of the Take The Best Heuristic. arXiv preprint arXiv:1911.00572(2019).
[54]
Rajesh PN Rao. 2010. Decision making under uncertainty: a neural model based on partially observable markov decision processes. Frontiers in computational neuroscience 4 (2010), 146.
[55]
Stuart J Russell and Devika Subramanian. 1994. Provably bounded-optimal agents. Journal of Artificial Intelligence Research 2 (1994), 575–609.
[56]
Sayan Sarcar, Jussi PP Jokinen, Antti Oulasvirta, Zhenxin Wang, Chaklam Silpasuwanchai, and Xiangshi Ren. 2018. Ability-based optimization of touchscreen interactions. IEEE Pervasive Computing 17, 1 (2018), 15–26.
[57]
Suraj Singh Senjam, Souvik Manna, and Covadonga Bascaran. 2021. Smartphones-Based Assistive Technology: Accessibility Features and Apps for People with Visual Impairment, and its Usage, Challenges, and Usability Testing. Clinical Optometry 13(2021), 311.
[58]
Siti Rohkmah Mohd Shukri and Andrew Howes. 2019. Children adapt drawing actions to their own motor variability and to the motivational context of action. International Journal of Human-Computer Studies 130 (2019), 152–165.
[59]
Benjamin L Somberg. 1986. A comparison of rule-based and positionally constant arrangements of computer menu items. ACM SIGCHI Bulletin 17, SI (1986), 255–260.
[60]
Matthijs TJ Spaan. 2012. Partially observable Markov decision processes. In Reinforcement Learning. Springer, 387–414.
[61]
Kashyap Todi, Gilles Bailly, Luis Leiva, and Antti Oulasvirta. 2021. Adapting user interfaces with model-based reinforcement learning. ACM, Online Virtual Conference, 1–13.
[62]
Yuan-Chi Tseng and Andrew Howes. 2015. The adaptation of visual search to utility, ecology and design. International Journal of Human-Computer Studies 80 (2015), 45–55.
[63]
Pavani Yalla and Bruce N Walker. 2008. Advanced auditory menus: Design and evaluation of auditory scroll bars. In Proceedings of the 10th international ACM SIGACCESS conference on Computers and accessibility. ACM, Halifax, Canada, 105–112.
[64]
Suwen Zhu, Yoonsang Kim, Jingjie Zheng, Jennifer Yi Luo, Ryan Qin, Liuping Wang, Xiangmin Fan, Feng Tian, and Xiaojun Bi. 2020. Using Bayes’ Theorem for Command Input: Principle, Models, and Applications. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu, HI, USA, 1–15.

Cited By

View all
  • (2025)Spatial Variability Models for Straight Pen-stroking Tasks with Temporal ConstraintsJournal of Information Processing10.2197/ipsjjip.33.9133(91-103)Online publication date: 2025
  • (2024)Mobile User Interface Adaptation Based on Usability Reward Model and Multi-Agent Reinforcement LearningMultimodal Technologies and Interaction10.3390/mti80400268:4(26)Online publication date: 24-Mar-2024
  • (2024)Enabling Uniform Computer Interaction Experience for Blind Users through Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675605(1-14)Online publication date: 27-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
CHI '23: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
April 2023
14911 pages
ISBN:9781450394215
DOI:10.1145/3544548
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 April 2023

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. accessibility
  2. boundedly optimal control
  3. computational rationality
  4. deep reinforcement learning
  5. menu selection

Qualifiers

  • Research-article
  • Research
  • Refereed limited

Funding Sources

  • DOD award
  • NIH award
  • NSF award

Conference

CHI '23
Sponsor:

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025
ACM CHI Conference on Human Factors in Computing Systems
April 26 - May 1, 2025
Yokohama , Japan

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)519
  • Downloads (Last 6 weeks)65
Reflects downloads up to 17 Feb 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Spatial Variability Models for Straight Pen-stroking Tasks with Temporal ConstraintsJournal of Information Processing10.2197/ipsjjip.33.9133(91-103)Online publication date: 2025
  • (2024)Mobile User Interface Adaptation Based on Usability Reward Model and Multi-Agent Reinforcement LearningMultimodal Technologies and Interaction10.3390/mti80400268:4(26)Online publication date: 24-Mar-2024
  • (2024)Enabling Uniform Computer Interaction Experience for Blind Users through Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675605(1-14)Online publication date: 27-Oct-2024
  • (2024)CRTypist: Simulating Touchscreen Typing Behavior via Computational RationalityProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642918(1-17)Online publication date: 11-May-2024
  • (2024)Heads-Up Multitasker: Simulating Attention Switching On Optical Head-Mounted DisplaysProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642540(1-18)Online publication date: 11-May-2024
  • (2024)Explaining crowdworker behaviour through computational rationalityBehaviour & Information Technology10.1080/0144929X.2024.232961644:3(552-573)Online publication date: 24-Apr-2024

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Full Text

View this article in Full Text.

Full Text

HTML Format

View this article in HTML Format.

HTML Format

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media