skip to main content
10.1145/3503252.3531317acmconferencesArticle/Chapter ViewAbstractPublication PagesumapConference Proceedingsconference-collections
research-article
Public Access

Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users

Published: 04 July 2022 Publication History

Abstract

For many blind users, interaction with computer applications using screen reader assistive technology is a frustrating and time-consuming affair, mostly due to the complexity and heterogeneity of applications’ user interfaces. An interview study revealed that many applications do not adequately convey their interface structure and controls to blind screen reader users, thereby placing additional burden on these users to acquire this knowledge on their own. This is often an arduous and tedious learning process given the one-dimensional navigation paradigm of screen readers. Moreover, blind users have to repeat this learning process multiple times, i.e., once for each application, since applications differ in their interface designs and implementations. In this paper, we propose a novel push-based approach to make non-visual computer interaction easy, efficient, and uniform across different applications. The key idea is to make screen reader interaction ‘structure-agnostic’, by automatically identifying and extracting all application controls and then instantly ‘pushing’ these controls on demand to the blind user via a custom overlay dashboard interface. Such a custom overlay facilitates uniform and efficient screen reader navigation across all applications. A user study showed significant improvement in user satisfaction and interaction efficiency with our approach compared to a state-of-the-art screen reader.

Supplementary Material

In this presentation we introduce the Accessbolt system. We start with sharing the findings of our preliminary study and the motivations behind creating Accessbolt. Then, we share the design and technical details of Accessbolt system. Next, we present the evaluation results of Accessbolt. Finally, we discuss its limitations and possible directions for future work. (UMAP22_Uckun.zip)

References

[1]
Apple. 2021. Accessibility API. https://developer.apple.com/accessibility/macos/.
[2]
Apple. 2021. Rotor function for VoiceOver. https://support.apple.com/guide/voiceover/with-the-voiceover-rotor-mchlp2719/mac
[3]
Apple. 2021. Vision Accessibility - Mac - Apple. https://www.apple.com/accessibility/mac/vision/
[4]
Edward C Bell and Natalia M Mino. 2015. Employment outcomes for blind and visually impaired adults. (2015).
[5]
Syed Masum Billah, Vikas Ashok, Donald E Porter, and IV Ramakrishnan. 2017. Ubiquitous accessibility for people with visual impairments: Are we there yet?. In Proceedings of the 2017 chi conference on human factors in computing systems. 5862–5868.
[6]
Yevgen Borodin, Jeffrey P Bigham, Glenn Dausch, and IV Ramakrishnan. 2010. More than meets the eye: a survey of screen-reader browsing strategies. In Proceedings of the 2010 International Cross Disciplinary Conference on Web Accessibility (W4A). ACM, 13.
[7]
Diego Bovenzi, Gerardo Canfora, and Anna Rita Fasolino. 2003. Enabling legacy system accessibility by web heterogeneous clients. In Seventh European Conference onSoftware Maintenance and Reengineering, 2003. Proceedings.IEEE, 73–81.
[8]
Lawrence H Boyd, Wesley L Boyd, and Gregg C Vanderheiden. 1990. The graphical user interface: Crisis, danger, and opportunity. Journal of Visual Impairment & Blindness 84, 10 (1990), 496–502.
[9]
James V Bradley. 1958. Complete counterbalancing of immediate sequential effects in a Latin square design. J. Amer. Statist. Assoc. 53, 282 (1958), 525–528.
[10]
Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Giulio Mori. 2012. Designing e-learning collaborative tools for blind people. E-Learning-Long-Distance and Lifelong Perspectives (2012), 125–144.
[11]
Jieshan Chen, Chunyang Chen, Zhenchang Xing, Xiwei Xu, Liming Zhut, Guoqiang Li, and Jinshui Wang. 2020. Unblind your apps: Predicting natural-language labels for mobile gui components by deep learning. In 2020 IEEE/ACM 42nd International Conference on Software Engineering (ICSE). IEEE, 322–334.
[12]
Michael F Chiang, Roy G Cole, Suhit Gupta, Gail E Kaiser, and Justin B Starren. 2005. Computer and world wide web accessibility by visually disabled patients: Problems and solutions. Survey of ophthalmology 50, 4 (2005), 394–405.
[13]
Michael Connolly, Christof Lutteroth, and Beryl Plimmer. 2010. Document resizing for visually impaired students. In Proceedings of the 22nd Conference of the Computer-Human Interaction Special Interest Group of Australia on Computer-Human Interaction. 128–135.
[14]
David Cournapeau. 2021. Scikit-learn software library for Python. https://scikit-learn.org/stable/
[15]
Christin Engel, Emma Franziska Müller, and Gerhard Weber. 2019. SVGPlott: an accessible tool to generate highly adaptable, accessible audio-tactile charts for and from blind and visually impaired people. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments. 186–195.
[16]
Freedom Scientific. 2021. JAWS® - Freedom Scientific. https://www.freedomscientific.com/products/software/jaws/
[17]
GNOME. 2021. Accessibility Toolkit. https://developer.gnome.org/atk/.
[18]
Andres Gonzalez and Loretta Guarino Reid. 2005. Platform-independent accessibility api: Accessible document object model. In Proceedings of the 2005 International Cross-Disciplinary Workshop on Web Accessibility (W4A). 63–71.
[19]
Yoichi Haga, Wataru Makishi, Kentaro Iwami, Kentaro Totsu, Kazuhiro Nakamura, and Masayoshi Esashi. 2005. Dynamic Braille display using SMA coil actuator and magnetic latch. Sensors and Actuators A: Physical 119, 2 (2005), 316–322.
[20]
Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139–183.
[21]
Rob Haverty. 2005. New Accessibility Model for Microsoft Windows and Cross Platform Development. SIGACCESS Access. Comput.82 (June 2005), 11–17. https://doi.org/10.1145/1077238.1077240
[22]
Jeff Hendy, Kellogg S. Booth, and Joanna McGrenere. 2010. Graphically Enhanced Keyboard Accelerators for GUIs. In Proceedings of Graphics Interface 2010(Ottawa, Ontario, Canada) (GI ’10). Canadian Information Processing Society, CAN, 3–10.
[23]
Sven Hertling, Markus Schröder, Christian Jilek, and Andreas Dengel. 2017. Where is that Button Again?!-Towards a Universal GUI Search Engine. In ICAART (2). 217–227.
[24]
Harry Hochheiser and Jonathan Lazar. 2010. Revisiting breadth vs. depth in menu structures for blind users of screen readers. Interacting with Computers 22, 5 (2010), 389–398.
[25]
Shaun K. Kane, Chandrika Jayant, Jacob O. Wobbrock, and Richard E. Ladner. 2009. Freedom to Roam: A Study of Mobile Device Adoption and Accessibility for People with Visual and Motor Disabilities. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility (Pittsburgh, Pennsylvania, USA) (Assets ’09). Association for Computing Machinery, New York, NY, USA, 115–122. https://doi.org/10.1145/1639642.1639663
[26]
Sri Hastuti Kurniawan, Alistair G Sutcliffe, and Paul Blenkhorn. 2003. How Blind Users’ Mental Models Affect Their Perceived Usability of an Unfamiliar Screen Reader. In INTERACT, Vol. 3. 631–638.
[27]
Jonathan Lazar, Aaron Allen, Jason Kleinman, and Chris Malarkey. 2007. What frustrates screen reader users on the web: A study of 100 blind users. International Journal of human-computer interaction 22, 3(2007), 247–269.
[28]
Hae-Na Lee, Vikas Ashok, and IV Ramakrishnan. 2020. Rotate-and-Press: A Non-visual Alternative to Point-and-Click?. In International Conference on Human-Computer Interaction. Springer, 291–305.
[29]
H. N. Lee, V. Ashok, and I. V. Ramakrishnan. 2020. Repurposing Visual Input Modalities for Blind Users: A Case Study of Word Processors. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). 2714–2721. https://doi.org/10.1109/SMC42975.2020.9283015
[30]
Barbara Leporini, Maria Claudia Buzzi, and Marina Buzzi. 2012. Interacting with Mobile Devices via VoiceOver: Usability and Accessibility Issues. In Proceedings of the 24th Australian Computer-Human Interaction Conference (Melbourne, Australia) (OzCHI ’12). Association for Computing Machinery, New York, NY, USA, 339–348. https://doi.org/10.1145/2414536.2414591
[31]
Vladimir I Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. 707–710.
[32]
Microsoft. 2021. Microsoft UI Automation. https://docs.microsoft.com/en-us/windows/win32/winauto/entry-uiauto-win32
[33]
Lourdes Morales, Sonia M Arteaga, and Sri Kurniawan. 2013. Design guidelines of a tool to help blind authors independently format their word documents. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. 31–36.
[34]
Giulio Mori, Maria Claudia Buzzi, Marina Buzzi, Barbara Leporini, and Victor MR Penichet. 2010. Making “Google Docs” user interface more accessible for blind people. In International Conference on Advances in New Technologies, Interactive Interfaces, and Communicability. Springer, 20–29.
[35]
NVDA team. 2021. NVDA. https://www.nvaccess.org/.
[36]
ORACLE. 2021. JAVA Accessibility API. https://docs.oracle.com/javase/7/docs/technotes/guides/access/jaapi.html.
[37]
Leonard H Poll and Berry H Eggen. 1996. Non-visual interaction with GUI objects. In People and Computers XI. Springer, 159–168.
[38]
John Gerard Schoeberlein and Yuanqiong Wang. 2014. Usability Evaluation of an Accessible Collaborative Writing Prototype for Blind Users.Journal of Usability Studies 10, 1 (2014).
[39]
Kristen Shinohara and Josh Tenenberg. 2007. Observing Sara: A Case Study of a Blind Person’s Interactions with Technology. In Proceedings of the 9th International ACM SIGACCESS Conference on Computers and Accessibility (Tempe, Arizona, USA) (Assets ’07). Association for Computing Machinery, New York, NY, USA, 171–178. https://doi.org/10.1145/1296843.1296873
[40]
Reeta Singh. 2012. Blind Handicapped Vs. Technology: How do Blind People use Computers?International Journal of Scientific & Engineering Research 3, 4(2012), 1–7.
[41]
J. Thatcher. 1994. Screen Reader/2: Access to OS/2 and the Graphical User Interface. In Proceedings of the First Annual ACM Conference on Assistive Technologies (Marina Del Rey, California, USA) (Assets ’94). Association for Computing Machinery, New York, NY, USA, 39–46. https://doi.org/10.1145/191028.191039
[42]
Katie Wang, Laura G Barron, and Michelle R Hebl. 2010. Making those who cannot see look best: Effects of visual resume formatting on ratings of job applicants with blindness.Rehabilitation psychology 55, 1 (2010), 68.
[43]
Mirza Muhammad Waqar, Muhammad Aslam, and Muhammad Farhan. 2019. An intelligent and interactive interface to support symmetrical collaborative educational writing among visually impaired and sighted users. Symmetry 11, 2 (2019), 238.
[44]
Tetsuya Watanabe, Shinichi Okada, and Tohru Ifukube. 1998. Development of a GUI screen reader for blind persons. Systems and Computers in Japan 29, 13 (1998), 18–27.
[45]
Brian Wentz and Jonathan Lazar. 2011. Usability evaluation of email applications by blind users. Journal of Usability Studies 6, 2 (2011), 75–89.
[46]
David Wicks. 2017. The coding manual for qualitative researchers. Qualitative research in organizations and management: an international journal (2017).
[47]
Cheng Xu, Ali Israr, Ivan Poupyrev, Olivier Bau, and Chris Harrison. 2011. Tactile display for the visually impaired using TeslaTouch. In CHI’11 Extended Abstracts on Human Factors in Computing Systems. 317–322.
[48]
Levent Yobas, Dominique M Durand, Gerard G Skebe, Frederick J Lisy, and Michael A Huff. 2003. A novel integrable microvalve for refreshable braille display system. Journal of microelectromechanical systems 12, 3 (2003), 252–263.
[49]
Zoom. 2021. Zoom Meetings. https://explore.zoom.us/meetings.
[50]
Hong Zou and Jutta Treviranus. 2015. ChartMaster: A Tool for Interacting with Stock Market Charts Using a Screen Reader. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility (Lisbon, Portugal) (ASSETS ’15). Association for Computing Machinery, New York, NY, USA, 107–116. https://doi.org/10.1145/2700648.2809862

Cited By

View all
  • (2024)Screen Reading Enabled by Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688491(1-5)Online publication date: 27-Oct-2024
  • (2024)Enabling Uniform Computer Interaction Experience for Blind Users through Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675605(1-14)Online publication date: 27-Oct-2024
  • (2024)When Being Smart Trumps AI: An Exploration into Consumer Preferences for Smart vs. AI-Powered ProductsComputers in Human Behavior10.1016/j.chb.2024.108405(108405)Online publication date: Aug-2024

Index Terms

  1. Taming User-Interface Heterogeneity with Uniform Overlays for Blind Users
        Index terms have been assigned to the content through auto-classification.

        Recommendations

        Comments

        Information & Contributors

        Information

        Published In

        cover image ACM Conferences
        UMAP '22: Proceedings of the 30th ACM Conference on User Modeling, Adaptation and Personalization
        July 2022
        360 pages
        ISBN:9781450392075
        DOI:10.1145/3503252
        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Sponsors

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        Published: 04 July 2022

        Permissions

        Request permissions for this article.

        Check for updates

        Author Tags

        1. Accessibility
        2. Desktop Usability
        3. Screen Reader
        4. Uniform Interaction
        5. Visual Impairments

        Qualifiers

        • Research-article
        • Research
        • Refereed limited

        Funding Sources

        Conference

        UMAP '22
        Sponsor:

        Acceptance Rates

        Overall Acceptance Rate 162 of 633 submissions, 26%

        Upcoming Conference

        UMAP '25

        Contributors

        Other Metrics

        Bibliometrics & Citations

        Bibliometrics

        Article Metrics

        • Downloads (Last 12 months)260
        • Downloads (Last 6 weeks)34
        Reflects downloads up to 17 Feb 2025

        Other Metrics

        Citations

        Cited By

        View all
        • (2024)Screen Reading Enabled by Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3688491(1-5)Online publication date: 27-Oct-2024
        • (2024)Enabling Uniform Computer Interaction Experience for Blind Users through Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675605(1-14)Online publication date: 27-Oct-2024
        • (2024)When Being Smart Trumps AI: An Exploration into Consumer Preferences for Smart vs. AI-Powered ProductsComputers in Human Behavior10.1016/j.chb.2024.108405(108405)Online publication date: Aug-2024

        View Options

        View options

        PDF

        View or Download as a PDF file.

        PDF

        eReader

        View online with eReader.

        eReader

        HTML Format

        View this article in HTML Format.

        HTML Format

        Login options

        Figures

        Tables

        Media

        Share

        Share

        Share this Publication link

        Share on social media