skip to main content
10.1145/3025171.3025229acmconferencesArticle/Chapter ViewAbstractPublication PagesiuiConference Proceedingsconference-collections
research-article

Web Screen Reading Automation Assistance Using Semantic Abstraction

Published: 07 March 2017 Publication History

Abstract

A screen reader's sequential press-and-listen interface makes for an unsatisfactory and often times painful web-browsing experience for blind people. To help alleviate this situation, we introduce Web Screen Reading Automation Assistant (SRAA) for automating users' screen-reading actions (e.g., finding price of an item) on demand, thereby letting them focus on what they want to do rather than on how to get it done. The key idea is to elevate the interaction from operating on (syntactic) HTML elements, as is done now, to operating on web entities (which are semantically meaningful collections of related HTML elements, e.g., search results, menus, widgets, etc.). SRAA realizes this idea of semantic abstraction by constructing a Web Entity Model (WEM), which is a collection of web entities of the underlying webpage, using an extensive generic library of custom-designed descriptions of commonly occurring web entities across websites. The WEM brings blind users closer to how sighted people perceive and operate on web entities, and together with a natural-language user interface, SRAA relieves users from having to press numerous shortcuts to operate on low-level HTML elements - the principal source of tedium and frustration. This paper describes the design and implementation of SRAA. Evaluation with 18 blind subjects demonstrates its usability and effectiveness.

References

[1]
Alvarez, M., A. Pan, J. Raposo, F. Bellas, and F. Cacheda, Finding and extracting data records from web pages. Journal of Signal Processing Systems, 2010. 59(1): p. 123--137.
[2]
Amazon. Echo. 2015 {cited 2015}; Available from: http://www.amazon.com/echo.
[3]
Andreessen, M., NCSA Mosaic technical summary. National Center for Supercomputing Applications, 1993. 605.
[4]
Android, S Voice. 2012.
[5]
Ashok, V., Y. Borodin, Y. Puzis, and I.V. Ramakrishnan, Capti-Speak: A Speech-Enabled Web Screen Reader, in Proceedings of the 12th Web for All Conference. 2015, ACM: Florence, Italy.
[6]
Ashok, V., Y. Borodin, S. Stoyanchev, Y. Puzis, and I.V. Ramakrishnan, Wizard-of-Oz evaluation of speech-driven web browsing interface for people with vision impairments, in Proceedings of the 11th Web for All Conference. 2014, ACM: Seoul, Korea. p. 1--9.
[7]
Bigham, J.P., T. Lau, and J. Nichols, TrailBlazer: Enabling Blind Users to Blaze Trails Through the Web, in No Code Required: Giving Users Tools to Transform the Web, A. Cypher, et al., Editors. 2010, Morgan Kaufmann.
[8]
Bolin, M., M. Webber, P. Rha, T. Wilson, and R.C. Miller, Automation and customization of rendered web pages, in Proceedings of the 18th annual ACM symposium on User interface software and technology. 2005, ACM: Seattle, WA, USA. p. 163--172.
[9]
Borodin, Y., F. Ahmed, M.A. Islam, Y. Puzis, V. Melnyk, S. Feng, I.V. Ramakrishnan, and G. Dausch, Hearsay: a new generation context-driven multimodal assistive web browser, in Proceedings of the 19th international conference on World wide web. 2010, ACM: Raleigh, North Carolina, USA.
[10]
Brooke, J., SUS-A quick and dirty usability scale. Usability evaluation in industry, 1996. 189: p. 194.
[11]
Chang, D., L. Dooley, and J.E. Tuovinen. Gestalt theory in visual screen design: a new look at an old subject. in Proceedings of the Seventh world conference on computers in education conference on Computers in education: Australian topics-Volume 8. 2002: Australian Computer Society, Inc.
[12]
Djamasbi, S., M. Siegel, and T. Tullis, Generation Y, web design, and eye tracking. International Journal of Human-Computer Studies, 2010. 68(5): p. 307--323.
[13]
Google. Google Now. 2012 {cited 2015}; Available from: http://www.google.com/landing/now/ utm_source=google&utm_medium=sem&utm_campa ign=GoogleNow.
[14]
Hick, W.E., On the rate of gain of information. Quarterly Journal of Experimental Psychology, 1952. 4(1): p. 11--26.
[15]
House, D., D. Novick, M. Fanty, and J. Walpole, Spoken-Language Access to Multimedia (SLAM): Masters Thesis.
[16]
HTML5. Hyper-Text Markup Language v.5.0. 2010 {cited 2010}; Available from: http://dev.w3.org/html5/spec/.
[17]
JAWS. Screen reader from Freedom Scientific. 2013 {cited 2015}; Available from: http://www.freedomscientific.com/products/fs/jawsproduct-page.asp.
[18]
JSay. Interoperability for Dragon NaturallySpeaking Professional. 2015 {cited 2015}; Available from: http://www.ngtvoice.com/products/software/astec/jsay/.
[19]
Lau, T., J. Cerruti, G. Manzato, M. Bengualid, J.P. Bigham, and J. Nichols, A conversational interface to web automation, in Proceedings of the 23nd annual ACM symposium on User interface software and technology, ACM: NY, NY, USA.
[20]
Lazar, J., A. Allen, J. Kleinman, and C. Malarkey, What frustrates screen reader users on the web: A study of 100 blind users. International Journal of human-computer interaction, 2007. 22(3): p. 247--269.
[21]
Leshed, G., E.M. Haber, T. Matthews, and T. Lau. CoScripter: automating & sharing how-to knowledge in the enterprise. in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2008: ACM.
[22]
Li, I., J. Nichols, T. Lau, C. Drews, and A. Cypher, Here's what i did: sharing and reusing web activity with ActionShot, in Proceedings of the 28th international conference on Human factors in computing systems. 2010, ACM: Atlanta, Georgia, USA. p. 723--732.
[23]
MacKenzie, I.S., Fitts' law as a research and design tool in human-computer interaction. Human-computer interaction, 1992. 7(1): p. 91--139.
[24]
Mahmud, J.U., Y. Borodin, and I.V. Ramakrishnan, CSurf: a context-driven non-visual web-browser, in Proceedings of the 16th International Conference on World Wide Web. 2007, ACM: Banff, Alberta, Canada.
[25]
Melnyk, V., V. Ashok, Y. Puzis, A. Soviak, and Y. Borodin, Widget Classification with Applications to Web Accessibility, in International Conference on Web Engineering (ICWE). 2014.
[26]
Microsoft. Cortana contextual awareness. 2014 {cited 2015}; Available from: http://www.bing.com/dev/en-us/contextual-awareness.
[27]
Miller, R.C., V.H. Chou, M. Bernstein, G. Little, M.V. Kleek, D. Karger, and m. schraefel, Inky: a sloppy command line for the web with rich visual feedback, in Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology. 2008, ACM: Monterey, CA, USA.
[28]
Nuance. Nina. 2012 {cited 2015}; Available from: http://www.nuance.com/landingpages/products/nina/default.asp.
[29]
Nuance. Dragon NaturallySpeaking Rich Internet Application. 2014 {cited 2014}; Available from: http://nuance.custhelp.com/app/answers/detail/a_id/69 40/~/information-on-rich-internet-application-support.
[30]
Puzis, Y., Y. Borodin, R. Puzis, and I.V. Ramakrishnan, Predictive Web Automation Assistant for People with Vision Impairments, in proceedings of the 22th international conference on world wide web. 2013, ACM: Rio de Janeiro, Brazil.
[31]
Siri. The Personal Assistant on Your Phone. 2013 {cited 2015}; Available from: http://siri.com/.
[32]
WAI. W3C Web Accessibility Initiative. 1997 {cited 2010}; Available from: http://www.w3.org/WAI/.
[33]
Zhai, Y. and B. Liu. Web data extraction based on partial tree alignment. in Proceedings of the 14th international conference on World Wide Web. 2005: ACM.
[34]
Zhong, Y., T.V. Raman, C. Burkhardt, F. Biadsy, and J.P. Bigham, JustSpeak: enabling universal voice control on Android, in Proceedings of the 11th Web for All Conference. 2014, ACM: Seoul, Korea. p. 1--4.
[35]
Zhu, J., Z. Nie, J.-R. Wen, B. Zhang, and W.-Y. Ma. Simultaneous record detection and attribute labeling in web data extraction. in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. 2006: ACM.

Cited By

View all
  • (2024)All in One Place: Ensuring Usable Access to Online Shopping Items for Blind UsersProceedings of the ACM on Human-Computer Interaction10.1145/36646398:EICS(1-25)Online publication date: 17-Jun-2024
  • (2024)Enabling Uniform Computer Interaction Experience for Blind Users through Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675605(1-14)Online publication date: 27-Oct-2024
  • (2024)You Shall Know a Forum by the Words they Keep: Analyzing Language Use in Accessibility Forums for Blind UsersProceedings of the 35th ACM Conference on Hypertext and Social Media10.1145/3648188.3675151(230-238)Online publication date: 10-Sep-2024
  • Show More Cited By

Index Terms

  1. Web Screen Reading Automation Assistance Using Semantic Abstraction

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    IUI '17: Proceedings of the 22nd International Conference on Intelligent User Interfaces
    March 2017
    654 pages
    ISBN:9781450343480
    DOI:10.1145/3025171
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 07 March 2017

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. accessibility
    2. assistant
    3. automation
    4. blind users
    5. blindness
    6. natural interfaces
    7. screen-reader
    8. web browsing

    Qualifiers

    • Research-article

    Funding Sources

    Conference

    IUI'17
    Sponsor:

    Acceptance Rates

    IUI '17 Paper Acceptance Rate 63 of 272 submissions, 23%;
    Overall Acceptance Rate 746 of 2,811 submissions, 27%

    Upcoming Conference

    IUI '25

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)32
    • Downloads (Last 6 weeks)2
    Reflects downloads up to 17 Feb 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2024)All in One Place: Ensuring Usable Access to Online Shopping Items for Blind UsersProceedings of the ACM on Human-Computer Interaction10.1145/36646398:EICS(1-25)Online publication date: 17-Jun-2024
    • (2024)Enabling Uniform Computer Interaction Experience for Blind Users through Large Language ModelsProceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3663548.3675605(1-14)Online publication date: 27-Oct-2024
    • (2024)You Shall Know a Forum by the Words they Keep: Analyzing Language Use in Accessibility Forums for Blind UsersProceedings of the 35th ACM Conference on Hypertext and Social Media10.1145/3648188.3675151(230-238)Online publication date: 10-Sep-2024
    • (2024)Assessing the Accessibility and Usability of Web Archives for Blind UsersLinking Theory and Practice of Digital Libraries10.1007/978-3-031-72437-4_12(203-221)Online publication date: 26-Sep-2024
    • (2023)Enabling Customization of Discussion Forums for Blind UsersProceedings of the ACM on Human-Computer Interaction10.1145/35932287:EICS(1-20)Online publication date: 19-Jun-2023
    • (2023)AutoDesc: Facilitating Convenient Perusal of Web Data Items for Blind UsersProceedings of the 28th International Conference on Intelligent User Interfaces10.1145/3581641.3584049(32-45)Online publication date: 27-Mar-2023
    • (2023)Enabling Efficient Web Data-Record Interaction for People with Visual Impairments via Proxy InterfacesACM Transactions on Interactive Intelligent Systems10.1145/357936413:3(1-27)Online publication date: 11-Sep-2023
    • (2023)Making Everyday Objects More Inclusive: A Case Study via Remote Participatory DesignMobile and Ubiquitous Systems: Computing, Networking and Services10.1007/978-3-031-34776-4_31(583-597)Online publication date: 27-Jun-2023
    • (2022)“Every Website Is a Puzzle!”: Facilitating Access to Common Website Features for People with Visual ImpairmentsACM Transactions on Accessible Computing10.1145/351903215:3(1-35)Online publication date: 8-Jul-2022
    • (2022)Customizable Tabular Access to Web Data Records for Convenient Low-vision Screen Magnifier InteractionACM Transactions on Accessible Computing10.1145/351704415:2(1-22)Online publication date: 19-May-2022
    • Show More Cited By

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Figures

    Tables

    Media

    Share

    Share

    Share this Publication link

    Share on social media