research-article

Storytelling to Sensemaking: A Systematic Framework for Designing Auditory Description Display for Interactives

Authors:

Taliesin L. Smith,

Emily B. MooreAuthors Info & Claims

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Pages 1 - 12

https://doi.org/10.1145/3313831.3376460

Published: 23 April 2020 Publication History

Abstract

Auditory description display is verbalized text typically used to describe live, recorded, or graphical displays to support access for people who are blind or visually impaired. Significant prior research has resulted in guidelines for auditory description for non-interactive or minimally interactive contexts. A lack of auditory description for complex interactive environments remains a tremendous barrier to access for people with visual impairments. In this work, we present a systematic design framework for designing auditory description within complex interactive environments. We illustrate how modular descriptions aligned with this framework can result in an interactive storytelling experience constructed through user interactions. This framework has been used in a set of published and widely used interactive science simulations, and in its generalized form could be applied to a variety of contexts.

Supplemental Material

MP4 File

Preview video

Download
1.57 MB

SBV File

Preview video captions

Download
.68 KB

References

[1]

Diagram Center. 2012. POET Training Tool, A Benetech Inititiative. (2012). https://poet.diagramcenter.org/index.html

[2]

Diagram Center. 2015. Image Description Guidelines. (2015). http://diagramcenter.org/table-of-contents-2.html

[3]

Joanmarie Diggs, Shane McCarron, Michael Cooper, Richard Schwerdtfeger, and James Craig. 2017. Accessible Rich Internet Applications (WAI-ARIA) 1.1. (Dec. 2017). https://www.w3.org/TR/wai-aria-1.1/

[4]

M. Fakrudeen, M. Ali, S. Yousef, and A. H. Hussein. 2013. Analysing the Mental Modal of Blind Users in Mobile Touch Screen Devices for Usability. In Proceedings of the World Congress on Engineering, Vol. 2. IAENG, Springer, London, U.K., unknown.

[5]

Steve Faulkner, Arron Eicholz, Travis Leithead, Alex Danilo, and Sangwhan Moon. 2017. HTML 5.2. (Dec. 2017). https://www.w3.org/TR/html52/

[6]

Leo Ferres, Gitte Lindgaard, Livia Sumegi, and Bruce Tsuji. 2013. Evaluating a Tool for Improving Accessibility to Charts and Graphs. ACM Transactions on Computer-Human Interaction 20, 5 (Nov. 2013), 1--32.

Digital Library

[7]

Donal Fitzpatrick, A. Jonathan R. Godfrey, and Volker Sorge. 2017. Producing Accessible Statistics Diagrams in R. In W4A '17 Proceedings of the 14th Web for All Conference on the The Future of Accessible Work. ACM Press, Perth, Australia, 1--4.

Digital Library

[8]

Nahum Gershon and Ward Page. 2001. What storytelling can do for information visualization. Commun. ACM 44, 8 (Aug. 2001), 31--37.

Digital Library

[9]

Andrew Holland. 2009. Audio Description in the Theatre and the Visual Arts: Images into Words. In Audiovisual Translation, Jorge Dìaz Cintas and Gunilla Anderman (Eds.). Palgrave Macmillan UK, London, 170--185.

[10]

T. de Jong. 2006. COMPUTER SIMULATIONS: Technological Advances in Inquiry Learning. Science 312, 5773 (April 2006), 532--533.

[11]

Kyle Keane. 2014. Interactive Scientific Graphics Recommended Practices for Verbal Description. Research. Wolfram Research, Inc., Champaign, IL. 53 pages. http://diagramcenter.org/ accessible-dynamic-scientific-graphics.html

[12]

Edward Kim and Kathleen F. McCoy. 2018. Multimodal Deep Learning using Images and Text for Information Graphic Classification. In Proceedings of the 20th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS '18. ACM Press, Galway, Ireland, 143--148.

Digital Library

[13]

Andrew Kirkpatrick, Joshua O Connor, Alistair Campbell, and Michael Cooper. 2018. Web Content Accessibility Guidelines (WCAG) 2.1. (June 2018). https://www.w3.org/TR/WCAG21/

[14]

Sri H. Kurniawan, Alistair G. Sutcliffe, Paul L. Blenkhorn, and Jae-Eun Shin. 2003. Investigating the usability of a screen reader and mental models of blind users in the Windows environment. International Journal of Rehabilitation Research 26, 2 (2003), 145--147. http://journals.lww.com/intjrehabilres/ Abstract/2003/06000/Investigating_the_usability_of_a_ screen_reader_and.11.aspx

[15]

Oussama Metatla, Alison Oldfield, Taimur Ahmed, Antonis Vafeas, and Sunny Miglani. 2019. Voice User Interfaces in Schools: Co-designing for Inclusion with Visually-Impaired and Sighted Pupils. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems - CHI '19. ACM Press, Glasgow, Scotland Uk, 1--15.

Digital Library

[16]

Emily B. Moore, Taliesin L. Smith, and Emily Randall. 2016. Exploring the Relationship Between Implicit Scaffolding and Inclusive Design in Interactive Science Simulations. In Universal Access in Human-Computer Interaction. Users and Context Diversity, Margherita Antona and Constantine Stephanidis (Eds.). Lecture Notes in Computer Science, Vol. 9739. Springer International Publishing, Cham, 112--123. http://link.springer.com/10.1007/978--3--319--40238--3_12

[17]

Elizabeth D. Mynatt and W. Keith Edwards. 1995. Audio GUIs: interacting with graphical applications in an auditory world. In Conference companion on Human factors in computing systems - CHI '95. ACM Press, Denver, Colorado, United States, 85--86.

Digital Library

[18]

Donald A. Norman. 1987. Some observations on mental models. In Readings in human-computer interaction: a multidisciplinary approach, Ronald M. Baecker and William A. S. Buxton (Eds.). M. Kaufmann, Los Altos, Calif, 241--244.

[19]

Tor Ole B. Odden and Rosemary S. Russ. 2018. Defining sensemaking: Bringing clarity to a fragmented theoretical construct. Science Education 103, 1 (Jan. 2018), 187--205.

[20]

Jaclyn Packer, Katie Vizenor, and Joshua A. Miele. 2015. An Overview of Video Description: History, Benefits, and Guidelines. Journal of Visual Impairment & Blindness 109, 2 (March 2015), 83--93.

[21]

PhET Interactive Simulations. 2008. PhET Simulation Design Process. (2008). https://phet.colorado.edu/ publications/phet_design_process.pdf

[22]

PhET Interactive Simulations. 2015. Research and Design, Accessibility. (2015). https://phet.colorado.edu/en/accessibility/research

[23]

PhET Interactive Simulations. 2019a. A11y View: Resistance in a Wire. (Dec. 2019). https://phet-dev.colorado.edu/html/resistance-in-a-wire/1. 7.0-dev.4/phet/resistance-in-a-wire_a11y_view.html

[24]

PhET Interactive Simulations. 2019b. Balloons and Static Electricity. (Oct. 2019). http://phet.colorado. edu/en/simulation/balloons-and-static-electricity Version 1.4.14.

[25]

PhET Interactive Simulations. 2019c. Friction. (Oct. 2019). https://phet.colorado.edu/en/simulation/friction Version 1.5.10.

[26]

PhET Interactive Simulations. 2019d. Gravity Force Lab: Basics. (Oct. 2019). https://phet.colorado.edu/en/ simulation/gravity-force-lab-basics Version 1.0.0.

[27]

PhET Interactive Simulations. 2019e. John Travoltage. (Oct. 2019). https://phet.colorado.edu/en/simulation/john-travoltage Version 1.5.12.

[28]

PhET Interactive Simulations. 2019f. Resistance in a Wire. (Oct. 2019). http://phet.colorado.edu/en/ simulation/resistance-in-a-wire Version 1.6.9.

[29]

William Pike and Mark Gahegan. 2007. Beyond ontologies: Toward situated representations of scientific knowledge. International Journal of Human-Computer Studies 65, 7 (July 2007), 674--688.

Digital Library

[30]

William A. Pike, John Stasko, Remco Chang, and Theresa A. O'Connell. 2009. The Science of Interaction. Information Visualization 8, 4 (Jan. 2009), 263--274.

Digital Library

[31]

Noah S. Podolefsky, Emily B. Moore, and Katherine K. Perkins. 2013. Implicit scaffolding in interactive simulations: Design strategies to support multiple educational goals. arXiv preprint arXiv:1306.6544 (2013). http://arxiv.org/abs/1306.6544

[32]

DCMP (Described and Captioned Media Program). 1994. Captioning Key. (1994). https://www.captioningkey.org/quality_captioning.html

[33]

Jaime Sànchez, Nelson Baloian, Tiago Hassler, and Ulrich Hoppe. 2003. AudioBattleship: blind learners collaboration through sound. In CHI '03 extended abstracts on Human factors in computing systems CHI '03. ACM Press, Ft. Lauderdale, Florida, USA, 798.

Digital Library

[34]

Taliesin Smith, Michael Kauzmann, and PhET Interactive Simulations. 2019. Dynamic Phrase Manipulator. (June 2019). https://htmlpreview.github.io/?https://github.com/phetsims/a11y-research/blob/master/js/ phrase-builder/phrase-builder.html

[35]

Taliesin L. Smith. 2016. Access, Action, & Agency: Inclusive Design for the Non-visual Use of a Highly Interactive Simulation. Master Research Project. OCAD University, Toronto, Ontario. http://openresearch.ocadu.ca/id/eprint/713

[36]

Taliesin L. Smith, Jesse Greenberg, Sam Reid, and Emily B. Moore. 2018. Parallel DOM Architecture for Accessible Interactive Simulations. In Proceedings of the Internet of Accessible Things on - W4A '18. ACM Press, Lyon, France, 1--8.

Digital Library

[37]

Taliesin L. Smith, Clayton Lewis, and Emily B. Moore. 2016. Demonstration: Screen Reader Support for a Complex Interactive Science Simulation. In Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility. ACM New York, NY, USA, Reno, Nevada, USA, 319--320.

Digital Library

[38]

Taliesin L. Smith, Clayton Lewis, and Emily B. Moore. 2017. Description Strategies to Make an Interactive Science Simulation Accessible. Journal on Technology & Persons with Disabilities 5, 22 (2017), 225--238. http://scholarworks.calstate.edu/handle/10211.3/190214

[39]

Volker Sorge, Mark Lee, and Sandy Wilkinson. 2015. End-to-end solution for accessible chemical diagrams. In Proceedings of the 12th Web for All Conference. ACM Press, Florence, Italy, 1--10.

Digital Library

[40]

Brianna J. Tomlinson, Prakriti Kaini, Bruce N. Walker, Jared M. Batterman, and Emily B. Moore. 2018. Supporting Simulation Use for Students with Intellectual and Developmental Disabilities. Journal on Technology & Persons with Disabilities Volume 6, 2018 (2018), 202 -- 218. http://hdl.handle.net/10211.3/202996

[41]

Brianna J. Tomlinson, Bruce N. Walker, and Emily B. Moore. 2020. Auditory Display in Interactive Science Simulations: Description and Sonifcation Support Interaction and Enhance Opportunities for Learning. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. ACM, Honolulu, HI, USA.

Digital Library

[42]

Hong Zou and Jutta Treviranus. 2015. ChartMaster: A Tool for Interacting with Stock Market Charts using a Screen Reader. In Proceedings of the 17th International ACM SIGACCESS Conference on Computers & Accessibility. ACM New York, NY, USA, Lisbon, Portugal, 107--115.

Digital Library

Cited By

Taşabat SÖzçay T(2025)Pushing the Limits of CreativityCombating Cyberbullying With Generative AI10.4018/979-8-3373-0543-1.ch002(31-62)Online publication date: 14-Feb-2025
https://doi.org/10.4018/979-8-3373-0543-1.ch002
Fiedler BSmith TGreenberg JEisenberg AMoore E(2024)Insights from Youth Co-designers on Remote Multimodal Prototyping with Paper PlaygroundProceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3659400(818-822)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3628516.3659400
Zhang LStangl ASharma TTseng YXu IGurari DWang YFindlater L(2024)Designing Accessible Obfuscation Support for Blind Individuals’ Visual Privacy ManagementProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642713(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642713
Show More Cited By

Index Terms

Storytelling to Sensemaking: A Systematic Framework for Designing Auditory Description Display for Interactives
1. Human-centered computing
  1. Accessibility
    1. Accessibility design and evaluation methods
    2. Accessibility systems and tools

Recommendations

The experience of interactive storytelling: comparing “fahrenheit” with “façade”
ICEC'11: Proceedings of the 10th international conference on Entertainment Computing

At the intersection of multimedia, artificial intelligence, and gaming technology, new visions of future entertainment media arise that approximate the “Holodeck” ® idea of interactive storytelling. We report exploratory experiments on the user ...
Immersive auditory display system 'sound cask': three-dimensional sound field reproduction system based on the boundary surface control principle
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology

Sound cask was developed to realize the perfect 3D auditory display that creates 3D sound waves around the listener's head just the same as the primary sound field, based on the boundary surface control (BoSC) principle.

If we consider the sound ...
Designing Storytelling Games That Encourage Narrative Play
ICIDS '09: Proceedings of the 2nd Joint International Conference on Interactive Digital Storytelling: Interactive Storytelling

Storytelling games are a form of competitive storytelling framed in the context of gameplay. However, most existing storytelling games emphasize competitive gameplay and winning at the expense of competitive <em>narrative</em> play; they tend to be ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

April 2020

10688 pages

ISBN:9781450367080

DOI:10.1145/3313831

General Chairs:
Regina Bernhaupt
Eindhoven University of Technology, Netherlands
,
Florian 'Floyd' Mueller
Monash University, Australia
,
David Verweij
Newcastle University, UK
,
Josh Andres
RMIT, Australia
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
Andy Cockburn
University of Canterbury, New Zealand
,
Ignacio Avellino
University of Maryland Baltimore County, USA
,
Alix Goguey
Grenoble Alpes University, France
,
Pernille Bjørn
University of Copenhagen, Denmark
,
Shengdong (Shen) Zhao
National University of Singapore, Singapore
,
Briane Paul Samson
Future University Hakodate, Japan & De La Salle University, Philippines
,
Rafal Kocielnik
University of Washington, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

National Science Foundation

Conference

CHI '20

Sponsor:

SIGCHI

CHI '20: CHI Conference on Human Factors in Computing Systems

April 25 - 30, 2020

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
794
Total Downloads

Downloads (Last 12 months)206
Downloads (Last 6 weeks)36

Reflects downloads up to 08 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Taşabat SÖzçay T(2025)Pushing the Limits of CreativityCombating Cyberbullying With Generative AI10.4018/979-8-3373-0543-1.ch002(31-62)Online publication date: 14-Feb-2025
https://doi.org/10.4018/979-8-3373-0543-1.ch002
Fiedler BSmith TGreenberg JEisenberg AMoore E(2024)Insights from Youth Co-designers on Remote Multimodal Prototyping with Paper PlaygroundProceedings of the 23rd Annual ACM Interaction Design and Children Conference10.1145/3628516.3659400(818-822)Online publication date: 17-Jun-2024
https://dl.acm.org/doi/10.1145/3628516.3659400
Zhang LStangl ASharma TTseng YXu IGurari DWang YFindlater L(2024)Designing Accessible Obfuscation Support for Blind Individuals’ Visual Privacy ManagementProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642713(1-19)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642713
Saini MSengupta E(2024)Artificial intelligence inspired fog-cloud-based visual-assistance framework for blind and visually-impaired peopleMultimedia Tools and Applications10.1007/s11042-024-20159-1Online publication date: 23-Sep-2024
https://doi.org/10.1007/s11042-024-20159-1
Boadi-Agyemang ACarter ESiu ASteinfeld AOrta Martinez M(2023)Understanding Experiences, Attitudes and Perspectives towards Designing Interactive Creative Tools for Teachers of Visually Impaired StudentsProceedings of the 25th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3597638.3614512(1-5)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3597638.3614512
Hill CHunt CCrowder SFiedler BMoore EEisenberg A(2023)Investigating Sensory Extensions as Input for Interactive SimulationsProceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction10.1145/3569009.3573108(1-7)Online publication date: 26-Feb-2023
https://dl.acm.org/doi/10.1145/3569009.3573108
Fiedler BSmith TGreenberg JMoore EAhmetovic DYaneva V(2022)For one or for all?Proceedings of the 19th International Web for All Conference10.1145/3493612.3520456(1-11)Online publication date: 25-Apr-2022
https://dl.acm.org/doi/10.1145/3493612.3520456
Holloway LGoncu CIlsar AButler MMarriott K(2022)Infosonics: Accessible Infographics for People who are Blind using Sonification and VoiceProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3517465(1-13)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3517465
Wirfs-Brock JFam ADevendorf LKeegan B(2021)Examining Narrative Sonification: Using First-Person Retrospection Methods to Translate Radio Production to Interaction DesignACM Transactions on Computer-Human Interaction10.1145/346176228:6(1-34)Online publication date: 15-Nov-2021
https://dl.acm.org/doi/10.1145/3461762
Tomlinson BWalker BMoore EBernhaupt RMueller FVerweij DAndres JMcGrenere JCockburn AAvellino IGoguey ABjørn PZhao SSamson BKocielnik R(2020)Auditory Display in Interactive Science Simulations: Description and Sonification Support Interaction and Enhance Opportunities for LearningProceedings of the 2020 CHI Conference on Human Factors in Computing Systems10.1145/3313831.3376886(1-12)Online publication date: 21-Apr-2020
https://dl.acm.org/doi/10.1145/3313831.3376886

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten