William Payne
ABSTRACT
Technologies for notating music pose usage barriers to blind and visually impaired musicians requiring many to overcome a significant learning curve and/or rely on complicated tool chains with limited screen reader support. To address a need for accessible music notation software, we present SoundCells, a browser-based system designed to make music notation easy, intuitive, and accessible to screen reader users, and output music in audio, print, and braille formats. We share findings from a co-design process, in which two experienced musicians used SoundCells for two months guided by four remote meetings, and from a Design Probe, in which five other musicians tried SoundCells with a screen reader and reflected on its usability and accessibility in the context of their current practices. Finally, we discuss design recommendations relevant to a broader ecosystem of creative technologies, including how text-editing and multi-modal output capabilities could be extended and improved, how SoundCells' current design facilitated remote collaboration between sighted researchers and blind musicians, and future opportunities for learning and sharing music on the web.
- Samuel Aaron. 2020. Sonic Pi v3.2 Released! https://www.patreon.com/posts/sonic-pi-v3-2-34414152Google Scholar
- Samuel Aaron, Alan F Blackwell, and Pamela Burnard. 2016. The development of Sonic Pi and its use in educational partnerships: Co-creating pedagogies for learning computer programming. Journal of Music, Technology & Education 9, 1 (2016), 75--94.Google ScholarCross Ref
- Joseph Michael Abramo and Amy Elizabeth Pierce. 2013. An ethnographic case study of music learning at a school for the blind. Bulletin of the Council for Research in Music Education 195, 195 (2013), 9--24. Google ScholarCross Ref
- Fabiha Ahmed, Dennis Kuzminer, Michael Zachor, Lisa Ye, Rachel Josepho, William Christopher Payne, and Amy Hurst. 2021. Sound Cells: Rendering Visual and Braille Music in the Browser. In The 23rd International ACM SIGACCESS Conference on Computers and Accessibility (Virtual Event, USA) (ASSETS '21). Association for Computing Machinery, New York, NY, USA, Article 89, 4 pages. Google ScholarDigital Library
- Ameer Armaly, Paige Rodeghero, and Collin McMillan. 2018. AudioHighlight: Code Skimming for Blind Programmers. In 2018 IEEE International Conference on Software Maintenance and Evolution (ICSME) (Madrid, Spain). IEEE, 206--216. Google ScholarCross Ref
- Safa Arooj, Shaban Zulfiqar, Muhammad Qasim Hunain, Suleman Shahid, and Asim Karim. 2020. Web-ALAP: A Web-based LaTeX Editor for Blind Individuals. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1--6.Google Scholar
- Catherine M. Baker, Lauren R. Milne, and Richard E. Ladner. 2015. Struct Jumper: A Tool to Help Blind Programmers Navigate and Understand the Structure of Code. Association for Computing Machinery, New York, NY, USA, 3043--3052. Google ScholarDigital Library
- Jeanne Shapiro Bamberger and Armando Hernandez. 2000. Developing musical intuitions: A project-based introduction to making and understanding music. Oxford University Press. https://web.mit.edu/jbamb/www/dmi.htmlGoogle Scholar
- Bootstrap. 2021. Accessibility: A brief overview of Bootstrap's features and limitations for the creation of accessible content. Retrieved May 1, 2021 from https://getbootstrap.com/docs/5.1/getting-started/accessibility/Google Scholar
- Wes Bos. 2022. Keycode.info. Retrieved May 1, 2021 from https://keycode.infoGoogle Scholar
- Chambers, John. 2021. JC's ABC Tune Finder. Retrieved May 1, 2021 from http://trillian.mit.edu/~jc/cgi/abc/tunefindGoogle Scholar
- Katie Chan. 2021. p5 Showcase. Retrieved Dec 1, 2021 from https://showcase.p5js.org/Google Scholar
- Chi Kim and Rocco Fiorentino. 2020. Flo Tools: Enhanced Workflow for Pro Tools Users With Visual Impairments. Retrieved May 1, 2021 from http://flotools.orgGoogle Scholar
- Michael Scott Cuthbert and Christopher Ariza. 2010. music21: A toolkit for computer-aided musicology and symbolic music data. Proceedings of the 11th International Society for Music Information Retrieval Conference (ISMIR 2010) (2010), 637--642.Google Scholar
- Cuthbert, Michael Scott. 2021. music21 Braille Translate. Retrieved May 1, 2021 from https://web.mit.edu/music21/doc/moduleReference/moduleBrailleTranslate.htmlGoogle Scholar
- Dancing Dots. 2020. Dancing Dots: Accessible Music Technology for Blind and Low Vision Performers since 1992. https://www.dancingdots.com/Google Scholar
- Maitraye Das, Darren Gergle, and Anne Marie Piper. 2019. "It Doesn't Win You Friends": Understanding Accessibility in Collaborative Writing for People with Vision Impairments. Proc. ACM Hum.-Comput. Interact. 3, CSCW, Article 191 (Nov. 2019), 26 pages. Google ScholarDigital Library
- Brian Dorn and Mark Guzdial. 2006. Graphic Designers Who Program as Informal Computer Science Learners. In Proceedings of the Second International Workshop on Computing Education Research (Canterbury, United Kingdom) (ICER '06). Association for Computing Machinery, New York, NY, USA, 127--134. Google ScholarDigital Library
- Duxbury Systems. 2021. Duxbury DBT: Braille Translation Software. Retrieved May 1, 2021 from https://www.duxburysystems.comGoogle Scholar
- Alistair D. N. Edwards. 2011. Auditory display in assistive technology. In The Sonification Handbook, Thomas Hermann, Andy Hunt, and John G. Neuhoff (Eds.). Logos Publishing House, Berlin, Germany, Chapter 17, 431--453.Google Scholar
- Associazione Giuseppe Paccini ETS. 2021. Braille Music Editor. Retrieved May 1, 2021 from https://braillemusiceditor.com/Google Scholar
- Michael Fairchild. 2022. Accessibility Support: aria-live attribute. Retrieved Dec 1, 2022 from https://a11ysupport.io/tech/aria/aria-live_attributeGoogle Scholar
- Filomen M. D'Agostino Greenberg Music School. 2021. FMDG School: Fostering education, access, and inclusion for people of all ages with vision loss. Retrieved May 1, 2021 from https://fmdgmusicschool.orgGoogle Scholar
- Jason Freeman, Brian Magerko, Tom McKlin, Mike Reilly, Justin Permar, Cameron Summers, and Eric Fruchter. 2014. Engaging underrepresented groups in high school introductory computing through computational remixing with EarSketch. In Proceedings of the 45th ACM technical symposium on Computer science education. 85--90.Google ScholarDigital Library
- Emma Frid. 2019. Accessible digital musical instruments---A review of musical interfaces in inclusive music practice. Multimodal Technologies and Interaction 3, 3 (2019). Google ScholarCross Ref
- Emma Frid and Alon Ilsar. 2021. Reimagining (Accessible) Digital Musical Instruments: A Survey on Electronic Music-Making Tools. In Proceedings of the International Conference on New Interfaces for Musical Expression. Shanghai, China, Article 28. Google ScholarCross Ref
- Bill Gaver, Tony Dunne, and Elena Pacenti. 1999. Design: Cultural Probes. Interactions 6, 1 (Jan. 1999), 21--29. Google ScholarDigital Library
- GNU Project. 2021. LilyPond. Retrieved May 1, 2021 from https://lilypond.orgGoogle Scholar
- David Goldstein. 2000. Music pedagogy for the blind. International Journal of Music Education 35, 1 (5 2000), 35--39. Google ScholarCross Ref
- D. Goto, T. Gotoh, R. Minamikawa-Tachino, and N. Tamura. 2007. A Transcription System from MusicXML Format to Braille Music Notation. Eurasip Journal on Advances in Signal Processing 2007, 1 (Jan. 2007), 152. Google ScholarDigital Library
- Toshiyuki Gotoh, Reiko Minamikawa-Tachino, and Naoyoshi Tamura. 2008. A Web-Based Braille Translation for Digital Music Scores. In Proceedings of the 10th International ACM SIGACCESS Conference on Computers and Accessibility (Halifax, Nova Scotia, Canada) (Assets '08). Association for Computing Machinery, New York, NY, USA, 259--260. Google ScholarDigital Library
- Lucy Green and David Baker. 2017. Digital Music Technologies: The Changing Landscape. In Insights in Sound: Visually Impaired Musicians' Lives and Learning. Routledge, Chapter 9, 158--173. Google ScholarCross Ref
- Thomas Haenselmann, Hendrik Lemelson, Kerstin Adam, and Wolfgang Effelsberg. 2009. A tangible MIDI sequencer for visually impaired people. In Proceedings of the seventeen ACM international conference on Multimedia - MM '09. ACM Press, New York, New York, USA, 993. Google ScholarDigital Library
- Marijn Haverbeke. 2018. Code editor screen reader accessiblity survey. Retrieved May 1, 2021 from https://discuss.codemirror.net/t/code-editor-screen-reader-accessiblity-survey/1790Google Scholar
- Marijn Haverbeke. 2021. Screen reader response for backspace is flaky. Retrieved Dec 1, 2021 from https://github.com/codemirror/codemirror.next/issues/563Google Scholar
- Marijn Haverbeke. 2022. Code Mirror 6. Retrieved May 1, 2021 from https://codemirror.net/6Google Scholar
- Marijn Haverbeke. 2022. Lezer Parser System. Retrieved May 1, 2021 from https://lezer.codemirror.netGoogle Scholar
- Marijn Haverbeke. 2022. Screen Reader Demo. Retrieved May 1, 2021 from https://codemirror.net/6/screenreaderdemo/Google Scholar
- Jordie Howell. 2020. Innovation in braille music translation: Processes and transcription practices to produce more music. In 7th General Assembly of the International Council on English Braille (ICEB '20). International Council on English Braille. http://www.iceb.org/GA20.htmlGoogle Scholar
- Nadine Jessel. 2015. Access to musical information for Blind People. In International Conference on Technologies for Music Notation and Representation. Institut de Recherche en Musicologie (IReMus), 232--237.Google Scholar
- Aaron Karp and Bryan Pardo. 2017. HaptEQ. In Proceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences. ACM, New York, NY, USA, 1--4. Google ScholarDigital Library
- Lang, Mario. 2009. FreeDots: MusicXML to Braille Music translation. Retrieved May 1, 2021 from https://delysid.org/freedots.htmlGoogle Scholar
- Matthias Leopold. 2006. HODDER-a fully automatic braille note production system. In International Conference on Computers for Handicapped Persons. Springer, 6--11.Google ScholarDigital Library
- Alex Michael Lucas, Miguel Ortiz, and Dr. Franziska Schroeder. 2019. Bespoke Design for Inclusive Music: The Challenges of Evaluation. Proceedings of the International Conference on New Interfaces for Musical Expression (2019), 105-109. https://novationmusic.com/keys/launchkey%0Ahttp://www.nime.org/proceedings/2019/nime2019_021.pdfGoogle Scholar
- Stephanie Ludi and Mary Spencer. 2017. Design Considerations to Increase Block-based Language Accessibility for Blind Programmers Via Blockly. 3, 1 (July 2017), 119--124. Google ScholarCross Ref
- Kelly Mack, Maitraye Das, Dhruv Jain, Danielle Bragg, John Tang, Andrew Begel, Erin Beneteau, Josh Urban Davis, Abraham Glasser, Joon Sung Park, and Venkatesh Potluri. 2021. Mixed Abilities and Varied Experiences: A Group Autoethnography of a Virtual Summer Internship. Association for Computing Machinery, New York, NY, USA. Google ScholarDigital Library
- Yotam Mann. 2015. Interactive Music with Tone.js. In Proceedings of the International Web Audio Conference (WAC '15), Samuel Goldszmidt, Norbert Schnell, Victor Saiz, and Benjamin Matuszewski (Eds.). IRCAM, Paris, France.Google Scholar
- Christina Matawa. 2009. Exploring the musical interests and abilities of blind and partially sighted children and young people with Retinopathy of Prematurity. British Journal of Visual Impairment 27, 3 (9 2009), 252--262. Google ScholarCross Ref
- Sean Mealin and Emerson Murphy-Hill. 2012. An exploratory study of blind software developers. In 2012 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 71--74. Google ScholarCross Ref
- Lauren R. Milne and Richard E. Ladner. 2018. Blocks4All: Overcoming Accessibility Barriers to Blocks Programming for Children with Visual Impairments. Association for Computing Machinery, New York, NY, USA, 1--10. Google ScholarDigital Library
- Frederick W Moss, Jr. 2009. Quality of Experience in Mainstreaming and Full Inclusion of Blind and Visually Impaired High School Instrumental Music Students.Google Scholar
- Mozilla. 2022. ARIA live regions. Retrieved July 1, 2021 from https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Live_RegionsGoogle Scholar
- National Federation of the Blind. 2021. Braille Music Transcribers. Retrieved May 1, 2021 from https://nfb.org/programs-services/braille-certification/music-braille-transcribingGoogle Scholar
- National Library Service for the Blind and Print Disabled. 2021. Music Braille Transcribing. Retrieved May 1, 2021 from https://www.loc.gov/nls/braille-audio-reading-materials/music-materials/circular-no-4-braille-music-transcribers/Google Scholar
- Uran Oh, Shaun K. Kane, and Leah Findlater. 2013. Follow that sound. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS '13. ACM Press, New York, New York, USA, 1--8. Google ScholarDigital Library
- Shotaro Omori and Ikuko Eguchi Yairi. 2013. Collaborative music application for visually impaired people with tangible objects on table. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility - ASSETS '13. ACM Press, New York, New York, USA, 1--2. Google ScholarDigital Library
- OpenScore. 2018. OpenScore: One Year On. Retrieved May 1, 2021 from https://openscore.cc/blog/2018/8/20/openscore-one-year-onGoogle Scholar
- OSMD Team. 2021. Open Sheet Music Display. Retrieved May 1, 2021 from https://opensheetmusicdisplay.orgGoogle Scholar
- Sharon Oviatt. 2006. Human-centered design meets cognitive load theory. In Proceedings of the 14th annual ACM international conference on Multimedia - MULTIMEDIA '06. ACM Press, New York, New York, USA, 871. Google ScholarDigital Library
- Pallets. 2021. Flask: Web development, one drop at a time. Retrieved May 1, 2021 from https://flask.palletsprojects.com/en/2.0.x/Google Scholar
- William Payne, Alex Xu, Amy Hurst, and S. Alex Ruthmann. 2019. Non-visual beats: Redesigning the Groove Pizza. In ASSETS 2019 - 21st International ACM SIGACCESS Conference on Computers and Accessibility. Association for Computing Machinery, Inc, 651--654. Google ScholarDigital Library
- William Christopher Payne, Alex Yixuan Xu, Fabiha Ahmed, Lisa Ye, and Amy Hurst. 2020. How Blind and Visually Impaired Composers, Producers, and Songwriters Leverage and Adapt Music Technology. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1--12.Google ScholarDigital Library
- Petrucci Music Library. 2021. IMSLP: Sharing the world's public domain music. Retrieved May 1, 2021 from https://imslp.org/Google Scholar
- Pickering, Heydon. 2017. Inclusive Components Home. Retrieved Jan 1, 2022 from https://inclusive-components.design/tabbed-interfaces/Google Scholar
- Venkatesh Potluri, Priyan Vaithilingam, Suresh Iyengar, Y. Vidya, Manohar Swaminathan, and Gopal Srinivasa. 2018. CodeTalk: Improving Programming Environment Accessibility for Visually Impaired Developers. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1-11. Google ScholarDigital Library
- NYU Ability Project. 2021. SoundCells. Retrieved Dec 1, 2021 from https://soundcells.herokuapp.com/Google Scholar
- NYU Ability Project. 2021. SoundCells GitHub Repository. Retrieved Dec 1, 2021 from https://github.com/Huriphoonado/soundcellsGoogle Scholar
- Public Forum. 2020. Logic Accessibility Google Group. Retrieved May 1, 2020 from https://groups.google.com/forum/#!forum/logic-accessibilityGoogle Scholar
- Public Forum. 2020. Pro Tools Accessibility Google Group. Retrieved May 1, 2020 from https://groups.google.com/forum/#!forum/ptaccessGoogle Scholar
- Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Evelyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, et al. 2009. Scratch: programming for all. Commun. ACM 52, 11 (2009), 60--67.Google ScholarDigital Library
- Rosen, Paul and Dyke, Gregory. 2021. abcjs: Javascript for rendering abc music notation. Retrieved May 1, 2021 from https://www.abcjs.net/Google Scholar
- Toby W Rush. 2019. Braille Music Notator. Retrieved May 1, 2021 from https://tobyrush.com/braillemusic/notator/Google Scholar
- Marc Sabatella. 2013. ABC For Blind Musicians. Retrieved May 1, 2021 from https://accessiblemusicnotation.wordpress.com/2013/08/21/abc-for-blind-musicians/Google Scholar
- Abir Saha and Anne Marie Piper. 2020. Understanding Audio Production Practices of People with Vision Impairments. In The 22nd International ACM SIGACCESS Conference on Computers and Accessibility. 1--13.Google Scholar
- Harini Sampath, Alice Merrick, and Andrew Macvean. 2021. Accessibility of Command Line Interfaces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, Article 489, 10 pages. Google ScholarDigital Library
- Lawrence R. Smith, Karin Auckenthaler, Gilbert Busch, Karen Gearreald, Dan Geminder, Beverly McKenney, Harvey Miller, and Tom Ridgeway. 2015. Braille Music Code. American Printing House for the Blind, Louisville, Kentucky.Google Scholar
- Andreas Stefik, Roger Alexander, Robert Patterson, and Jonathan Brown. 2007. WAD: A Feasibility study using the Wicked Audio Debugger. In 15th IEEE International Conference on Program Comprehension (ICPC '07). 69--80. Google ScholarDigital Library
- Andreas Stefik, Andrew Haywood, Shahzada Mansoor, Brock Dunda, and Daniel Garcia. 2009. SODBeans. In 2009 IEEE 17th International Conference on Program Comprehension. 293--294. Google ScholarCross Ref
- Andreas Stefik, Christopher Hundhausen, and Robert Patterson. 2011. An empirical investigation into the design of auditory cues to enhance computer program comprehension. International Journal of Human-Computer Studies 69, 12 (2011), 820--838. Google ScholarDigital Library
- Kevin M Storer, Harini Sampath, and M. Alice Alice Merrick. 2021. "It's Just Everything Outside of the IDE That's the Problem": Information Seeking by Software Developers with Visual Impairments. Association for Computing Machinery, New York, NY, USA. Google ScholarDigital Library
- Atau Tanaka and Adam Parkinson. 2016. Haptic Wave. (2016), 2150--2161. Google ScholarDigital Library
- W3C Web Accessibility Initiative. 2022. WAI-ARIA Overview. Retrieved Dec 1, 2022 from https://www.w3.org/WAI/standards-guidelines/wcag/Google Scholar
- B. N. Walker and L. M. Mauney. 2010. Universal Design of Auditory Graphs. ACM Transactions on Accessible Computing 2, 3 (3 2010), 1--16. Google ScholarDigital Library
- Chris Walshaw. 2021. abc notation. Retrieved May 1, 2021 from https://abcnotation.com/Google Scholar
- World Wide Web Consortium. 2021. Web Audio API. Retrieved July 1, 2021 from https://www.w3.org/TR/webaudio/Google Scholar
- World Wide Web Consortium. 2021. Web Content Accessibility Guidelines (WCAG) Overview. Retrieved May 1, 2021 from https://www.w3.org/WAI/standards-guidelines/wcag/Google Scholar
Index Terms
- SoundCells: designing a browser-based music technology for braille and print notation
Recommendations
Empowering Blind Musicians to Compose and Notate Music with SoundCells
ASSETS '22: Proceedings of the 24th International ACM SIGACCESS Conference on Computers and AccessibilityCommercial technologies for notating music pose usage barriers to blind and visually impaired (BVI) musicians because they use graphic user interfaces and only produce visual, print scores. However, more research to date has studied how to make existing ...
Sound Cells: Rendering Visual and Braille Music in the Browser
ASSETS '21: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and AccessibilityMany blind musicians and composers read and write music using braille. Yet, braille music is not as widely available as print (visual) music, sighted collaborators and educators do not read braille music, and workflows and toolchains for converting ...
How Blind and Visually Impaired Composers, Producers, and Songwriters Leverage and Adapt Music Technology
ASSETS '20: Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and AccessibilityToday, music creation software and hardware are central to the workflow of most professional composers, producers, and songwriters. Music is an aural art form, but it is notated graphically, and highly visual mainstream technologies pose significant ...
Comments