Benjamin Petry
Thavishi Illandara
Don Samitha Elvitigala
Suranga Nanayakkara
ABSTRACT
Rhythm is the first musical concept deaf people learn in music classes. However, hearing loss limits the amount of information that allows a deaf person to evaluate his or her performance and stay in sync with other musicians. In this paper, we investigated how a visual and vibrotactile music-sensory-substitution device, MuSS-Bits++, affects rhythm discrimination, reproduction, and expressivity of deaf people. We conducted a controlled study with 11 deaf children and found that most participants felt more confident wearing the device in vibration mode even when it did not objectively improve their accuracy. Furthermore, we studied how MuSS-Bits++ can be used in music classes at deaf schools and what challenges and opportunities arise in such a setting. Based on these studies, we discuss insights and future directions that support the design and development of music-sensory-substitution systems for music making.
Supplemental Material
Available for Download
pdf
paper486.pdf (7.6 MB)
Published in the ACM DL on April 21, 2018 and replaced on July 5th, with approval of the CHI '18 chairs.
- 2017. Music and the Deaf. (March 2017). Retrieved March 6, 2017 from http://matd.org.uk/.Google Scholar
- Caterina Bertini, Fabrizio Leo, Alessio Avenanti, and Elisabetta Ladavas. 2010. Independent mechanisms for ventriloquism and multisensory integration as revealed by theta-burst stimulation. European Journal of Neuroscience 31, 10 (2010), 1791--1799.Google ScholarCross Ref
- John Brooke. 1996. SUS-A quick and dirty usability scale. Usability evaluation in industry 189, 194 (1996), 4--7.Google Scholar
- Marshal Chasin. 2003. Music and hearing aids. The Hearing Journal 56, 7 (July 2003), 36.Google ScholarCross Ref
- Summer Crider. 2009. Re-Defining Music Through Deaf Lens. Master's thesis. Gallaudet University, Washington, DC.Google Scholar
- Alice-Ann Darrow. 1993. The Role of Music in Deaf Culture: Implications for Music Educators. Journal of Research in Music Education 41, 2 (July 1993), 93--110.Google ScholarCross Ref
- Ward R. Drennan and Jay T. Rubinstein. 2008. Music perception in cochlear implant users and its relationship with psychophysical capabilities. Journal of rehabilitation research and development 45, 5 (2008), 779--789.Google ScholarCross Ref
- Wiliam G Fawkes. 2006. The Teaching of Music to Hearing Impaired Children and Teenagers. http://www.maryharehistory.org.uk/articles/fawkes/fawkes_2006.pdfGoogle Scholar
- Andy Field and Graham Hole. 2002. How to Design and Report Experiments. SAGE. Google-Books-ID: LN6QAwAAQBAJ.Google Scholar
- Sean Forbes. 2017. Sean Forbes. (2017). Retrieved March 6, 2017 from http://www.deafandloud.com/.Google Scholar
- David Fourney. 2015. Making The Invisible Visible: Visualization Of Music And Lyrics For Deaf And Hard Of Hearing Audiences. Ph.D. Dissertation. Ryerson University, Toronto, Ontario, Canada.Google Scholar
- John J. Galvin, Qian-Jie Fu, and Robert V. Shannon. 2009. Melodic Contour Identification and Music Perception by Cochlear Implant Users. Annals of the New York Academy of Sciences 1169, 1 (July 2009), 518--533.Google ScholarCross Ref
- Evelyn Glennie. 2003. How to truly listen. (2003). https://www.ted.com/talks/evelyn_glennie_shows_how_to_listenGoogle Scholar
- E Goldstein. 2009. Sensation and perception. Cengage Learning.Google Scholar
- Maria Karam, Carmen Branje, Gabe Nespoli, Norma Thompson, Frank A. Russo, and Deborah I. Fels. 2010. The Emoti-chair: An Interactive Tactile Music Exhibit. In Proc. CHI EA 2010. ACM, New York, NY, USA, 3069--3074. Google ScholarDigital Library
- M. Karam, F.A. Russo, and D.I. Fels. 2009. Designing the Model Human Cochlea: An Ambient Crossmodal Audio-Tactile Display. IEEE Transactions on Haptics 2, 3 (July 2009), 160--169. Google ScholarDigital Library
- Árni Kristjánsson, Alin Moldoveanu, Ómar I. Jóhannesson, Oana Balan, Simone Spagnol, Vigdís Vala Valgeirsdóttir, and Rúnar Unnthorsson. 2016. Designing sensory-substitution devices: Principles, pitfalls and potential. Restorative Neurology and Neuroscience 34, 5 (2016), 769--787.Google ScholarCross Ref
- Zefir Kurtisi, Xiaoyuan Gu, and Lars Wolf. 2006. Enabling Network-centric Music Performance in Wide-area Networks. Commun. ACM 49, 11 (Nov. 2006), 52--54. Google ScholarDigital Library
- Daniel J Levitin. 2011. This is your brain on music: Understanding a human obsession. Atlantic Books Ltd.Google Scholar
- Charles Limb. 2011. Building the musical muscle. (2011). Retrieved November 11, 2014 from http://www.ted.com/talks/charles_limb_building_the_musical_muscle.Google Scholar
- Suranga Nanayakkara, Elizabeth Taylor, Lonce Wyse, and S H. Ong. 2009. An Enhanced Musical Experience for the Deaf: Design and Evaluation of a Music Display and a Haptic Chair. In Proc. CHI 2009. ACM, New York, NY, USA, 337--346. Google ScholarDigital Library
- Carol A. Padden and Tom L. Humphries. 1990. Deaf in America: Voices from a Culture. Harvard University Press, Cambridge, Mass.Google Scholar
- Russ Palmer. 1994. Tac-Tile Sound System. (1994). http://www.russpalmer.com/tactile.htmlGoogle Scholar
- Benjamin Petry, Jochen Huber, and Suranga Nanayakkara. 2018. Scaffolding the Music Listening and Music Making Experience for the Deaf. In Assistive Augmentation. Springer, Singapore, 23--48. https://link. springer.com/chapter/10.1007/978--981--10--6404--3_3Google Scholar
- Benjamin Petry, Thavishi Illandara, Juan Pablo Forero, and Suranga Nanayakkara. 2016b. Ad-Hoc Access to Musical Sound for Deaf Individuals. In Proc. ASSETS 2016. ACM, New York, NY, USA, 285--286. Google ScholarDigital Library
- Benjamin Petry, Thavishi Illandara, and Suranga Nanayakkara. 2016a. MuSS-Bits: Sensor-Display Blocks for Deaf People to Explore Musical Sounds. In Proc. OzCHI 2016. ACM, New York, NY, USA, 72--80. Google ScholarDigital Library
- Dirk-Jan Povel and Peter Essens. 1985. Perception of Temporal Patterns. Music Perception: An Interdisciplinary Journal 2, 4 (July 1985), 411--440. http://mp.ucpress.edu/content/2/4/411Google ScholarCross Ref
- Janine Roebuck. 2007. I am a deaf opera singer. (Sept. 2007). Retrieved August 26, 2015 from https://www.theguardian.com/theguardian/2007/sep/29/weekend7.weekend2.Google Scholar
- Gottfried Schlaug. 2015. Musicians and music making as a model for the study of brain plasticity. In Progress in Brain Research. Music, Neurology, and Neuroscience: Evolution, the Musical Brain, Medical Conditions, and Therapies, Vol. 217. Elsevier, 37--55.Google Scholar
- Gottfried Schlaug, Eckart Altenmüller, and Michael Thaut. 2010. Music listening and music making in the treatment of neurological disorders and impairments. Music Perception: An Interdisciplinary Journal 27, 4 (2010), 249--250.Google ScholarCross Ref
- Nathan Schuett. 2002. The effects of latency on ensemble performance. Bachelor Thesis, CCRMA Department of Music, Stanford University (2002).Google Scholar
- Andrew Sears and Vicki L. Hanson. 2012. Representing Users in Accessibility Research. ACM Trans. Access. Comput. 4, 2 (March 2012), 7:1--7:6. Google ScholarDigital Library
- D Shibata. 2001. Brains of deaf people hear music. International Arts-Medicine Association Newsletter 16 (2001), 4.Google Scholar
- Kristen Shinohara and Josh Tenenberg. 2009. A Blind Person's Interactions with Technology. Commun. ACM 52, 8 (Aug. 2009), 58--66. Google ScholarDigital Library
- Carl Stumpf. 1883. Tonpsychologie. Leipzig: Hirzel 1 (1883).Google Scholar
- Joachim Taelman, S. Vandeput, A. Spaepen, and S. Van Huffel. 2009. Influence of Mental Stress on Heart Rate and Heart Rate Variability. In 4th European Conference of the International Federation for Medical and Biological Engineering. Springer, Berlin, Heidelberg, 1366--1369.Google Scholar
- Eric Thul. 2008. Measuring the complexity of musical rhythm. Ph.D. Dissertation. McGill University.Google Scholar
- Godfried T Toussaint. 2004. A Comparison of Rhythmic Similarity Measures. In Proc. ISMIR 2001.Google Scholar
- Mitchell Tyler, Yuri Danilov, and Paul Bach-Y-Rita. 2003. Closing an open-loop control system: vestibular substitution through the tongue. Journal of Integrative Neuroscience 02, 02 (Dec. 2003), 159--164.Google ScholarCross Ref
- Ronald T Verillo. 1991. Vibration sensing in humans. Music Perception 9, 3 (1991), 281--302.Google ScholarCross Ref
- Georg Von Bekesy. 1959. Similarities between hearing and skin sensations. Psychological Review 66, 1 (1959), 1--22.Google ScholarCross Ref
- Yinsheng Zhou, Khe Chai Sim, Patsy Tan, and Ye Wang. 2012. MOGAT: Mobile Games with Auditory Training for Children with Cochlear Implants. In Proc. MM 2012. ACM, New York, NY, USA, 429--438. Google ScholarDigital Library
Index Terms
- Supporting Rhythm Activities of Deaf Children using Music-Sensory-Substitution Systems
Recommendations
Ad-Hoc Access to Musical Sound for Deaf Individuals
ASSETS '16: Proceedings of the 18th International ACM SIGACCESS Conference on Computers and AccessibilityLearning a musical instrument can be a challenging task for a deaf person due to limited access to sound. Prior work has developed visual and vibrotactile approaches to provide music-to-sound feedback to deaf people. However, these systems are not ...
MuSS-bits: sensor-display blocks for deaf people to explore musical sounds
OzCHI '16: Proceedings of the 28th Australian Conference on Computer-Human InteractionHearing loss makes learning a musical instrument a challenging task. Prior work suggests that a universal sensory substitution system that works uniformly across all deaf users may not exist given the diversity within the deaf community. In this paper, ...
An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair
CHI '09: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsMusic is a multi-dimensional experience informed by much more than hearing alone, and is thus accessible to people of all hearing abilities. In this paper we describe a prototype system designed to enrich the experience of music for the deaf by ...
Comments