Haptic Vibrations for Hearing Impaired to Experience Aspects of Live Music

Abstract

Listening to music contributes to community building and benefits mental health. The ability of hearing-impaired and deaf individuals to experience and benefit from music is hindered. Musical instruments vibrate to generate sound. The tactile feelings of these vibrations were utilized by deaf historical figures of Beethoven and Hellen Keller. Snakes have a keen sense of tactile sensations induced by both ground-borne and airborne vibrations, and the induced neural processing occurs in both the somatic system and the auditory cortex. Congenitally deaf individuals have been shown to have an enhanced ability to process tactile information, including processing in the auditory cortex. Wearable garments and furniture have recently been developed to convert audible sounds and music into vibrations to enable hearing-impaired and deaf individuals to have some level of experience. These devices show promise, but more work needs to be done to improve the conversion of sound into haptic vibrations in a more meaningful way. The objective of our project was to develop a wearable electronic system that would extract volume and frequency features from the recent moments of live music, and to use that information to generate vibrations at multiple locations on the skin toward enabling an experience of the music. A prototype was designed and developed. Several submodule tests were conducted to evaluate functionality. A human-subject pilot test was conducted to evaluate whether the vibration pattern would relate to the music, and possibly help to distinguish types of songs that had different genre. In the test, subjects were tasked with selecting which song was being used by the systems to generate the haptic vibrations. The subjects appeared to be only slightly more accurate in their song selection than would be expected by chance. The system shows promise, but more development and testing would be required toward wider application.