Skip to main content
Springer Nature Link
Log in

Multisensory Experience for People with Hearing Loss: A Preliminary Study Using Haptic Interfaces to Sense Music

  • Conference paper
  • First Online:
HCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging (HCII 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13521))

Included in the following conference series:

Abstract

In this paper, we describe a preliminary study on a multisensory music experience for people with hearing loss. Our main goal is to provide a music event through visual and tactile stimuli, granting a multisensory experience using haptic interfaces and taking advantage of visual feedback, vibrations and pressure to induce feelings. In this context, a mobile application was developed, allowing the user to interact with recorded audio samples that exploit vibrations to trigger emotions, such as fear, adrenaline, anxiety, suspense, drama, adventure, or even more complex moods like when dancing and relaxing. We thus describe our methodology (design, implementation and user assessment) for a preliminary study of a music experience based on a user-centered design approach. Indeed, we gathered promising results as the experience was considered effective and satisfying. We also uncovered some development issues to be addressed in future work, having to do with the use of specific hardware for providing a fully immersive experience.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Stephanidis, C.: User interfaces for all: new perspectives into human-computer interaction. In: Stephanidis, C. (ed.) User Interfaces for All - Concepts, Methods, and Tools, p. 760. Lawrence Erlbaum Associates, Mahwah, NJ (2001)

    Google Scholar 

  2. Reis, A., et al.: Developing a system for post-stroke rehabilitation: an exergames approach. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2016. LNCS, vol. 9739, pp. 403–413. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-40238-3_39

    Chapter  Google Scholar 

  3. Constantine, S., Pier, L.E.: `Connecting’ to the information society: a European perspective. Technology and Disability 10(1). IOS Press, pp. 21–44, Jan. 01, 1999. (2015). http://content.iospress.com/articles/technology-and-disability/tad00013

  4. Stephanidis, C., Salvendy, G.: Toward an information society for all: an international research and development agenda. Int. J. Human-Computer Interaction 10(2), 107–134 (1998). https://doi.org/10.1207/s15327590ijhc1002_2

    Article  Google Scholar 

  5. Stephanidis, C., et al.: Toward an information society for all: HCI challenges and R&D recommendations. Int. J. Human-Computer Interaction 11(1), 1–28 (1999). https://doi.org/10.1207/s15327590ijhc1101_1

    Article  Google Scholar 

  6. World Health Organization (WHO): Deafness and hearing loss. https://www.who.int/en/news-room/fact-sheets/detail/deafness-and-hearing-loss Accessed 8 Jan 2022

  7. World Health Organization (WHO): International Classification of Diseases (ICD). https://www.who.int/standards/classifications/classification-of-diseases Accessed 8 Jan 2022

  8. Martins, P., Rodrigues, H., Rocha, T., Francisco, M., Morgado, L.: Accessible options for deaf people in e-learning platforms: technology solutions for sign language translation. Procedia Computer Sci. 67, 263–272 (2015). https://doi.org/10.1016/J.PROCS.2015.09.270

    Article  Google Scholar 

  9. Friedner, M., Helmreich, S.: Sound Studies Meets Deaf Studies. 7(1), 72–86 (2015). https://doi.org/10.2752/174589312X13173255802120

  10. Manaf, M.B.A., Sulaiman, S.B.: Integrating vibration sensing and non-speech visualization to notify hearing impaired students on fire in a controlled situation. In: 2015 International Symposium on Mathematical Sciences and Computing Research, iSMSC 2015 - Proceedings, pp. 36–41 (2016). https://doi.org/10.1109/ISMSC.2015.7594024

  11. Holmes, J.A.: Expert listening beyond the limits of hearing: music and deafness. J. Am. Musicol. Soc. 70(1), 171–220 (2017). https://doi.org/10.1525/JAMS.2017.70.1.171

    Article  Google Scholar 

  12. Goodman, S.: Sonic Warfare : Sound, Affect, and the Ecology of Fear. MIT Press (2010)

    Google Scholar 

  13. Srinivasan, M.A., Basdogan, C.: Haptics in virtual environments: taxonomy, research status, and challenges. Comput. Graph. 21(4), 393–404 (1997). https://doi.org/10.1016/S0097-8493(97)00030-7

    Article  Google Scholar 

  14. Carvalho, D., Bessa, M., Magalhães, L., Carrapatoso, E.: Age Group Differences in Performance Using Diverse Input Modalities: Insertion Task Evaluation (2016)

    Google Scholar 

  15. Carvalho, D., Bessa, M., Magalhães, L.: Different interaction paradigms for different user groups: an evaluation regarding content selection. In: Proceedings of the XV International Conference on Human Computer Interaction – Interaccion’14, p. 40 (2014). https://doi.org/10.1145/2662253.2662293

  16. Hale, K.S., Stanney, K.M.: Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations. IEEE Comput. Graphics Appl. 24(2), 33–39 (2004). https://doi.org/10.1109/MCG.2004.1274059

    Article  Google Scholar 

  17. Khan, M., Sulaiman, S., Said, M.D.A., Tahir, M.: Exploring the quantitative and qualitative measures for haptic systems. In: 2010 International Symposium on Information Technology, pp. 31–36 (2010). https://doi.org/10.1109/ITSIM.2010.5561305

  18. Li, Y., Johnson, S., Nam, C.: Haptically enhanced user interface to support science learning of visually impaired. In: Jacko, J.A. (ed.) HCI 2011. LNCS, vol. 6764, pp. 68–76. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21619-0_10

    Chapter  Google Scholar 

  19. Johnson, S., Li, Y., Nam, C.S., Yamaguchi, T.: Analyzing user behavior within a haptic system. In: Jacko, J.A. (ed.) HCI 2011. LNCS, vol. 6762, pp. 62–70. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21605-3_7

    Chapter  Google Scholar 

  20. Ichiyanagi, Y., Cooper, E.W., Kryssanov, V.V., Ogawa, H.: A haptic emotional model for audio system interface. In: Jacko, J.A. (ed.) HCI 2011. LNCS, vol. 6763, pp. 535–542. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-21616-9_60

    Chapter  Google Scholar 

  21. Freedman, J.: At the Heart of Leadership: How To Get Results with Emotional Intelligence. 3rd edition. Six Seconds (2012)

    Google Scholar 

  22. Ekman, P., Freisen, W., Ancoli, S.: Facial signs of emotional experience. J. Personality Social Psychology 39(6), 1125–1134 (1980). https://doi.org/10.1037/h0077722

  23. Plutchik, R.: A general psychoevolutionary theory of emotion. Theories of Emotion, pp. 3–33 (1980). https://doi.org/10.1016/B978-0-12-558701-3.50007-7

  24. Imbir, K.K.: Psychoevolutionary Theory of Emotion (Plutchik). In: Encyclopedia of Personality and Individual Differences, Springer International Publishing, Cham, pp. 1–9 (2017). https://doi.org/10.1007/978-3-319-28099-8_547-1

  25. Cowen, A.S., Keltner, D.: Self-report captures 27 distinct categories of emotion bridged by continuous gradients. Proc. Natl. Acad. Sci. U.S.A. 114(38), E7900–E7909 (2017). https://doi.org/10.1073/PNAS.1702247114/-/DCSUPPLEMENTAL

    Article  Google Scholar 

  26. Ohtsuka, S., Chiba, H., Sasaki, N., Harakawa, T.: Alternative vibration presentation methods for the two-point Body-Braille system. In: 2016 IEEE 5th Global Conference on Consumer Electronics, GCCE 2016 (2016). https://doi.org/10.1109/GCCE.2016.7800451

  27. Yao, L., Shi, Y., Chi, H., Ji, X., Ying, F.: Music-touch shoes: vibrotactile interface for hearing impaired dancers. p. 276 (2010). https://doi.org/10.1145/1709886.1709944

  28. Mazzoni, A., Bryan-Kinns, N.: How does it feel like? An exploratory study of a prototype system to convey emotion through haptic wearable devices | IEEE Conference Publication | IEEE Xplore. In: 7th International Conference on Intelligent Technologies for Interactive Entertainment (INTETAIN), pp. 64–68 (2015)

    Google Scholar 

  29. Petry, B., Illandara, T., Forero, J.P., Nanayakkara, S.: Ad-hoc access to musical sound for deaf individuals. In: ASSETS 2016 - Proceedings of the 18th International ACM SIGACCESS Conference on Computers and Accessibility, pp. 285–286 (2016). https://doi.org/10.1145/2982142.2982213

  30. Fourney, D.W., Fels, D.I.: Creating access to music through visualization. In: 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH), pp. 939–944 (2009). https://doi.org/10.1109/TIC-STH.2009.5444364

  31. Mori, J., Fels, D.I.: Seeing the music can animated lyrics provide access to the emotional content in music for people who are deaf or hard of hearing?, In: 2009 IEEE Toronto International Conference Science and Technology for Humanity (TIC-STH), pp. 951–956 (2009). https://doi.org/10.1109/TIC-STH.2009.5444362

  32. Zhou, X., et al.: Cortical speech processing in postlingually deaf adult cochlear implant users, as revealed by functional near-infrared spectroscopy. Trends in Hearing, vol. 22, p. 233121651878685 (2018). https://doi.org/10.1177/2331216518786850

  33. Karam, M., Russo, F.A., Fels, D.I.: Designing the model human cochlea: an ambient crossmodal audio-tactile display. IEEE Trans. Haptics 2(3), 160–169 (2009). https://doi.org/10.1109/TOH.2009.32

    Article  Google Scholar 

  34. Nanayakkara, S., Wyse, L., Taylor, E.A.: The haptic chair as a speech training aid for the deaf. In: Proceedings of the 24th Australian Computer-Human Interaction Conference on - OzCHI ’12, pp. 405–410 (2012). https://doi.org/10.1145/2414536.2414600

  35. Flores, G., Kurniawan, S., Manduchi, R., Martinson, E., Morales, L.M., Sisbot, E.A.: Vibrotactile guidance for wayfinding of blind walkers. IEEE Trans. Haptics 8(3), 306–317 (2015). https://doi.org/10.1109/TOH.2015.2409980

    Article  Google Scholar 

  36. Tranchant, P., Shiell, M.M., Giordano, M., Nadeau, A., Peretz, I., Zatorre, R.J.: Feeling the beat: Bouncing synchronization to vibrotactile music in hearing and early deaf people. Frontiers in Neuroscience 11, 507 (2017). https://doi.org/10.3389/FNINS.2017.00507/BIBTEX

  37. Trivedi, U., Alqasemi, R., Dubey, R.: Wearable musical haptic sleeves for people with hearing impairment. In: Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pp. 146–151 (2019). https://doi.org/10.1145/3316782.3316796

  38. Sridhar, P.K., Petry, B., Pakianathan, P.V.S., Kartolo, A.S., Nanayakkara, S.: Towards one-pixel-displays for sound information visualization. In: Proceedings of the 28th Australian Conference on Computer-Human Interaction - OzCHI ‘16, pp. 91–95 (2016). https://doi.org/10.1145/3010915.3010980

  39. Rocha, T., Paredes, H., Soares, D., Fonseca, B., Barroso, J.: MyCarMobile: a travel assistance emergency mobile app for deaf people. In: Bernhaupt, R., Dalvi, G., Joshi, A., Balkrishan, D.K., O’Neill, J., Winckler, M. (eds.) INTERACT 2017. LNCS, vol. 10513, pp. 56–65. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-67744-6_4

    Chapter  Google Scholar 

  40. Logic Pro - Apple. https://www.apple.com/logic-pro/ Accessed 8 Jan 2022

  41. Music production with Live and Push | Ableton. https://www.ableton.com/ Accessed 8 Jan 2022

Download references

Acknowledgments

This work is a result of the project INOV@UTAD, POCI-01–0247-FEDER-049337, financed by FEEI and supported by FEDER, through the Competitiveness and Inter-nationalization Operational Program. Furthermore, we thank all people who directly or indirectly helped in this study, particularly to José Vieira, João Faiões e Tiago Teixeira.

Author information

Authors and Affiliations

  1. UTAD - University of Trás-Os-Montes E Alto Douro, Quinta de Prados, 5001-801, Vila Real, Portugal

    Diana Carvalho, João Barroso & Tânia Rocha

  2. INESC TEC, Rua Dr. Roberto Frias, 4200-465, Porto, Portugal

    Diana Carvalho, João Barroso & Tânia Rocha

Authors
  1. Diana Carvalho
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. João Barroso
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tânia Rocha
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Diana Carvalho .

Editor information

Editors and Affiliations

  1. Purdue University West Lafayette, West Lafayette, IN, USA

    Vincent G. Duffy

  2. Tsinghua University, Beijing, China

    Qin Gao

  3. Chongqing University, Chongqing, China

    Jia Zhou

  4. Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Margherita Antona

  5. University of Crete and Foundation for Research and Technology – Hellas (FORTH), Heraklion, Crete, Greece

    Constantine Stephanidis

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Carvalho, D., Barroso, J., Rocha, T. (2022). Multisensory Experience for People with Hearing Loss: A Preliminary Study Using Haptic Interfaces to Sense Music. In: Duffy, V.G., Gao, Q., Zhou, J., Antona, M., Stephanidis, C. (eds) HCI International 2022 – Late Breaking Papers: HCI for Health, Well-being, Universal Access and Healthy Aging. HCII 2022. Lecture Notes in Computer Science, vol 13521. Springer, Cham. https://doi.org/10.1007/978-3-031-17902-0_21

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-17902-0_21

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-17901-3

  • Online ISBN: 978-3-031-17902-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics