HarmonyWave: Immersive Space Sound Therapy with Audio-Visual Synesthesia
Authors: 
Jian Yu, Ling Li, Wei ZengAuthors Info & Claims
Jian Yu, Ling Li, Wei ZengAuthors Info & ClaimsArticle No.: 34, Pages 1 - 8
Abstract
The paper presents a study on sound therapy installation HarmonyWave utilizing audio-visual synesthesia to alleviate anxiety. At the core of HarmonyWave lies the mapping of sounds to visuals with three distinct methods: laser vibration for line-based effects, water vibration for color-based effects, and water vibration for texture-based effects. On the basis, we design an interactive installation for an immersive space sound therapy experience within 3D space. The setup enables the audience to simultaneously hear the sound and observe superimposed images while actively engaging with the interactive elements. HarmonyWave was implemented with 20 natural sounds, specifically focusing on white noise known for inducing a calm and soothing emotional state in urban environments. We envision HarmonyWave as a tool to alleviate tension and anxiety in urban settings, serving a therapeutic function. We also delve into the possibilities and diversity offered by the three sensory mediums for mapping audio to visuals.
References
[1]
Jacob A Benfield, B Derrick Taff, Peter Newman, and Joshua Smyth. 2014. Natural sound facilitates mood recovery. Ecopsychology 6, 3 (2014), 183–188.
[2]
Ned Block. 2002. Some concepts of consciousness., 206–218 pages.
[3]
Ernst Florens Friedrich Chladni. 2015. Treatise on Acoustics: The First Comprehensive English Translation of EFF Chladni’s Traité d’Acoustique. Springer.
[4]
Steve Gibson. 2022. Moving Towards the Performed Image (Colour Organs, Synesthesia and Visual Music): Early Modernism (1900–1955). In Live Visuals. Routledge, 41–61.
[5]
Jonathan Goldman. 2022. Healing sounds: The power of harmonics. Simon and Schuster.
[6]
Penelope Gouk. 2017. Musical healing in cultural contexts. Routledge.
[7]
Brian Kane. 2014. Sound unseen: Acousmatic sound in theory and practice. Oxford University Press.
[8]
Peter Kivy. 1989. Sound sentiment: An essay on the musical emotions, including the complete text of the corded shell. Temple University Press.
[9]
Jennifer Leigh and Nicole Brown. 2021. Embodied inquiry: Research methods. Bloomsbury Publishing.
[10]
Zhongzhe Li, Meihui Ba, and Jian Kang. 2021. Physiological indicators and subjective restorativeness with audio-visual interactions in urban soundscapes. Sustainable Cities and Society 75 (2021), 103360.
[11]
Xue Mao, Jie Xu, Jiahong Lang, and Shangshu Zhang. 2023. Visualization of isomorphism-synesthesia of colour and music. Visual Informatics 7, 4 (2023), 110–114.
[12]
Qi Meng, Jiani Jiang, Fangfang Liu, and Xiaoduo Xu. 2020. Effects of the musical sound environment on communicating emotion. International Journal of Environmental Research and Public Health 17, 7 (2020), 2499.
[13]
You Jin Oh and Sojin Kim. 2012. Experimental study of cymatics. International Journal of Engineering and Technology 4, 4 (2012), 434.
[14]
Janani Ramaswamy and Sukhendu Das. 2020. See the sound, hear the pixels. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2970–2979.
[15]
BJWS Rayleigh. 1896. The Theory of Sound. Vol. 3. Macmillan google schola.
[16]
Adam Sheppard, Christina Stocking, Massimo Ralli, and Richard Salvi. 2020. A review of auditory gain, low-level noise and sound therapy for tinnitus and hyperacusis. International journal of audiology 59, 1 (2020), 5–15.
[17]
O Şimşek, SA Balkan, and A Koca. 2022. The concept of synesthesia (Multisensory Perception) in spatial experience and investigation of its change with technology. Journal of Architectural Sciences and Applications, 7 (Special Issue), 40 59 (2022).
[18]
Charles Spence and Nicola Di Stefano. 2023. Sensory translation between audition and vision. Psychonomic Bulletin & Review (2023), 1–28.
[19]
Michael L Stanchina, Muhanned Abu-Hijleh, Bilal K Chaudhry, Carol C Carlisle, and Richard P Millman. 2005. The influence of white noise on sleep in subjects exposed to ICU noise. Sleep medicine 6, 5 (2005), 423–428.
[20]
Paul Tourle. 2017. White noise: Sound, materiality and the crowd in contemporary heritage practice. International Journal of Heritage Studies 23, 3 (2017), 234–247.
[21]
Weixing Wang, Qianqian Li, Jingwen Xie, Ningfeng Hu, Ziao Wang, and Ning Zhang. 2023. Research on emotional semantic retrieval of attention mechanism oriented to audio-visual synesthesia. Neurocomputing 519 (2023), 194–204.
[22]
Yake Wei, Di Hu, Yapeng Tian, and Xuelong Li. 2022. Learning in audio-visual context: A review, analysis, and new perspective. arXiv preprint arXiv:https://arXiv.org/abs/2208.09579 (2022).
[23]
Bruce J West and Michael Shlesinger. 1990. The noise in natural phenomena. American Scientist 78, 1 (1990), 40–45.
[24]
Emily Witten, Jasmiina Ryynanen, Stella Wisdom, Cheryl Tipp, and Stella WY Chan. 2023. Effects of soothing images and soothing sounds on mood and well-being. British Journal of Clinical Psychology 62, 1 (2023), 158–179.
[25]
Boyan Zhang and Jian Kang. 2022. Effect of environmental contexts pertaining to different sound sources on the mood states. Building and Environment 207 (2022), 108456.
Index Terms
- HarmonyWave: Immersive Space Sound Therapy with Audio-Visual Synesthesia
Recommendations
Immersive auditory display system 'sound cask': three-dimensional sound field reproduction system based on the boundary surface control principle
VRST '18: Proceedings of the 24th ACM Symposium on Virtual Reality Software and TechnologySound cask was developed to realize the perfect 3D auditory display that creates 3D sound waves around the listener's head just the same as the primary sound field, based on the boundary surface control (BoSC) principle.
If we consider the sound ...
Inhibition of oscillation in a neuronal network model for tinnitus management by sound therapy
AIC'10/BEBI'10: Proceedings of the 10th WSEAS international conference on applied informatics and communications, and 3rd WSEAS international conference on Biomedical electronics and biomedical informaticsTinnitus is a state in which one hears sounds in the ear or head without any external sound. There are many therapeutic approaches for tinnitus and sound therapy is one of the techniques for its treatment that have been proposed. In order to investigate ...
The immersive computer-controlled audio sound theater: history and current trends in multi-modal sound diffusion
SIGGRAPH '09: SIGGRAPH 2009: TalksMulti-channel sound diffusion has been an essential part of the electronic and computer music performance from the beginnings of the genre in the 1950's. We see early experimentation in sound spatialization in Varèse's use of loudspeaker "paths" in ...
Comments
Information & Contributors
Information
Published In
Copyright © 2024 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 December 2024
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Guangzhou Basic and Applied Basic Research Foundation
Conference
VINCI 2024
VINCI '24: The 17th International Symposium on Visual Information Communication and Interaction
December 11 - 13, 2024
Hsinchu, Taiwan
Acceptance Rates
Overall Acceptance Rate 71 of 193 submissions, 37%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 164Total Downloads
- Downloads (Last 12 months)164
- Downloads (Last 6 weeks)58
Reflects downloads up to 05 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format