Balandino Di Donato
ABSTRACT
In this paper, we explore human-centered interaction design aspects that determine the realisation and appreciation of musical works (installations, composition and performance), interfaces for sound design and musical expression, augmented instruments, sonic aspects of virtual environments and interactive audiovisual performances. In this first work, with the human at the centre of the design, we started sketching modes of interaction with sound that could result direct, engaging, natural and embodied in a collaborative, interactive, inclusive and diverse music environment. We define this as Human-Sound Interaction (HSI). To facilitate the exploration of HSIs, we prototyped SoundSculpt, a cross-modal audio, holographic projection and mid-air haptic feedback system. During an informal half-day workshop, we observed that HSIs through SoundSculpt have the potential to foster new ways of interaction with sound and to make them accessible to diverse musicians, sound artists and audience.
- Laura Bianchini, Michelangelo Lupone, Silvia Lanzalone, and Alessio Gabriele. 2019. Augmented Instruments at CRM - Centro Ricerche Musicali of Rome: Feed-Drum, SkinAct, WindBack and ResoFlute. In Proceedings of the International Computer Music Conference (ICMC '19). New York, NY, USA.Google Scholar
- Jamie Bullock and Balandino Di Donato. 2016. Approaches to Visualising the Spatial Position of "Sound-Objects". In Proceedings of the Conference on Electronic Visualisation and the Arts (EVA '16). BCS Learning Development Ltd., London, United Kingdom, 15--22. https://doi.org/10.14236/ewic/EVA2016.4Google ScholarDigital Library
- Baptiste Caramiaux, Patrick Susini, Tommaso Bianco, et al. 2011. Gestural Embodiment of Environmental Sounds: an Experimental Study. In Proc. of the International Conference on New Interaces for Musical Expression (NIME '11). Oslo, Norway, 144--148.Google Scholar
- Tom Carter, Sue Ann Seah, Benjamin Long, Bruce Drinkwater, and Sriram Subramanian. 2013. UltraHaptics: multi-point mid-air haptic feedback for touch surfaces. In Proc. of the 26th annual ACM symposium on User interface software and technology. ACM, 505--514.Google ScholarDigital Library
- Perry R. Cook. 2001. Principles for Designing Computer Music Controllers. In Proceedings of the International Conference on New Interfaces for Musical Expression (NIME'01). Seattle, WA, 3--6.Google Scholar
- Alice-Ann Darrow and Julie Novak. 2007. The Effect of Vision and Hearing Loss on Listeners' Perception of Referential Meaning in Music. Journal of Music Therapy 44, 1 (03 2007), 57--73. https://doi.org/10.1093/jmt/44.1.57Google ScholarCross Ref
- Christopher Dewey and Jonathan P. Wakefield. 2019. Exploring the Container Metaphor for Equalisation Manipulation. In Proceedings of the International Conference on New Interfaces for Musical Expression. UFRGS, Porto Alegre, Brazil, 128--129.Google Scholar
- Paul Dourish. 2001. Where The Action Is: The Foundations of Embodied Interaction. MIT Press.Google Scholar
- John Levack Drever. 2019. 'Primacy of the Ear' - But Whose Ear?: The case for auraldiversity in sonic arts practice and discourse. Organised Sound 24, 1 (2019), 85--95. https://doi.org/10.1017/S1355771819000086Google ScholarCross Ref
- Karmen Franinović and Stefania Serafin. 2013. Sonic interaction design. Mit Press.Google Scholar
- Alessio Gabriele, Michelangelo Lupone, Paola Inverardi, and Patrizio Pelliccione. 2010. Ad-opera: music-inspired self-adaptive systems. In Proceedings of the FSE/SDP workshop on Future of software engineering research. 119--124.Google ScholarDigital Library
- William W. Gaver. 1991. Technology Affordances. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI '91). ACM, New York, NY, USA, 79--84. https://doi.org/10.1145/108844.108856Google Scholar
- Steven Gelineck, Dannie Korsgaard, and Morten Büchert. 2015. Stage- vs. Channel-strip Metaphor - Comparing Performance when Adjusting Volume and Panning of a Single Channel in a Stereo Mix. In Proceedings of the International Conference on New Interfaces for Musical Expression. Baton Rouge, Louisiana, USA, 343--346.Google ScholarDigital Library
- Steven Gelineck and Anders Kirk Uhrenholt. 2016. Exploring visualisation of channel activity, levels and EQ for user interfaces implementing the stage metaphor for music mixing. In 2nd AES Workshop on Intelligent Music Production.Google Scholar
- Rolf Inge Godøy. 2010. Gestural affordances of musical sound. In Musical Gestures: Sound, Movement, and Meaning, Rolf Inge Godøy and Marc Leman (Eds.). Routledge, New York, New York, 115--137.Google Scholar
- Rolf Inge Godøy, Egil Haga, and Alexander Refsum Jensenius. 2006. Exploring Music-Related Gestures by Sound-Tracing: A Preliminary Study. In Proc. of the 2nd International Symposium on Gesture Interfaces for Multimedia Systems (GIMS2006). Leeds, United Kingdom.Google Scholar
- Sukeshini A Grandhi, Gina Joue, and Irene Mittelberg. 2011. Understanding naturalness and intuitiveness in gesture production. In Proc. of the SIGCHI Conference on Human Factors in Computing Systems (CHI). Vancouver, BC, Canada, 821--824.Google ScholarDigital Library
- Rex Hartson. 2003. Cognitive, physical, sensory, and functional affordances in interaction design. Behaviour & Information Technology 22, 5 (2003), 315--338. https://doi.org/10.1080/01449290310001592587Google ScholarCross Ref
- Andrew Hugill. 2019. Aural Diversity. In Given at the Equality, Diversity and Inclusion in Music Higher Education conference. City University, London, United Kingdom.Google Scholar
- Takayuki Iwamoto, Mari Tatezono, and Hiroyuki Shinoda. 2008. Non-contact method for producing tactile sensation using airborne ultrasound. In International Conference on Human Haptic Sensing and Touch Enabled Computer Applications. Springer, 504--513.Google ScholarDigital Library
- Michelangelo Lupone, Silvia Lanzalone, Alessio Gabriele, and Martina De Luca. 2015. Forme Immateriali by Michelangelo Lupone: structure, creation, interaction, evolution of a permanent adaptive music work. In Proceedings of the Electroacoustic Winds 2015 Conference. Aveiro, Portugal.Google Scholar
- Donald A Norman and Stephen W Draper. 1986. User Centered System Design: New Perspectives on Human-computer Interaction. Lawrence Erlbaum Associates, Hillsdale, New Jersey.Google ScholarDigital Library
- Ian O'connell and James Rock. 2011. Projection apparatus and method for pepper's ghost illusion. (Feb. 8 2011). US Patent 7,883,212.Google Scholar
- Lidia Oshlyansky, Harold Thimbleby, and Paul Cairns. 2004. Breaking Affordance: Culture as Context. In Proceedings of the Third Nordic Conference on Human-Computer Interaction (NordiCHI '04). Association for Computing Machinery, New York, NY, USA, 81--84. https://doi.org/10.1145/1028014.1028025Google ScholarDigital Library
- Stefano Papetti and Charalampos Saitis. 2018. Musical Haptics. Springer, Cham.Google Scholar
- Linda Pring, Katherine Woolf, and Valerie Tadic. 2008. Melody and Pitch Processing in Five Musical Savants with Congenital Blindness. Perception 37, 2 (2008), 290--307. https://doi.org/10.1068/p5718Google ScholarCross Ref
- Davide Rocchesso, Stefania Serafin, Frauke Behrendt, Nicola Bernardini, Roberto Bresin, Gerhard Eckel, Karmen Franinovic, Thomas Hermann, Sandra Pauletto, Patrick Susini, and Yon Visell. 2008. Sonic Interaction Design: Sound, Information and Experience. In CHI '08 Extended Abstracts on Human Factors in Computing Systems (CHI EA '08). Association for Computing Machinery, New York, NY, USA, 3969--3972. https://doi.org/10.1145/1358628.1358969Google Scholar
- Charalampos Saitis, Hanna Järveläinen, and Claudia Fritz. 2018. The Role of Haptic Cues in Musical Instrument Quality Perception. Springer International Publishing, Cham, 73--93. https://doi.org/10.1007/978-3-319-58316-7_5Google Scholar
- Suranga Chandima Sanayakkara, Lonce Wyse, S. H. Ong, and Elizabeth A. Taylor. 2013. Enhancing Musical Experience for the Hearing-Impaired Using Visual and Haptic Displays. Human-Computer Interaction 28, 2 (2013), 115--160. https://doi.org/10.1080/07370024.2012.697006Google Scholar
- Michael Schutz and Scott Lipscomb. 2007. Hearing Gestures, Seeing Music: Vision Influences Perceived Tone Duration. Perception 36, 6 (2007), 888--897. https://doi.org/10.1068/p5635Google Scholar
- B.R. Shelton and C.L Searle. 1980. The influence of vision on the absolute identification of sound-source position. Perception & Psychophysics 28, 6 (1980), 589--596. https://doi.org/10.3758/BF03198830Google ScholarCross Ref
- Dag Svanæs. 2000. Understanding interactivity: steps to a phenomenology of human-computer interaction. Norges teknisk-naturvitenskapelige universitet.Google Scholar
- Atau Tanaka, Alessandro Altavilla, and Neil Spowage. 2012. Gestural Musical Affordances. In Proceedings of the 9th International Conference on Sound and Music Computing. Copenhangen, Denmark.Google Scholar
- Atau Tanaka and Adam Parkinson. 2016. Haptic Wave: A Cross-Modal Interface for Visually Impaired Audio Producers. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). Association for Computing Machinery, New York, NY, USA, 2150--2161. https://doi.org/10.1145/2858036.2858304Google ScholarDigital Library
- Oliver Vitouch, Sandra Sovdat, and Norman Holler. 2006. Audio-vision: Visual input drives perceived music tempo. In Proceedings of the 9th International Conference on Music Perception & Cognition. Bologna, Italy, 86--89.Google Scholar
- Mads Walther-Hansen. 2017. New and Old User Interface Metaphors in Music Production. Journal on the Art of Record Production (JARP) 11 (2017).Google Scholar
Index Terms
- Human-Sound Interaction: Towards a Human-Centred Sonic Interaction Design approach
Recommendations
Form Follows Sound: Designing Interactions from Sonic Memories
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing SystemsSonic interaction is the continuous relationship between user actions and sound, mediated by some technology. Because interaction with sound may be task oriented or experience-based it is important to understand the nature of action-sound relationships ...
Robot-human interaction with an anthropomorphic percussionist
CHI '06: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThe paper presents our approach for human-machine interaction with an anthropomorphic mechanical percussionist that can listen to live players, analyze perceptual musical aspects in real-time, and use the product of this analysis to play along in a ...
Comments