Jörg Müller
Matthias Geier
Christina Dicke
ABSTRACT
In this paper we present a system that allows to "touch", grab and manipulate sounds in mid-air. Further, arbitrary objects can seem to emit sound. We use spatial sound reproduction for sound rendering and computer vision for tracking. Using our approach, sounds can be heard from anywhere in the room and always appear to originate from the same (possibly moving) position, regardless of the listener's position. We demonstrate that direct "touch" interaction with sound is an interesting alternative to indirect interaction mediated through controllers or visual interfaces. We show that sound localization is surprisingly accurate (11.5 cm), even in the presence of distractors. We propose to leverage the ventriloquist effect to further increase localization accuracy. Finally, we demonstrate how affordances of real objects can create synergies of auditory and visual feedback. As an application of the system, we built a spatial music mixing room.
- Berkhout, A. A holographic approach to acoustic control. Journal of the Audio Engineering Society 36, 12 (1988), 977--995.Google Scholar
- Blauert, J. Spatial Hearing: The Psychophysics of Human Sound Localization, revised ed. MIT Press, 1996.Google Scholar
- Brewster, S., Lumsden, J., Bell, M., Hall, M., and Tasker, S. Multimodal 'eyes-free' interaction techniques for wearable devices. In SIGCHI Conference on Human Factors in Computing Systems (2003). Google ScholarDigital Library
- Brungart, D. S., and Rabinowitz, W. M. Auditory localization of nearby sources. Head-related transfer functions. Journal of the Acoustical Socienty of America 106, 3 (1999), 1465--1479.Google Scholar
- Daniel, J. Spatial sound encoding including near field effect: Introducing distance coding filters and a viable, new Ambisonic format. In 23rd International Conference of the Audio Engineering Society (2003).Google Scholar
- de Vries, D. Wave Field Synthesis. AES Monograph. Audio Engineering Society, 2009.Google Scholar
- Fohl, W., and Nogalski, M. A gesture control interface for a Wave Field Synthesis system. In International Conference on New Interfaces for Musical Expression (2013).Google Scholar
- Geier, M., and Spors, S. Spatial audio reproduction with the SoundScape Renderer. In 27th Tonmeistertagung - VDT International Convention (2012).Google Scholar
- Hutchins, E. L., Hollan, J. D., and Norman, D. A. Direct manipulation interfaces. Human-Computer Interaction 1, 4 (1985), 311--338. Google ScholarDigital Library
- Ishii, H., Mazalek, A., and Lee, J. Bottles as a minimal interface to access digital information. In SIGCHI Conference on Human Factors in Computing Systems (2001). Google ScholarDigital Library
- Ishii, H., and Ullmer, B. Tangible bits: towards seamless interfaces between people, bits and atoms. In SIGCHI Conference on Human Factors in Computing Systems (1997). Google ScholarDigital Library
- Jackson, C. V. Visual factors in auditory localization. Quarterly Journal of Experimental Psychology 5, 2 (1953), 52--65.Google ScholarCross Ref
- Leslie, G., Zamborlin, B., Jodlowski, P., and Schnell, N. Grainstick: A collaborative, interactive sound installation. In International Computer Music Conference (2010).Google Scholar
- Melchior, F., Laubach, T., and de Vries, D. Authoring and user interaction for the production of Wave Field Synthesis content in an augmented reality system. In Fourth IEEE and ACM International Symposium on Mixed and Augmented Reality (2005). Google ScholarDigital Library
- Melchior, F., Sladeczek, C., de Vries, D., and Fröhlich, B. User-dependent optimization of Wave Field Synthesis reproduction for directive sound fields. In 124th Convention of the Audio Engineering Society (2008).Google Scholar
- Mynatt, E. D., Back, M., Want, R., Baer, M., and Ellis, J. B. Designing Audio Aura. In SIGCHI Conference on Human Factors in Computing Systems (1998). Google ScholarDigital Library
- Pick, H. L., Warren, D. H., and Hay, J. C. Sensory conflict in judgments of spatial direction. Perception & Psychophysics 6, 4 (1969), 203--205.Google ScholarCross Ref
- Shneiderman, B. The future of interactive systems and the emergence of direct manipulation. Behaviour & Information Technology 1, 3 (1982), 237--256.Google ScholarCross Ref
- Spors, S. Extension of an analytic secondary source selection criterion for Wave Field Synthesis. In 123rd Convention of the Audio Engineering Society (2007).Google Scholar
- Spors, S., Wierstorf, H., Raake, A., Melchior, F., Frank, M., and Zotter, F. Spatial sound with loudspeakers and its perception: A review of the current state. IEEE Proceedings 101, 9 (2013), 1920--1938.Google ScholarCross Ref
- Springer, J. P., Sladeczek, C., Scheffer, M., Hochstrate, J., Melchior, F., and Fröhlich, B. Combining Wave Field Synthesis and multi-viewer stereo displays. In IEEE Virtual Reality Conference (2006). Google ScholarDigital Library
- Völk, F., Mühlbauer, U., and Fastl, H. Minimum audible distance (MAD) by the example of Wave Field Synthesis. In German Annual Conference on Acoustics (DAGA) (2012).Google Scholar
- Wierstorf, H., Raake, A., Geier, M., and Spors, S. Perception of focused sources in Wave Field Synthesis. Journal of the Audio Engineering Society 61, 1/2 (2013), 5--16.Google Scholar
- Yost, W. A., Dye, R. H., and Sheft, S. A simulated cocktail party with up to three sound sources. Perception & Psychophysics 58 (1996), 1026--1036.Google ScholarCross Ref
- Zotter, F., and Spors, S. Is sound field control determined at all frequencies? How is it related to numerical acoustics? In 52nd Conference of the Audio Engineering Society (2013).Google Scholar
Index Terms
- The boomRoom: mid-air direct interaction with virtual sound sources
Recommendations
The interactive soundscape renderer for loudspeaker- and headphone-based spatial sound presentation
VRST '17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and TechnologyWe demonstrate the SoundScape Renderer (SSR), which is an open-source software package that performs realtime spatial sound reproduction based on various different loudspeaker-based methods as well as headphones. The implemented algorithms include wave ...
3D Tabletop AR: A Comparison of Mid-Air, Touch and Touch+Mid-Air Interaction
AVI '20: Proceedings of the International Conference on Advanced Visual InterfacesThis paper contributes a first comparative study of three techniques for selecting 3D objects anchored to the table in tabletop Augmented Reality (AR). The impetus for this study is that touch interaction makes more sense when the targeted objects are ...
Should I Stay or Should I Go? Selecting Between Touch and Mid-Air Gestures for Large-Display Interaction
Human-Computer Interaction – INTERACT 2015AbstractUsers can interact with large displays in many ways, including touch and mid-air gestures. However, it remains unclear how these ways compare and when users choose one over the other. In a first experiment, we compare touch and mid-air gestures to ...
Comments