Abstract
We present an application of sonification in an artistic context, namely to augment the visual impressions of train journeys. While sonification and auditory displays are typically used as means to present data and to inform the user, our project sound/tracks aims at enhancing the visual experience of looking out of a moving train’s window at the passing landscape by adding a sound dimension. This allows for reflecting upon the visual impressions and deepening the state of contemplation. To this end, sound/tracks translates the perceived movement of the scenery and other visual impressions, such as passing trains, into music. The continuously changing view outside the window is captured with a camera and transformed into instantaneously played music. The application can be run on mobile phones with a built-in camera and on laptops with a Web-cam. The paper proposes and discusses different sonification approaches and presents different application scenarios.
Similar content being viewed by others
References
Sound/tracks sonification methods. http://www.cp.jku.at/projects/soundtracks/methods.html
Bresin R (2005) What is the color of that music performance. In: Proceedings of the international computer music conference, Barcelona, Spain
Collier WG, Hubbard TL (2004) Musical scales and brightness evaluations: Effects of pitch, direction, and scale mode. Music Sci 8:151–173
Datteri DL, Howard JN (2004) The sound of color. In: Proceedings of the 8th international conference on music perception & cognition, Evanston, IL, USA
Rusconi E, Kwan B, Giordano BL, Umiltà C, Butterworth B (2006) Spatial representation of pitch height: the SMARC effect. Cognition 99(2):113–129
Gondry M (2002) Music video for “The Chemical Brothers—Star Guitar”
Gröhn L (2007) Sound of paintings. http://www.synestesia.fi
Jensenius AR (2007) Action—sound: developing methods and tools to study music-related body movement. PhD thesis, University of Oslo, Norway
Jo K, Nagano N (2008) Monalisa: See the sound, hear the image. In: Proceedings of the 8th international conference on new interfaces for musical expression
Ängeslevä J, Cooper R (2005) Last clock. IEEE Comput Graph Appl 25(1):20–23
Knees P, Pohle T, Widmer G Sound/tracks Web site. http://www.cp.jku.at/projects/soundtracks
Krumhansl CL, Kessler EJ (1982) Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys. Psychol Rev 89:334–368
Marks LE (1974) On associations of light and sound: The mediation of brightness, pitch, and loudness. Am J Psychol 87(1–2):173–188
Meijer PB The vOICe. http://www.seeingwithsound.com
Meijer PB (1992) An experimental system for auditory image representations. IEEE Trans Biomed Eng 39(2):112–121
Nunes M, Greenberg S, Carpendale S, Gutwin C (2007) What did I miss? Visualizing the past through video traces. In: Proceedings of the 10th European conference on computer supported cooperative work, Limerick, Ireland
Pohle T, Knees P, Widmer G (2008) Sound/tracks: real-time synaesthetic sonification and visualisation of passing landscapes. In: Proceedings of the 16th ACM conference on multimedia, Vancouver, BC, Canada
Reinhart M, Widrich V tx-transform. http://www.tx-transform.com
Tanaka A (2004) Bondage. http://www.xmira.com/atau/bondage/
Wikipedia (2008) Alexander Scriabin—Wikipedia, the free encyclopedia. http://en.wikipedia.org/w/index.php?title=Alexander_Scriabin&oldid=189505713
Acknowledgements
This research is supported by the Austrian Science Fund (FWF) under project numbers L511-N15, TRP 109-N23, and Z159 (Wittgenstein Award).
Author information
Authors and Affiliations
Corresponding author
Additional information
This article is a revised and updated version of the paper “sound/tracks: Real-Time Synaesthetic Sonification and Visualisation of Passing Landscapes” by the same authors originally published in the proceedings of the ACM Multimedia 2008 conference, cf. [17].
Rights and permissions
About this article
Cite this article
Knees, P., Pohle, T. & Widmer, G. Sound/tracks: artistic real-time sonification of train journeys. J Multimodal User Interfaces 6, 87–93 (2012). https://doi.org/10.1007/s12193-011-0089-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12193-011-0089-x