skip to main content
10.1145/1459359.1459440acmconferencesArticle/Chapter ViewAbstractPublication PagesmmConference Proceedingsconference-collections
research-article

Sound/tracks: real-time synaesthetic sonification and visualisation of passing landscapes

Published: 26 October 2008 Publication History

Abstract

When travelling on a train, many people enjoy looking out of the window at the landscape passing by. We present sound/tracks, an application that translates the perceived movement of the scenery and other visual impressions, such as passing trains, into music. The continuously changing view outside the window is captured with a camera and translated into MIDI events that are replayed instantaneously. This allows for a reflection of the visual impression, adding a sound dimension to the visual experience and deepening the state of contemplation. The application is intended to be run on both mobile phones (with built-in camera) and on laptops (with a connected Web-cam). We propose and discuss different approaches to translating the video signal into an audio stream, present different application scenarios, and introduce a method to visualise the dynamics of complete train journeys by "re-transcribing" the captured video frames used to generate the music.

References

[1]
W. G. Collier and T. L. Hubbard. Musical scales and brightness evaluations: Effects of pitch, direction, and scale mode. Musicae Scientiae, 8:151--173 2004.
[2]
Darcee L. Datteri and Jeffrey N. Howard. The sound of color. In Proceedings of the 8th International Conference on Music Perception & Cognition (ICMPC8), Evanston, IL, USA, August 3-7 2004.
[3]
Michel Gondry. Music video for "The Chemical Brothers - Star Guitar", 2002.
[4]
Lauri Gröhn. Sound of Paintings. URL: http://www.synestesia.fi/
[5]
Kazuhiro Jo and Norihisa Nagano. Monalisa: "See the sound, hear the image". In Proceedings of the 8th International Conference New Interface for Musical Expression. NIME '08, pages 315--318, Genova, Italy, June 5-7 2008.
[6]
C. L. Krumhansl and E. J. Kessler. Tracing the dynamic changes in perceived tonal organization in a spatial representation of musical keys. Psychological Review, 89:334--368, 1982.
[7]
L. E. Marks. On associations of light and sound: The mediation of brightness, pitch, and loudness. American Journal of Psychology, 87 (1-2):173--188, 1974.
[8]
Daphne Maurer, Thanujeni Pathman, and Catherine J. Mondloch. The shape of boubas: sound-shape correspondences in toddlers and adults. Developmental Science, 9:3:316--322, 2006.
[9]
H. Purwins. Profiles of Pitch Classes Circularity of Relative Pitch and Key - Experiments, Models, Computational Music Analysis, and Perspectives. PhD thesis, TU Berlin, 2005.
[10]
Martin Reinhart and Virgil Widrich. tx-transform. URL: http://www.tx-transform.com/
[11]
Elena Rusconi, Bonnie Kwan, Bruno L. Giordano, Carlo Umiltà, and Brian Butterworth. Spatial representation of pitch height: the SMARC effect. Cognition, 99:2:113--129, 2006.
[12]
Atau Tanaka. Bondage. URL: http://www.xmira.com/atau/bondage/, 2004.
[13]
Wikipedia. Alexander Scriabin. URL: http://en.wikipedia.org/wiki/Alexander_Scriabin

Cited By

View all
  • (2017)Large-Scale Interaction with a Sound Installation as a Design ToolProceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences10.1145/3123514.3123564(1-6)Online publication date: 23-Aug-2017
  • (2016)Searching for Audio by Sketching Mental Images of SoundProceedings of the 2016 ACM on International Conference on Multimedia Retrieval10.1145/2911996.2912021(95-102)Online publication date: 6-Jun-2016
  • (2014)A Closing Circle: Back to AuthoringIEEE MultiMedia10.1109/MMUL.2014.3721:3(4-6)Online publication date: Jul-2014
  • Show More Cited By

Index Terms

  1. Sound/tracks: real-time synaesthetic sonification and visualisation of passing landscapes

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Conferences
    MM '08: Proceedings of the 16th ACM international conference on Multimedia
    October 2008
    1206 pages
    ISBN:9781605583037
    DOI:10.1145/1459359
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 26 October 2008

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. mobile music generation
    2. sonification
    3. train journey

    Qualifiers

    • Research-article

    Conference

    MM08
    Sponsor:
    MM08: ACM Multimedia Conference 2008
    October 26 - 31, 2008
    British Columbia, Vancouver, Canada

    Acceptance Rates

    Overall Acceptance Rate 2,145 of 8,556 submissions, 25%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)5
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 17 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2017)Large-Scale Interaction with a Sound Installation as a Design ToolProceedings of the 12th International Audio Mostly Conference on Augmented and Participatory Sound and Music Experiences10.1145/3123514.3123564(1-6)Online publication date: 23-Aug-2017
    • (2016)Searching for Audio by Sketching Mental Images of SoundProceedings of the 2016 ACM on International Conference on Multimedia Retrieval10.1145/2911996.2912021(95-102)Online publication date: 6-Jun-2016
    • (2014)A Closing Circle: Back to AuthoringIEEE MultiMedia10.1109/MMUL.2014.3721:3(4-6)Online publication date: Jul-2014
    • (2012)A Synaesthetic Approach for Image Slideshow GenerationProceedings of the 2012 IEEE International Conference on Multimedia and Expo10.1109/ICME.2012.75(985-990)Online publication date: 9-Jul-2012

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media