ABSTRACT
Scientists increasingly rely on computational models of atoms and molecules to observe, understand and make predictions about the microscopic world. Atoms and molecules are in constant motion, with vibrations and structural fluctuations occurring at very short time-scales and corresponding length-scales. But can these microscopic oscillations be converted into sound? And, what would they sound like? In this paper we present our initial steps towards a generalised approach for sonifying data produced by a real-time molecular dynamics simulation. The approach uses scanned synthesis to translate real-time geometric simulation data into audio. The process is embedded within a stand alone application as well as a variety of audio plugin formats to enable the process to be used as an audio synthesis method for music making. We review the relevant background literature before providing an overview of our system. Simulations of three molecules are then considered: 17-alanine, graphene and a carbon nanotube. Four examples are then provided demonstrating how the technique maps molecular features and parameters onto the auditory character of the resulting sound. A case study is then provided in which the sonification/synthesis method is used within a musical composition.
Supplemental Material
Available for Download
Supplemental material.
- Andrew Allen and Nikunj Raghuvanshi. 2015. Aerophones in Flatland: Interactive Wave Simulation of Wind Instruments. ACM Trans. Graph. 34, 4, Article Article 134 (July 2015), 11 pages. Google ScholarDigital Library
- Norman L Allinger, Young H Yuh, and Jenn Huei Lii. 1989. Molecular mechanics. The MM3 force fileld for hydrocarbons. 1. Journal of the American Chemical Society 111, 23 (1989), 8551--8566.Google ScholarCross Ref
- Robert E. Arbon, Alex J. Jones, Thomas Mitchell, Lars A. Bratholm, and David R Glowacki. 2018. Sonifying Stochastic Walks on Biomolecular Energy Landscapes. In Proceedings of the 24th International Conference on Auditory Display. Houghton.Google ScholarCross Ref
- Holger Ballweg, Agnieszka K Bronowska, and Paul Vickers. 2016. Interactive Sonification for Structural Biology and Structure-Based Drug Design. In Proceedings of ISon 2016, 5th Interactive Sonification Workshop. Bielefeld, 41--47. http://interactive-sonification.org/ISon2016/papers/ISon2016-06-BallwegBronowskaVickers.pdfGoogle Scholar
- Aldo Borgonovo and Goffredo Haus. 1986. Sound Synthesis by Means of Two-Variable Functions: Experimental Criteria and Results. Computer Music Journal 10, 3 (1986), 57--71. http://www.jstor.org/stable/3680260Google ScholarCross Ref
- Richard Boulanger, Paris Smaragdis, and John Fitch. 2000. Scanned Synthesis: An Introduction and Demonstration of a New Synthesis and Signal Processing Technique. In International Computer Music Conference.Google Scholar
- Cockos. 2020. Reaper Digital Audio Workstation. https://www.reaper.fm (accessed: 03.07.2020).Google Scholar
- D. Creasey. 2016. Audio Processes: Musical Analysis, Modification, Synthesis, and Control. Routledge.Google ScholarCross Ref
- Florian Dombois and Gerhard Eckel. 2011. Audification. In The sonification handbook.Google Scholar
- Peter Eastman, Jason Swails, John D. Chodera, Robert T. McGibbon, Yutong Zhao, Kyle A. Beauchamp, Lee-Ping Wang, Andrew C. Simmonett, Matthew P. Harrigan, Chaya D. Stern, Rafal P. Wiewiora, Bernard R. Brooks, and Vijay S. Pande. 2017. OpenMM 7: Rapid development of high performance algorithms for molecular dynamics. PLOS Computational Biology 13, 7 (07 2017), 1--17. Google ScholarCross Ref
- Sebastian L Franjou, Mario Milazzo, Chi-Hua Yu, and Markus J Buehler. 2019. Sounds interesting: can sonification help us design new proteins? Expert Review of Proteomics 16, 11--12 (dec 2019), 875--879. Google ScholarCross Ref
- Daan Frenkel and Berend Smit. 2001. Understanding molecular simulation: from algorithms to applications. Vol. 1. Elsevier.Google ScholarDigital Library
- Miguel Angel Garcia-Ruiz and Jorge Rafael Gutierrez-Pulido. 2006. An overview of auditory display to assist comprehension of molecular information. Interacting with Computers 18, 4 (2006), 853 -- 868. Special Theme Papers from Special Editorial Board Members (contains Regular Papers). Google ScholarDigital Library
- Andre K Geim and Konstantin S Novoselov. 2010. The rise of graphene. In Nanoscience and technology: a collection of reviews from nature journals. World Scientific, 11--19.Google Scholar
- Google. 2020. GRPC. https://grpc.io (accessed: 03.07.2020).Google Scholar
- Florian Grond and Jonathan Berger. 2011. Parameter mapping sonification. In The sonification handbook.Google Scholar
- Florian Grond and Fabio Dall'Antonia. 2008. Sumo. A Sonification Utility for Molecules. In Proceedings of ICAD 2008. International Community for Auditory Display, 1--7. https://smartech.gatech.edu/handle/1853/49952Google Scholar
- Florian Grond and Thomas Hermann. 2014. Interactive Sonification for Data Exploration: How listening modes and display purposes define design guidelines. Organised Sound 19, 1 (2014), 41--51.Google ScholarCross Ref
- Florian Grond, Stefan Janssen, Stefanie Schirmer, and Thomas Hermann. 2010. Browsing RNA Structures by Interactive Sonification. In Proceedings of ISon 2010, 3rd Interactive Sonification Workshop. Stockholm, 11--16. http://www.interactive-sonification.org/ISon2010/proceedings/papers/GrondJanssenSchirmerHermann{_}ISon2010.pdfGoogle Scholar
- Roald Hoffmann. 1990. Molecular Beauty. The Journal of Aesthetics and Art Criticism 48, 3 (jan 1990), 191--204. Google ScholarCross Ref
- Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. 2016. Jupyter Notebooks-a publishing format for reproducible computational workflows.. In ELPUB. 87--90.Google Scholar
- Gregory Kramer. 1994. Some organizing principles for representing data with sound. Auditory Display-Sonification, Audification, and Auditory Interfaces (1994), 185--221.Google Scholar
- Intangible Realities Laboratory. 2020. Narupa iMD Code Repository. https://gitlab.com/intangiblerealities/narupa-applications/narupa-imd (accessed: 03.07.2020).Google Scholar
- Intangible Realities Laboratory. 2020. Narupa Protocol Code Repository. https://gitlab.com/intangiblerealities/narupa-protocol (accessed: 03.07.2020).Google Scholar
- Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Christensen, Marcin Dułak, Jesper Friis, Michael N Groves, Bjørk Hammer, Cory Hargus, Eric D Hermes, Paul C Jennings, Peter Bjerre Jensen, James Kermode, John R Kitchin, Esben Leonhard Kolsbjerg, Joseph Kubal, Kristen Kaasbjerg, Steen Lysgaard, Jón Bergmann Maronsson, Tristan Maxson, Thomas Olsen, Lars Pastewka, Andrew Peterson, Carsten Rostgaard, Jakob Schiøtz, Ole Schütt, Mikkel Strange, Kristian S Thygesen, Tejs Vegge, Lasse Vilhelmsen, Michael Walter, Zhenhua Zeng, and Karsten W Jacobsen. 2017. The atomic simulation environment---a Python library for working with atoms. Journal of Physics: Condensed Matter 29, 27 (2017), 273002. http://stacks.iop.org/0953-8984/29/i=27/a=273002Google ScholarCross Ref
- Ryan McGee. 2013. VOSIS: a Multi-touch Image Sonification Interface.. In NIME. 460--463.Google Scholar
- Ryan McGee, Daniel Ashbrook, and Sean White. 2012. SenSynth: a Mobile Application for Dynamic Sensor to Sound Mapping.. In NIME 2012.Google Scholar
- Thomas Mitchell, Joseph Hyde, Philip Tew, and David R Glowacki. 2014. dance-room Spectroscopy: At the Frontiers of Physics, Performance, Interactive Art and Technology. Leonardo 49, 2 (aug 2014), 138--147. Google ScholarCross Ref
- Thomas J. Mitchell and Alex J. Jones. 2020. Graphene Synth Repository. https://bitbucket.org/teamaxe/graphenesynth (accessed: 03.07.2020).Google Scholar
- Donald A Norman. 2004. Emotional design: Why we love (or hate) everyday things. Basic Civitas Books.Google Scholar
- Michael B. O'Connor, Simon J. Bennie, Helen M. Deeks, Alexander Jamieson-Binnie, Alex J. Jones, Robin J. Shannon, Rebecca Walters, Thomas J. Mitchell, Adrian J. Mulholland, and David R. Glowacki. 2019. Interactive molecular dynamics in virtual reality from quantum chemistry to drug binding: An open-source multi-person framework. arXiv:1902.01827 Google ScholarCross Ref
- Joshua Peschke and Axel Berndt. 2017. The geometric oscillator: Sound synthesis with cyclic shapes. In ACM International Conference Proceeding Series, Vol. Part F1319. Association for Computing Machinery. Google ScholarDigital Library
- Benjamin Rau, Florian Frieb, Michael Krone, Christoph Muller, and Thomas Ertl. 2016. Enhancing visualization of molecular simulations using sonification. In 2016 IEEE 1st International Workshop on Virtual and Augmented Reality for Molecular Science (VARMS@IEEEVR). IEEE, Arles, France, 25--30. Google ScholarCross Ref
- Roli. 2020. The JUCE Library. https://juce.com (accessed: 03.07.2020).Google Scholar
- Julius.O. Smith. 2010. Physical Audio Signal Processing, online book, 2010 edition. W3K Publishing.Google Scholar
- Robert D Sorkin. 1988. Why are people turning off our alarms? The Journal of the Acoustical Society of America 84, 3 (1988), 1107--1108.Google ScholarCross Ref
- Marc Sosnick and William Hsu. 2011. Implementing a Finite Difference-Based Real-time Sound Synthesizer using GPUs. In Proceedings of the International Conference on New Interfaces for Musical Expression. 264--267. Google ScholarCross Ref
- Mark D Temple. 2017. An auditory display tool for DNA sequence analysis. BMC bioinformatics 18, 1 (2017), 1--11.Google Scholar
- Noam Tractinsky. 2004. Toward the study of aesthetics in information technology. ICIS 2004 proceedings (2004), 62.Google Scholar
- Robert Tubb, Anssi Klapuri, and Simon Dixon. 2012. The Wablet: Scanned Synthesis on a Multi-Touch Interface. In International Conference on Digital Audio Effects.Google Scholar
- Bill Verplank, Max Mathews, and Rob Shaw. 2001. Scanned Synthesis. The Journal of the Acoustical Society of America 109, 5 (2001). http://www.billverplank.com/ScannedSynthesis.PDFGoogle ScholarCross Ref
- Paul Vickers. 2006. Lemma 4: Haptic input+ auditory display= musical instrument?. In International Workshop on Haptic and Audio Interaction Design. Springer.Google ScholarDigital Library
- Paul Vickers. 2016. Sonification and Music, Music and Sonification. Taylor Francis, United Kingdom, 135--144.Google Scholar
- Chi-Hua Yu, Zhao Qin, Francisco J. Martin-Martinez, and Markus J. Buehler. 2019. A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design Using Artificial Intelligence. ACS Nano 13, 7 (jul 2019), 7471--7482. Google ScholarCross Ref
Index Terms
- Towards molecular musical instruments: interactive sonifications of 17-alanine, graphene and carbon nanotubes
Recommendations
Was that me?: exploring the effects of error in gestural digital musical instruments
AM '20: Proceedings of the 15th International Audio Mostly ConferenceTraditional Western musical instruments have evolved to be robust and predictable, responding consistently to the same player actions with the same musical response. Consequently, errors occurring in a performance scenario are typically attributed to ...
Seeking for spectral manipulation of the sound of musical instruments using metamaterials
AM '20: Proceedings of the 15th International Audio Mostly ConferenceThe sound of practically all traditional musical instruments is unique and depends on the collective behavior of various vibrators, each one with their own acoustic and mechanical properties. If we pluck a string of an acoustic guitar, a part of the ...
DJuggling: Sonification of expressive movement performance
SIGGRAPH '23: ACM SIGGRAPH 2023 Real-Time Live!In the real-time demo, we demonstrate how to create a musical performance using juggling movement as an instrument. We have equipped juggling balls with accelerometers, gyroscopes, and WiFi sensors. The system measures acceleration and rotation in a ...
Comments