skip to main content
10.1145/3411109.3411143acmotherconferencesArticle/Chapter ViewAbstractPublication PagesamConference Proceedingsconference-collections
research-article

Towards molecular musical instruments: interactive sonifications of 17-alanine, graphene and carbon nanotubes

Published:16 September 2020Publication History

ABSTRACT

Scientists increasingly rely on computational models of atoms and molecules to observe, understand and make predictions about the microscopic world. Atoms and molecules are in constant motion, with vibrations and structural fluctuations occurring at very short time-scales and corresponding length-scales. But can these microscopic oscillations be converted into sound? And, what would they sound like? In this paper we present our initial steps towards a generalised approach for sonifying data produced by a real-time molecular dynamics simulation. The approach uses scanned synthesis to translate real-time geometric simulation data into audio. The process is embedded within a stand alone application as well as a variety of audio plugin formats to enable the process to be used as an audio synthesis method for music making. We review the relevant background literature before providing an overview of our system. Simulations of three molecules are then considered: 17-alanine, graphene and a carbon nanotube. Four examples are then provided demonstrating how the technique maps molecular features and parameters onto the auditory character of the resulting sound. A case study is then provided in which the sonification/synthesis method is used within a musical composition.

Skip Supplemental Material Section

Supplemental Material

References

  1. Andrew Allen and Nikunj Raghuvanshi. 2015. Aerophones in Flatland: Interactive Wave Simulation of Wind Instruments. ACM Trans. Graph. 34, 4, Article Article 134 (July 2015), 11 pages. Google ScholarGoogle ScholarDigital LibraryDigital Library
  2. Norman L Allinger, Young H Yuh, and Jenn Huei Lii. 1989. Molecular mechanics. The MM3 force fileld for hydrocarbons. 1. Journal of the American Chemical Society 111, 23 (1989), 8551--8566.Google ScholarGoogle ScholarCross RefCross Ref
  3. Robert E. Arbon, Alex J. Jones, Thomas Mitchell, Lars A. Bratholm, and David R Glowacki. 2018. Sonifying Stochastic Walks on Biomolecular Energy Landscapes. In Proceedings of the 24th International Conference on Auditory Display. Houghton.Google ScholarGoogle ScholarCross RefCross Ref
  4. Holger Ballweg, Agnieszka K Bronowska, and Paul Vickers. 2016. Interactive Sonification for Structural Biology and Structure-Based Drug Design. In Proceedings of ISon 2016, 5th Interactive Sonification Workshop. Bielefeld, 41--47. http://interactive-sonification.org/ISon2016/papers/ISon2016-06-BallwegBronowskaVickers.pdfGoogle ScholarGoogle Scholar
  5. Aldo Borgonovo and Goffredo Haus. 1986. Sound Synthesis by Means of Two-Variable Functions: Experimental Criteria and Results. Computer Music Journal 10, 3 (1986), 57--71. http://www.jstor.org/stable/3680260Google ScholarGoogle ScholarCross RefCross Ref
  6. Richard Boulanger, Paris Smaragdis, and John Fitch. 2000. Scanned Synthesis: An Introduction and Demonstration of a New Synthesis and Signal Processing Technique. In International Computer Music Conference.Google ScholarGoogle Scholar
  7. Cockos. 2020. Reaper Digital Audio Workstation. https://www.reaper.fm (accessed: 03.07.2020).Google ScholarGoogle Scholar
  8. D. Creasey. 2016. Audio Processes: Musical Analysis, Modification, Synthesis, and Control. Routledge.Google ScholarGoogle ScholarCross RefCross Ref
  9. Florian Dombois and Gerhard Eckel. 2011. Audification. In The sonification handbook.Google ScholarGoogle Scholar
  10. Peter Eastman, Jason Swails, John D. Chodera, Robert T. McGibbon, Yutong Zhao, Kyle A. Beauchamp, Lee-Ping Wang, Andrew C. Simmonett, Matthew P. Harrigan, Chaya D. Stern, Rafal P. Wiewiora, Bernard R. Brooks, and Vijay S. Pande. 2017. OpenMM 7: Rapid development of high performance algorithms for molecular dynamics. PLOS Computational Biology 13, 7 (07 2017), 1--17. Google ScholarGoogle ScholarCross RefCross Ref
  11. Sebastian L Franjou, Mario Milazzo, Chi-Hua Yu, and Markus J Buehler. 2019. Sounds interesting: can sonification help us design new proteins? Expert Review of Proteomics 16, 11--12 (dec 2019), 875--879. Google ScholarGoogle ScholarCross RefCross Ref
  12. Daan Frenkel and Berend Smit. 2001. Understanding molecular simulation: from algorithms to applications. Vol. 1. Elsevier.Google ScholarGoogle ScholarDigital LibraryDigital Library
  13. Miguel Angel Garcia-Ruiz and Jorge Rafael Gutierrez-Pulido. 2006. An overview of auditory display to assist comprehension of molecular information. Interacting with Computers 18, 4 (2006), 853 -- 868. Special Theme Papers from Special Editorial Board Members (contains Regular Papers). Google ScholarGoogle ScholarDigital LibraryDigital Library
  14. Andre K Geim and Konstantin S Novoselov. 2010. The rise of graphene. In Nanoscience and technology: a collection of reviews from nature journals. World Scientific, 11--19.Google ScholarGoogle Scholar
  15. Google. 2020. GRPC. https://grpc.io (accessed: 03.07.2020).Google ScholarGoogle Scholar
  16. Florian Grond and Jonathan Berger. 2011. Parameter mapping sonification. In The sonification handbook.Google ScholarGoogle Scholar
  17. Florian Grond and Fabio Dall'Antonia. 2008. Sumo. A Sonification Utility for Molecules. In Proceedings of ICAD 2008. International Community for Auditory Display, 1--7. https://smartech.gatech.edu/handle/1853/49952Google ScholarGoogle Scholar
  18. Florian Grond and Thomas Hermann. 2014. Interactive Sonification for Data Exploration: How listening modes and display purposes define design guidelines. Organised Sound 19, 1 (2014), 41--51.Google ScholarGoogle ScholarCross RefCross Ref
  19. Florian Grond, Stefan Janssen, Stefanie Schirmer, and Thomas Hermann. 2010. Browsing RNA Structures by Interactive Sonification. In Proceedings of ISon 2010, 3rd Interactive Sonification Workshop. Stockholm, 11--16. http://www.interactive-sonification.org/ISon2010/proceedings/papers/GrondJanssenSchirmerHermann{_}ISon2010.pdfGoogle ScholarGoogle Scholar
  20. Roald Hoffmann. 1990. Molecular Beauty. The Journal of Aesthetics and Art Criticism 48, 3 (jan 1990), 191--204. Google ScholarGoogle ScholarCross RefCross Ref
  21. Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian E Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica B Hamrick, Jason Grout, Sylvain Corlay, et al. 2016. Jupyter Notebooks-a publishing format for reproducible computational workflows.. In ELPUB. 87--90.Google ScholarGoogle Scholar
  22. Gregory Kramer. 1994. Some organizing principles for representing data with sound. Auditory Display-Sonification, Audification, and Auditory Interfaces (1994), 185--221.Google ScholarGoogle Scholar
  23. Intangible Realities Laboratory. 2020. Narupa iMD Code Repository. https://gitlab.com/intangiblerealities/narupa-applications/narupa-imd (accessed: 03.07.2020).Google ScholarGoogle Scholar
  24. Intangible Realities Laboratory. 2020. Narupa Protocol Code Repository. https://gitlab.com/intangiblerealities/narupa-protocol (accessed: 03.07.2020).Google ScholarGoogle Scholar
  25. Ask Hjorth Larsen, Jens Jørgen Mortensen, Jakob Blomqvist, Ivano E Castelli, Rune Christensen, Marcin Dułak, Jesper Friis, Michael N Groves, Bjørk Hammer, Cory Hargus, Eric D Hermes, Paul C Jennings, Peter Bjerre Jensen, James Kermode, John R Kitchin, Esben Leonhard Kolsbjerg, Joseph Kubal, Kristen Kaasbjerg, Steen Lysgaard, Jón Bergmann Maronsson, Tristan Maxson, Thomas Olsen, Lars Pastewka, Andrew Peterson, Carsten Rostgaard, Jakob Schiøtz, Ole Schütt, Mikkel Strange, Kristian S Thygesen, Tejs Vegge, Lasse Vilhelmsen, Michael Walter, Zhenhua Zeng, and Karsten W Jacobsen. 2017. The atomic simulation environment---a Python library for working with atoms. Journal of Physics: Condensed Matter 29, 27 (2017), 273002. http://stacks.iop.org/0953-8984/29/i=27/a=273002Google ScholarGoogle ScholarCross RefCross Ref
  26. Ryan McGee. 2013. VOSIS: a Multi-touch Image Sonification Interface.. In NIME. 460--463.Google ScholarGoogle Scholar
  27. Ryan McGee, Daniel Ashbrook, and Sean White. 2012. SenSynth: a Mobile Application for Dynamic Sensor to Sound Mapping.. In NIME 2012.Google ScholarGoogle Scholar
  28. Thomas Mitchell, Joseph Hyde, Philip Tew, and David R Glowacki. 2014. dance-room Spectroscopy: At the Frontiers of Physics, Performance, Interactive Art and Technology. Leonardo 49, 2 (aug 2014), 138--147. Google ScholarGoogle ScholarCross RefCross Ref
  29. Thomas J. Mitchell and Alex J. Jones. 2020. Graphene Synth Repository. https://bitbucket.org/teamaxe/graphenesynth (accessed: 03.07.2020).Google ScholarGoogle Scholar
  30. Donald A Norman. 2004. Emotional design: Why we love (or hate) everyday things. Basic Civitas Books.Google ScholarGoogle Scholar
  31. Michael B. O'Connor, Simon J. Bennie, Helen M. Deeks, Alexander Jamieson-Binnie, Alex J. Jones, Robin J. Shannon, Rebecca Walters, Thomas J. Mitchell, Adrian J. Mulholland, and David R. Glowacki. 2019. Interactive molecular dynamics in virtual reality from quantum chemistry to drug binding: An open-source multi-person framework. arXiv:1902.01827 Google ScholarGoogle ScholarCross RefCross Ref
  32. Joshua Peschke and Axel Berndt. 2017. The geometric oscillator: Sound synthesis with cyclic shapes. In ACM International Conference Proceeding Series, Vol. Part F1319. Association for Computing Machinery. Google ScholarGoogle ScholarDigital LibraryDigital Library
  33. Benjamin Rau, Florian Frieb, Michael Krone, Christoph Muller, and Thomas Ertl. 2016. Enhancing visualization of molecular simulations using sonification. In 2016 IEEE 1st International Workshop on Virtual and Augmented Reality for Molecular Science (VARMS@IEEEVR). IEEE, Arles, France, 25--30. Google ScholarGoogle ScholarCross RefCross Ref
  34. Roli. 2020. The JUCE Library. https://juce.com (accessed: 03.07.2020).Google ScholarGoogle Scholar
  35. Julius.O. Smith. 2010. Physical Audio Signal Processing, online book, 2010 edition. W3K Publishing.Google ScholarGoogle Scholar
  36. Robert D Sorkin. 1988. Why are people turning off our alarms? The Journal of the Acoustical Society of America 84, 3 (1988), 1107--1108.Google ScholarGoogle ScholarCross RefCross Ref
  37. Marc Sosnick and William Hsu. 2011. Implementing a Finite Difference-Based Real-time Sound Synthesizer using GPUs. In Proceedings of the International Conference on New Interfaces for Musical Expression. 264--267. Google ScholarGoogle ScholarCross RefCross Ref
  38. Mark D Temple. 2017. An auditory display tool for DNA sequence analysis. BMC bioinformatics 18, 1 (2017), 1--11.Google ScholarGoogle Scholar
  39. Noam Tractinsky. 2004. Toward the study of aesthetics in information technology. ICIS 2004 proceedings (2004), 62.Google ScholarGoogle Scholar
  40. Robert Tubb, Anssi Klapuri, and Simon Dixon. 2012. The Wablet: Scanned Synthesis on a Multi-Touch Interface. In International Conference on Digital Audio Effects.Google ScholarGoogle Scholar
  41. Bill Verplank, Max Mathews, and Rob Shaw. 2001. Scanned Synthesis. The Journal of the Acoustical Society of America 109, 5 (2001). http://www.billverplank.com/ScannedSynthesis.PDFGoogle ScholarGoogle ScholarCross RefCross Ref
  42. Paul Vickers. 2006. Lemma 4: Haptic input+ auditory display= musical instrument?. In International Workshop on Haptic and Audio Interaction Design. Springer.Google ScholarGoogle ScholarDigital LibraryDigital Library
  43. Paul Vickers. 2016. Sonification and Music, Music and Sonification. Taylor Francis, United Kingdom, 135--144.Google ScholarGoogle Scholar
  44. Chi-Hua Yu, Zhao Qin, Francisco J. Martin-Martinez, and Markus J. Buehler. 2019. A Self-Consistent Sonification Method to Translate Amino Acid Sequences into Musical Compositions and Application in Protein Design Using Artificial Intelligence. ACS Nano 13, 7 (jul 2019), 7471--7482. Google ScholarGoogle ScholarCross RefCross Ref

Index Terms

  1. Towards molecular musical instruments: interactive sonifications of 17-alanine, graphene and carbon nanotubes

          Recommendations

          Comments

          Login options

          Check if you have access through your login credentials or your institution to get full access on this article.

          Sign in
          • Published in

            cover image ACM Other conferences
            AM '20: Proceedings of the 15th International Audio Mostly Conference
            September 2020
            281 pages
            ISBN:9781450375634
            DOI:10.1145/3411109

            Copyright © 2020 ACM

            Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

            Publisher

            Association for Computing Machinery

            New York, NY, United States

            Publication History

            • Published: 16 September 2020

            Permissions

            Request permissions about this article.

            Request Permissions

            Check for updates

            Qualifiers

            • research-article

            Acceptance Rates

            AM '20 Paper Acceptance Rate29of47submissions,62%Overall Acceptance Rate177of275submissions,64%

          PDF Format

          View or Download as a PDF file.

          PDF

          eReader

          View online with eReader.

          eReader